Skip to content

Add asynchronous load method #10327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 81 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
01e7518
new blank whatsnew
TomNicholas Oct 24, 2024
83e553b
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Oct 24, 2024
e44326d
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 8, 2024
4e4eeb0
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 20, 2024
d858059
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 21, 2024
d377780
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 21, 2024
3132f6a
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 23, 2024
900eef5
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 29, 2024
4c4462f
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Dec 4, 2024
5b9b749
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jan 6, 2025
fadb953
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jan 8, 2025
57d9d23
Merge branch 'main' of https://github.com/TomNicholas/xarray
TomNicholas Mar 13, 2025
11170fc
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Mar 19, 2025
0b8fa41
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Mar 20, 2025
f769f85
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Mar 20, 2025
4eef318
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Apr 7, 2025
29242a4
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas May 15, 2025
e6b3b3b
test async load using special zarr LatencyStore
TomNicholas May 15, 2025
3ceab60
don't use dask
TomNicholas May 16, 2025
071c35a
async all the way down
TomNicholas May 16, 2025
29374f9
remove assert False
TomNicholas May 16, 2025
ab12bb8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 16, 2025
62aa39d
add pytest-asyncio to CI envs
TomNicholas May 16, 2025
dfe8bf7
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas May 16, 2025
a906dec
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 16, 2025
629ab31
assert results are identical
TomNicholas May 16, 2025
7e9ae0f
implement async load for dataarray and dataset
TomNicholas May 17, 2025
d288351
factor out common logic
TomNicholas May 17, 2025
e0731a0
consolidate tests via a parametrized fixture
TomNicholas May 17, 2025
9b41e78
async_load -> load_async
TomNicholas May 17, 2025
67ba26a
make BackendArray an ABC
TomNicholas May 18, 2025
9344e2e
explain how to add async support for any backend in the docs
TomNicholas May 18, 2025
f8f8563
add new methods to api docs
TomNicholas May 19, 2025
30ce9be
whatsnew
TomNicholas May 19, 2025
5d15bbd
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas May 19, 2025
1f02de1
Merge branch 'main' into async.load
TomNicholas May 19, 2025
2342b50
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 19, 2025
b6d4a82
Fix ci/minimum_versions.py
TomNicholas May 19, 2025
2079d7e
fix formatting
TomNicholas May 21, 2025
48e4534
concurrently load different variables in ds.load_async using asyncio.…
TomNicholas May 21, 2025
cca7589
test concurrent loading of multiple variables in one dataset
TomNicholas May 21, 2025
dfe9b87
fix non-awaited load_async
TomNicholas May 21, 2025
84099f3
rearrange test order
TomNicholas May 21, 2025
ab000c8
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 21, 2025
a8b7b46
add test for orthogonal indexing
TomNicholas May 23, 2025
82c7654
explicitly forbid orthogonal indexing
TomNicholas May 23, 2025
5eacdb0
support async orthogonal indexing via https://github.com/zarr-develop…
TomNicholas May 23, 2025
9f33c09
Merge branch 'async.load' of https://github.com/TomNicholas/xarray in…
TomNicholas May 23, 2025
093bf50
add test for vectorized indexing (even if it doesn't work)
TomNicholas May 23, 2025
4073a24
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 23, 2025
842a06c
add test for basic indexing
TomNicholas May 23, 2025
e19ab55
correct test to actually use vectorized indexing
TomNicholas May 23, 2025
b9e8e06
refactor to parametrize indexing test
TomNicholas May 23, 2025
8bc7bea
implement async vectorized indexing
TomNicholas May 29, 2025
6c47e3f
revert breaking change to BackendArray
TomNicholas May 29, 2025
a86f646
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 29, 2025
884ce13
remove indirection in _ensure_cached method
TomNicholas May 29, 2025
a43af86
IndexingAdapters don't need async get
dcherian May 29, 2025
17d7a0e
Add tests
dcherian May 29, 2025
d824a2d
Add decoding test
dcherian May 29, 2025
6a13611
Add IndexingAdapter mixin
dcherian May 29, 2025
d79ed54
[cherry] Making decoding arrays lazy too
dcherian May 30, 2025
1da3359
parametrized over isel and sel
TomNicholas May 30, 2025
dded9e0
mock zarr.AsyncArray.getitem in test
TomNicholas May 30, 2025
4c347ad
tidy up the mocking
TomNicholas May 30, 2025
b4ed8ee
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jun 20, 2025
0b55247
Merge branch 'main' into async.load
TomNicholas Jun 20, 2025
4018e28
ensure the correct zarr class's method is patched for each test
TomNicholas Jun 20, 2025
6da81ce
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 20, 2025
7972164
add degenerate test case of no indexing
TomNicholas Jun 20, 2025
618424a
factor out the Latency part of LatencyStore
TomNicholas Jun 20, 2025
cd97481
use mocks in multiple objects test
TomNicholas Jun 20, 2025
75abdec
use mocks in multiple variables test
TomNicholas Jun 20, 2025
7409372
trim latencystore down to just what's needed to dodge https://github.…
TomNicholas Jun 20, 2025
1f79034
parametrizing indexing test over xarray classes
TomNicholas Jun 20, 2025
9881e8d
ensure we actually test vectorized indexing for Variable
TomNicholas Jun 20, 2025
1197798
use create_test_data
TomNicholas Jun 20, 2025
642fd48
add @pytest.mark.asyncio
dcherian Jun 20, 2025
a5a44f5
Merge branch 'main' into async.load
TomNicholas Jul 22, 2025
f22b56b
remove outdated readonly_store
TomNicholas Jul 22, 2025
ebfede5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 22, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/minimum_versions.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
"coveralls",
"pip",
"pytest",
"pytest-asyncio",
"pytest-cov",
"pytest-env",
"pytest-mypy-plugins",
Expand Down
1 change: 1 addition & 0 deletions ci/requirements/all-but-dask.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ dependencies:
- pip
- pydap
- pytest
- pytest-asyncio
- pytest-cov
- pytest-env
- pytest-mypy-plugins
Expand Down
1 change: 1 addition & 0 deletions ci/requirements/all-but-numba.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ dependencies:
- pyarrow # pandas raises a deprecation warning without this, breaking doctests
- pydap
- pytest
- pytest-asyncio
- pytest-cov
- pytest-env
- pytest-mypy-plugins
Expand Down
1 change: 1 addition & 0 deletions ci/requirements/bare-minimum.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ dependencies:
- coveralls
- pip
- pytest
- pytest-asyncio
- pytest-cov
- pytest-env
- pytest-mypy-plugins
Expand Down
1 change: 1 addition & 0 deletions ci/requirements/environment-3.14.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ dependencies:
- pyarrow # pandas raises a deprecation warning without this, breaking doctests
- pydap
- pytest
- pytest-asyncio
- pytest-cov
- pytest-env
- pytest-mypy-plugins
Expand Down
1 change: 1 addition & 0 deletions ci/requirements/environment-windows-3.14.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ dependencies:
- pyarrow # importing dask.dataframe raises an ImportError without this
- pydap
- pytest
- pytest-asyncio
- pytest-cov
- pytest-env
- pytest-mypy-plugins
Expand Down
1 change: 1 addition & 0 deletions ci/requirements/environment-windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ dependencies:
- pyarrow # importing dask.dataframe raises an ImportError without this
- pydap
- pytest
- pytest-asyncio
- pytest-cov
- pytest-env
- pytest-mypy-plugins
Expand Down
1 change: 1 addition & 0 deletions ci/requirements/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ dependencies:
- pydap
- pydap-server
- pytest
- pytest-asyncio
- pytest-cov
- pytest-env
- pytest-mypy-plugins
Expand Down
1 change: 1 addition & 0 deletions ci/requirements/min-all-deps.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ dependencies:
- pip
- pydap=3.5.0
- pytest
- pytest-asyncio
- pytest-cov
- pytest-env
- pytest-mypy-plugins
Expand Down
1 change: 1 addition & 0 deletions doc/api-hidden.rst
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,7 @@
Variable.isnull
Variable.item
Variable.load
Variable.load_async
Variable.max
Variable.mean
Variable.median
Expand Down
Loading
Loading