Can't call open_mfdataset without creating chunked dask arrays

What happened?

Passing chunks=None to xr.open_dataset/open_mfdataset is supposed to avoid using dask at all, returning lazily-indexed numpy arrays even if dask is installed. However chunks=None doesn't currently work for xr.open_mfdataset as it gets silently coerced internally to chunks={}, which creates dask chunks aligned with the on-disk files.

Offending line of code:

xarray/xarray/backends/api.py

Line 1040 in 12123be

open_kwargs = dict(engine=engine, chunks=chunks or {}, **kwargs)

What did you expect to happen?

Passing chunks=None to open_mfdataset should return lazily-indexed numpy arrays, like open_dataset does.

Minimal Complete Verifiable Example

ds = xr.tutorial.open_dataset("air_temperature")

ds1 = ds.isel(time=slice(None, 1000))
ds2 = ds.isel(time=slice(1000, None))

ds1.to_netcdf('air1.nc')
ds2.to_netcdf('air2.nc')

combined = xr.open_mfdataset(['air1.nc', 'air2.nc'], chunks=None)

print(type(combined['air'].data))

MVCE confirmation

Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
Complete example — the example is self-contained, including all data and the text of any traceback.
Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
New issue — a search of GitHub Issues suggests this is not a duplicate.
Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

dask.array.core.Array

Anything else we need to know?

As the default is None, changing this without changing the default would be a breaking change. But the current behaviour is also not intended.

Environment

main

Passing chunks=None to open_mfdataset should return lazily-indexed numpy arrays, like open_dataset does.

Can't do this without virtual concat machinery (#4628) which someone decided to implement elsewhere 🙄 ;)

We could change the default to chunks={} in anticipation though.

Can't do this without virtual concat machinery (#4628) which someone decided to implement elsewhere 🙄 ;)

😅

It's still broken at the moment though - I had a (ridiculous) case where I don't care that concat will load everything in memory, I just want to completely avoid creating dask.array objects, and right now there is no possible input option to open_mfdataset to do that.

We could change the default to chunks={} in anticipation though.

That's probably more useful, as well as actually being consistent.

See #5704 for changing chunks={} and more discussion.

Passing chunks=None to xr.open_dataset/open_mfdataset is supposed to avoid using dask at all, returning lazily-indexed numpy arrays even if dask is installed.

Agree with having the ability to totally avoid dask and/or any chunking manager, and have the possibility of letting the backend engine handle things on its own. Will also call out that the lazy-indexing shouldn't just return NumPy arrays, but allow for other Array API types (e.g. CuPy arrays).

To elaborate on my use-case, I'm hitting into issues at xarray-contrib/cupy-xarray#81 (comment) because xr.open_mfdataset can't avoid using dask at all, and dask defaults to returning numpy.arrays, but I want CuPy arrays 🙂 (edit: went down the 🐇🕳️ and found #8733 (comment) that mentioned cupy!). Using xr.open_dataset(..., chunks=None) works fine though, it is just that chunks=chunks or {} part that is tripping things up in xr.open_mfdataset.

Add requires_dask marker to two unit tests checking open_mfdataset

Navigation Menu

Repository navigation

Can't call open_mfdataset without creating chunked dask arrays #9038

What happened?

What did you expect to happen?

Minimal Complete Verifiable Example

MVCE confirmation

Relevant log output

Anything else we need to know?

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Can't call open_mfdataset without creating chunked dask arrays #9038

Description

What happened?

What did you expect to happen?

Minimal Complete Verifiable Example

MVCE confirmation

Relevant log output

Anything else we need to know?

Environment

Activity

dcherian commented on May 21, 2024

TomNicholas commented on May 21, 2024

Illviljan commented on May 22, 2024

weiji14 commented on Jun 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions