package xarray and xarray-core in conda-forge

[dcherian]

What is your issue?

The current set of Xarray dependencies is very minimal.

xarray/pyproject.toml

Lines 25 to 29 in 3fd162e

	dependencies = [
	"numpy>=1.23",
	"packaging>=23.1",
	"pandas>=2.0",
	]

This is pretty unfriendly to a new user, and not a great out-of-the-box experience. You can't read any files (except npz, csv, parquet I guess), you can't access any tutorial datasets, you can't make plots, and you're missing a bunch of effectively free performance optimizations.

I think the current set of minimal dependencies is more appropriate to an xarray-core package.
Here are our optional dependencies for example:

xarray/pyproject.toml

Lines 31 to 48 in 3fd162e

	[project.optional-dependencies]
	accel = ["scipy", "bottleneck", "numbagg", "flox", "opt_einsum"]
	complete = ["xarray[accel,io,parallel,viz,dev]"]
	dev = [
	"hypothesis",
	"mypy",
	"pre-commit",
	"pytest",
	"pytest-cov",
	"pytest-env",
	"pytest-xdist",
	"pytest-timeout",
	"ruff",
	"xarray[complete]",
	]
	io = ["netCDF4", "h5netcdf", "scipy", 'pydap; python_version<"3.10"', "zarr", "fsspec", "cftime", "pooch"]
	parallel = ["dask[complete]"]
	viz = ["matplotlib", "seaborn", "nc-time-axis"]

Proposal

I suggest that we migrate to xarray-core and xarray packages in conda-forge.:

1. xarray-core will have the current set of minimal dependencies.
2. For xarray I propose the following dependencies:
   1. flox, opt_einsum, numbagg for accelerated computations
   2. fsspec, netcdf, zarr for reading common datasets & "cloud"
   3. matplotlib for plotting.
   4. pooch to read tutorial datasets

Related: dask packages dask-core and I think matplotlib packages matplotlib-base

[dcherian]

Note that there are many user survey comments asking for performance improvements

• "whatever can speed up computations would be welcomed"
• Optimizations, especially for "resample" and "rolling".
• faster computation
• Faster/smaller dask graph parallelizations?
• faster, less overhead
• "also, can we fix the fact that groupby() on a dimension with only one chunk returns something with a chunk size of one on that dimension? It produces huge graph sizes."

And then this counter-example 🤷🏾‍♂️ : "lightweight version without heavy dependencies"

[max-sixty]

(Yes, I thought I asked something similar a while ago around pooch but can't find it)

Ideally we would allow {name="xarray", default-features=false} for the minority of users that want the slim version. But IIUC python doesn't have any notion of "default but not required dependencies".

• So +1 on xarray-core vs xarray in that case
• Another option would be encouraging xarray[standard], but that doesn't seem like a common thing in python either

[dcherian]

Looking at the 👍🏾 on the first post we seem to have general agreement.

How shall we proceed here? We could:

1. Begin with an announcement
2. Start distributing xarray-core
3. Change the recipe for xarray in two months?

@shoyer it'd be good to get your vote here.

[shoyer]

I am fine changing the package on conda-forge, this seems more in line with user expectations. Not sure we need opt-einsum, but otherwise the core list of recommended dependencies looks good.

…

On Wed, Jul 10, 2024 at 9:53 PM Deepak Cherian ***@***.***> wrote: Looking at the 👍🏾 on the first post we seem to have general agreement. How shall we proceed here? We could: 1. Begin with an announcement 2. Start distributing xarray-core 3. Change the recipe for xarray in two months? @shoyer <https://github.com/shoyer> it'd be good to get your vote here. — Reply to this email directly, view it on GitHub <#9149 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJJFVQPUQ5AAICW5GZ2SM3ZLYFTZAVCNFSM6AAAAABJU2RXDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRSGAZDOMJVGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dcherian]

@pydata/xarray can someone volunteer to take this on please?