Skip to content

Copernicusmarine streaming performance#49

Open
erikvansebille wants to merge 4 commits into
mainfrom
Copernicus_streaming
Open

Copernicusmarine streaming performance#49
erikvansebille wants to merge 4 commits into
mainfrom
Copernicus_streaming

Conversation

@erikvansebille

@erikvansebille erikvansebille commented Jul 17, 2026

Copy link
Copy Markdown
Member

This PR adds a script to test the performance of directly streaming from the copernicusmarine() store. As also found in Parcels-code/virtualship#357, directly accessing the data from there is much slower than temporarily saving the data to local disk

A quick performance overview of the script here on my MacBook:

  • Running with --load_mode as_file takes 21 seconds. In this case, the full DataSet is stored to a local netcdf file, and then loaded again. Note that it automatically loads as NumPy (probably because the file is only 256MB in size)
  • Running with --load_mode ds_load takes 20 seconds. In this case, the full DataSet is not stored on disk but loaded directly in memory with a ds.load()
  • Running with --load_mode streaming takes 518 seconds, even when we use to_windowed_arrays(). As this is the preferred way to stream data from CopernicusMarine (no need for temporary storage, and able to scale to much larger datasets than required here), the 25x slowdown is not acceptable.

Note 1: I've been able to reduce the --load_mode streaming time to 142 seconds in 7c5a0ac, by setting the options copernicusmarine.open_dataset(..., service="arco-geo-series", chunk_size_limit=1). This is inspired by a discussion at mercator-ocean/copernicus-marine-toolbox#267

Note 2: I can get --load_mode streaming even faster (equivalent to --load_mode ds_load) by setting chunk_size_limit=0, but this disables dask altogether so will have a major memory footprint (as it bypasses the WindowedArrays)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant