Copernicusmarine streaming performance#49
Open
erikvansebille wants to merge 4 commits into
Open
Conversation
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a script to test the performance of directly streaming from the
copernicusmarine()store. As also found in Parcels-code/virtualship#357, directly accessing the data from there is much slower than temporarily saving the data to local diskA quick performance overview of the script here on my MacBook:
--load_mode as_filetakes 21 seconds. In this case, the full DataSet is stored to a local netcdf file, and then loaded again. Note that it automatically loads as NumPy (probably because the file is only 256MB in size)--load_mode ds_loadtakes 20 seconds. In this case, the full DataSet is not stored on disk but loaded directly in memory with ads.load()--load_mode streamingtakes 518 seconds, even when we useto_windowed_arrays(). As this is the preferred way to stream data from CopernicusMarine (no need for temporary storage, and able to scale to much larger datasets than required here), the 25x slowdown is not acceptable.Note 1: I've been able to reduce the
--load_mode streamingtime to 142 seconds in 7c5a0ac, by setting the optionscopernicusmarine.open_dataset(..., service="arco-geo-series", chunk_size_limit=1). This is inspired by a discussion at mercator-ocean/copernicus-marine-toolbox#267Note 2: I can get
--load_mode streamingeven faster (equivalent to--load_mode ds_load) by settingchunk_size_limit=0, but this disables dask altogether so will have a major memory footprint (as it bypasses the WindowedArrays)