Repository navigation
Add the offset chunk key encoding and Array.prepend - #4516
Draft
rhoadesScholar wants to merge 6 commits into
Draft
rhoadesScholar wants to merge 6 commits into
rhoadesScholar wants to merge 6 commits into
Conversation
The `offset` encoding stores each chunk under the `default` key of its grid coordinate plus a fixed per-dimension integer offset, so that an array grows in the negative direction of an axis by whole chunks as a metadata-only change: `shape` grows, `offset` drops, and every existing chunk keeps its key. `Array.prepend_chunks` performs that edit and `Array.prepend` grows and writes in one call; both refuse an array on another encoding rather than rename its chunks. The offset must have one entry per dimension, checked when the metadata is built; the `ChunkKeyEncodingParams` shortcut carries `offset` through to the configuration instead of dropping it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
The store key addresses a shard, so the unit of growth is the chunk grid's shape, not the inner chunk; and a negative axis has to be normalised before the preserved-shape check and the write selection use it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
Drops the public prepend_chunks; Array.prepend grows the array by the data's extent (whole chunks, or shards when sharded) and writes it. Docstrings and the docs paragraph are cut to what the API needs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
This was referenced Oct 15, 2026
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4516 +/- ##
==========================================
+ Coverage 94.71% 94.74% +0.02%
==========================================
Files 94 94
Lines 13663 13741 +78
==========================================
+ Hits 12941 13019 +78
Misses 722 722
🚀 New features to boost your workflow:
|
rhoadesScholar
marked this pull request as ready for review
October 16, 2026 00:13
Author
|
Ready for review; the registry entry is zarr-developers/zarr-extensions#80. cc @d-v-b |
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CSaEKF9nB55gU7PQgQ22X5
Contributor
|
thanks for this PR! I really want to see this feature added to |
rhoadesScholar
marked this pull request as draft
October 16, 2026 19:48
Author
|
Converting to draft while the design is worked out in zarr-developers/zarr-extensions#79. This branch is a prototype and will be reworked to match. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the
offsetchunk key encoding (zarr-developers/zarr-extensions#80): thedefaultkeyof
chunk_coords + offset, one integer per dimension, any sign. Under it an array grows inthe negative direction of an axis by whole chunks as a metadata-only change, and
Array.prepend(data, axis), the mirror ofArray.append, does exactly that:shapegrows,offsetdrops, onezarr.jsonwrite, every existing chunk keeps its key, thendataiswritten in front.
Before / after. Today the only ways to add data before index 0 touch every chunk: rename
each chunk key (filesystem only), get/set/delete each chunk through the store API, or copy the
array. Measured on one machine (NVMe, 64³ uint8 chunks, two chunk rows added in front):
prependminus the write)Script: https://github.com/rhoadesScholar/zarr-chunk-offset/tree/main/bench. The test
test_prepend_writes_in_front_and_leaves_every_existing_chunk_untouchedasserts the propertydirectly: every pre-existing key holds the same bytes after the growth, and only the new keys
are added.
Design
OffsetChunkKeyEncoding(offset, separator="/")inzarr.core.chunk_key_encodings,registered as
"offset". The offset length is checked against the shape whenArrayV3Metadatais built, so a wrong-rank offset fails at creation, not at the first key.ChunkKeyEncodingParamsshortcut ({"name": ..., "separator": ...}) now carriesoffsetinto the configuration; before, that branch rebuilt the configuration with onlyseparator.prependmirrorsappend(same signature, same shape check) with one constraint theformat imposes: the data must span a whole number of chunks (shards, when sharded) along the
axis, since a sub-chunk growth cannot leave the existing chunks in place. On a
defaultorv2array it raises aTypeErrornaming the encoding rather than renaming chunks behindthe caller's back; switching an existing array is a one-line metadata rewrite that the
plugin's
adoptperforms and that could become a core helper later.must_understandcannot befalse for a chunk key encoding), which is the reason this is a new name and not a field on
the regular grid.
Tests: 20 new (encoding round trips, negative components, rank checks, the shortcut, prepend
on plain, sharded, dot-separated and negative-axis arrays, refusals);
tests/test_array.py,tests/test_metadata,tests/test_apiandtests/test_chunk_key_encodings.pypass; mypy and ruff clean. Docs: a "Prepending"paragraph in the arrays guide and a changelog fragment.
Issues
Refs zarr-developers/zarr-specs#9 and zarr-developers/zarr-extensions#30; the registration PR
is zarr-developers/zarr-extensions#80.