feat(zarr-metadata)!: read metadata against definitions in a scope, and make every model a pair of document and scope

.github/workflows/zarr-metadata.yml

@@ -102,7 +102,7 @@ jobs:
           repo: casey/just
           version: 1.58.0
       - name: Run pyright
-        # The pyright version and interpreter pins live in the justfile.
+        # Pyright runs unpinned; the interpreter pin lives in the justfile.
         run: just typecheck
 
   docs:

packages/zarr-metadata/README.md

@@ -13,9 +13,9 @@ Two layers and an optional integration:
   and [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html)
   specifications, plus types for [`zarr-extensions`](https://github.com/zarr-developers/zarr-extensions/)
   and a few widely-used-but-unspecified entities (e.g. consolidated metadata).
-- **Document models** (`zarr_metadata.model`): canonical frozen-dataclass
-  models of whole metadata documents, with validators, loc-aware
-  parsers, and store-key (de)serialization. A document produced by `to_json`
+- **Document models** (`zarr_metadata.model`): each model is a metadata
+  document and the scope it was read in, with validators, loc-aware
+  readers, and store-key (de)serialization. A document produced by `to_json`
   shares no mutable state with the model that produced it.
 - **Optional Pydantic integration** (`zarr_metadata.pydantic`, requires
   Pydantic 2.13 or newer): each model as a Pydantic field type that validates
@@ -44,20 +44,34 @@ parser and returns the same normalized model class:
 from pydantic import TypeAdapter
 import zarr_metadata.pydantic as zmp
 
-metadata = TypeAdapter(zmp.ZarrV3ArrayMetadata).validate_python(raw)
+adapter: TypeAdapter[zmp.ZarrV3ArrayMetadata] = TypeAdapter(zmp.ZarrV3ArrayMetadata)
+metadata = adapter.validate_python(raw)
 encoded = metadata.to_key_value()["zarr.json"]
 ```
 
 A bare `TypeAdapter` over a public document `TypedDict` is a coercive shape
 adapter, not a Zarr conformance validator; it may coerce values or discard
 members that the strict model parser rejects.
 
+## Dependencies
+
+The core of this package depends on `typing-extensions` and
+`annotated-types` only. It does not depend on a validation framework,
+and it will not: a dependency on pydantic, or any other framework with
+its own release cadence and compiled parts, would pin that framework for
+every consumer of zarr-metadata and collide with the pins consumers
+already carry. Instead the package reads a `TypedDict` as the typing spec
+defines it with its own checker (`zarr_metadata.typed_json.check`), and
+spells bounds in the `annotated-types` vocabulary, which pydantic reads
+too. `zarr_metadata.pydantic` is an optional integration over the models;
+nothing in the core imports it.
+
 ## Validation boundary
 
 The model validators enforce the declared document structure and a small set
 of context-free consistency rules, including fixed format literals, finite
-JSON numbers, non-negative dimensions, and one `dimension_names` entry per
-array dimension. In a v3 document they also read
+JSON numbers outside `attributes`, non-negative dimensions, and one
+`dimension_names` entry per array dimension. In a v3 document they also read
 each extension point -- the data type, chunk grid, chunk key encoding, each
 codec and each storage transformer -- through the definition that claims its
 name in a scope, `CORE_AND_EXTENSIONS` unless a `context` is passed: a
@@ -71,13 +85,163 @@ against the chunk it is handed, a shard's inner and index codecs too.
 The validators do no arithmetic on values: whether a fill value survives
 a `cast_value` round trip is not judged.
 
-The Pydantic integration's generated JSON Schemas express independently
-checkable document structure and field constraints, but they are not a
-replacement for runtime model validation. Standard JSON Schema treats a
-mathematically integral number such as `1.0` as an integer, while the runtime
-boundary requires Python `int` values, and it cannot express arbitrary
-same-length relations such as `dimension_names` versus `shape` or v2 `chunks`
-versus `shape`. Consumers should run the model parser after schema validation.
+Two choices the specs' words leave open, or settle two ways:
+
+- **`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** The spec
+  interprets no attribute, and zarr-python and xarray write those numbers
+  there (a CF `_FillValue`, say). The models read them, and `to_key_value`
+  writes them back as those bare tokens, which a strict JSON parser
+  refuses. `check`, from `zarr_metadata.typed_json`, refuses a non-finite
+  number wherever it is, attributes included.
+- **A reader walks 256 levels of nesting.** A value nested deeper is an
+  `invalid_value` at the level past the last, wherever it sits. Every
+  reader, writer and comparison takes one frame for each level, and
+  `copy.deepcopy`, and `pickle` before Python 3.12, two: a document at
+  the cap takes about half of the interpreter's default limit, and the
+  rest is the caller's.
+- **`consolidated_metadata: null` is a problem.** zarr-python 3.0 and 3.1
+  wrote it on a group they had not consolidated; the spec says an object,
+  and the package models nothing else as right.
+  `read_repaired_node_metadata_v3` removes it before reading.
+- **An extension is named as the spec names one**, `^[a-z][a-z0-9-_.]+$`,
+  or by a URI, which earlier versions of the spec required; any other
+  name is refused before a definition is asked, so `""` and `"foo/bar"`
+  are not unknown extensions but problems.
+- **`must_understand: false` is refused at every extension point**, codecs
+  and storage transformers too, though the core spec names only the data
+  type, chunk grid and chunk key encoding: a reader that skips a codec
+  reads wrong bytes as surely as one that skips a data type reads wrong
+  values. It keeps its meaning on an unknown top-level member, which a
+  reader can skip.
+
+A regular grid's chunk lengths are at least 1, along a dimension of
+length 0 too: "Chunk sizes must be greater than zero", the regular grid
+spec says. The core spec's "non-zero when the corresponding dimensions
+of the arrays have non-zero length" says less, and allows nothing more,
+so a document with a 0 there, as zarr-python 3.0 and 3.1 wrote for an
+empty dimension, is refused.
+
+`read_array_metadata_v3` reads a document once and returns everything
+the read found: each field as the scope read it -- `AcceptedField` by the
+definition that claims its name, `UnclaimedField` when none does, or
+`RefusedField` -- with where it sits and the kind it was read as, each codec
+with the chunk it is handed, every problem, and the model when there is
+none; `from_json` is that model, or the problems raised. A consumer's
+own policy is a walk over the fields, with nothing read twice:
+`with_problems` gives each with its problems, those located in it and in
+the fields it holds, as zod's `flattenError` groups issues, and
+`canonical_of` spells a field with none in the fewest words, without
+reading it again. Which fields go beyond the core spec, say -- a field
+that names nothing is a problem already -- and how each is spelled most
+simply:
+
+```python
+from zarr_metadata.model import read_array_metadata_v3
+from zarr_metadata.v3.definition import CORE, canonical_of, with_problems
+
+reading = read_array_metadata_v3(raw)
+beyond_core = [
+    loc
+    for loc, field in reading.fields()
+    if field.name is not None and CORE.claimant(field.read_as, field.name) is None
+]
+simplest = {
+    loc: canonical_of(field, problems)
+    for loc, field, problems in with_problems(reading.fields(), reading.problems)
+}
+metadata = reading.metadata  # None when reading.problems is not empty
+```
+
+`read_group_metadata_v3` reads a group the same way, and each document
+its consolidated metadata holds once. `read_node_metadata_v3` reads a
+`zarr.json` of either kind as the node its `node_type` says it is, as
+a discriminated union reads its tag: a document that says neither reads
+as `ZarrV3UnknownNodeReading`, with the problems, and nothing else of it
+is read but its `zarr_format`, so a document of another format says it
+is not v3. `node_metadata_from_json_v3` and `node_metadata_from_key_value_v3`
+build the model of either kind, as the models' own `from_json` and
+`from_key_value` build one.
+
+Consolidated metadata holds the hierarchy below its group, the group its
+root: the document of the node at `/a/b` sits at the key `a/b`, and the
+documents and the group make a tree in which only groups hold nodes and
+each node's parent is held. `NodeName` and `NodePath`, in
+`zarr_metadata.v3`, are the strings the spec's rules for node names and
+paths hold of, modeled on zarrs' types of those names, and
+`validate_node_name_v3`, `is_node_name_v3` and `parse_node_name_v3`, and
+their `node_path` twins, judge a string by them.
+
+A member the spec does not define is not a field; the model's
+`must_understand_fields` names those a reader must understand.
+
+A v3 model is its document and the scope it was read in:
+`ZarrV3ArrayMetadata(document, context=None)` reads the document in the
+scope, `CORE_AND_EXTENSIONS` when none is given, and raises
+`MetadataValidationError` with every problem, so no model is built
+invalid; `to_json` writes the document as it was written, and
+`to_key_value` writes it as it is. Every typed member is a view of that
+read: each field as the scope read it, an `AcceptedField` or an `UnclaimedField`, and
+`shape`, `attributes` and the rest as the read refined them, read-only
+at every level: a list given for an array as a tuple, an object as a
+read-only mapping; `to_json` gives plain containers. A model is changed
+by `update`, which puts
+JSON members in place of the document's and reads the result in the
+model's own scope, so no scope is passed back in; `with_context` reads
+the document in another scope, and `refined_in` only in one that claims
+what this one left unclaimed and contradicts nothing, raising
+`ScopeConflictError` otherwise. The documents a group's
+`consolidated_metadata` holds are models of the group's scope, built from
+the group's one read; it takes node models as entries too, each accepted
+when its claims refine into the group's scope and refused at its path
+otherwise, and `Context.joined` is the scope to consolidate children of
+several scopes in. Every reader takes `context=None` for the default
+scope, `CORE_AND_EXTENSIONS`.
+
+Two models are equal when they mean the same document, however each is
+spelled. What the package interprets -- each field, and the fill value
+against its data type -- compares by its canonical spelling, as
+`canonical_of` and `canonical_fill_value` give it: `"NaN"` and
+`"0x7fc00000"` are one `float32` fill value, `0.0` and `-0.0` two, and a
+blosc with and without the `typesize` that `noshuffle` ignores one
+codec, and a v2 `dtype` by its family and size, `<b1` and `|b1` one
+dtype. What it does not interpret -- attributes, extra fields, and the
+configuration of a field nothing in scope claims -- compares as JSON
+text, which tells `true` from `1` and `-0.0` from `0.0`, and takes `NaN`
+for itself. Equal models hash alike, and may write two documents:
+`to_json` writes each as it was given. A model holds nothing that can be
+changed in place: what it hands out is read-only at every level.
+
+`node_metadata_json_schema_v3` writes what the validators read as a
+JSON Schema, draft 2020-12, for an editor that checks a `zarr.json` as it
+is written, or a validator in another language. Each extension point is
+a field as its scope reads it: a configuration as its definition's
+TypedDict says, bounds and all, and a name nothing in the scope claims
+with any configuration. The fill value is what the data type it names
+takes. `field_json_schema(kind, context)`, in
+`zarr_metadata.v3.definition`, writes one field's schema, and
+`json_schema`, in `zarr_metadata.typed_json`, any TypedDict's, as `check`
+reads it. A schema says what each member is, and not what the rules say
+of members together, so a document it accepts may still have a problem;
+a JSON document the validators accept, it accepts. A validator reads
+JSON as a parser gives it, arrays as lists: a model's `to_json` writes
+tuples, which a Python validator does not take for arrays.
+
+```python
+import json
+from zarr_metadata.model import node_metadata_json_schema_v3
+
+with open("zarr.schema.json", "w") as f:
+    json.dump(node_metadata_json_schema_v3(), f, indent=2)
+```
+
+The Pydantic integration's field types have JSON Schemas of their own,
+for a model that holds them: an extension point there is a name and any
+configuration, read in no scope, and v2 documents have one too. For a
+`zarr.json`, use `node_metadata_json_schema_v3`. Neither replaces the
+validators: JSON Schema takes a number such as `1.0` for an integer,
+where the models require an `int`, and says nothing of what members read
+together say, such as `dimension_names` against `shape` or v2 `chunks`
+against `shape`. Run the model parser after schema validation.
 
 ## Scope
 

packages/zarr-metadata/changes/4420.bugfix.md

@@ -14,10 +14,8 @@ failing to decode the document. `is_json`, `validate_json` and
 `parse_json` judge by RFC 8259 alone, so a document the node guards
 accept can hold a value `is_json` refuses.
 
-`to_key_value` validates the document it writes as `from_key_value`
-validates what it reads, and raises `MetadataValidationError` with every
-problem instead of writing a document the reader refuses. A model built
-by hand is not validated, so the writer is where these are caught:
+A model whose document the reader refuses raises
+`MetadataValidationError` with every problem, instead of being written:
 
 - a non-finite number outside attributes, which was refused before too;
 - a non-string attribute key, which was written as a string;

packages/zarr-metadata/changes/4434.feature.1.md

@@ -5,7 +5,6 @@ gives wrong bytes as surely as ignoring a data type gives wrong values,
 and the spec naming only three points is read as an oversight. A document
 that declares a codec or a storage transformer ignorable now has a problem
 at `codecs.N.must_understand` or `storage_transformers.N.must_understand`,
-so the readers refuse it, and `to_key_value` refuses to write a model that
-holds one. The JSON schema `zarr_metadata.pydantic` generates for an array
+so the readers refuse it, and no model writes one. The JSON schema `zarr_metadata.pydantic` generates for an array
 document says the same. A metadata field read on its own still takes
 `must_understand: false`.

packages/zarr-metadata/changes/4434.feature.md

@@ -17,13 +17,13 @@ when built, and nothing happens at class creation.
 Reading a field is three steps, each usable on its own by a caller
 holding nothing but JSON. `check(value, SomeTypedDict)`, from
 `zarr_metadata.typed_json`, type-checks JSON against a TypedDict and needs
-nothing else; `definition.judge(configuration)` is the check, each nested
+nothing else; `definition.read_configuration(configuration)` is the check, each nested
 field's envelope judged, and then the rules; `resolve(field,
 CodecDefinition, scope)` reads a whole field in a scope -- its envelope
 judged, its name related to a definition, its configuration judged, each
 nested field read the same way -- and returns what the scope made of it,
-`Resolved`, with every problem located. A name nothing in scope claims is
-`out_of_scope` and left unjudged. A shard's pipelines, a cast's target
+with every problem located. A name nothing in scope claims is left
+unjudged. A shard's pipelines, a cast's target
 type and a struct's field types are nested fields, annotated `CodecField`
 and `DataTypeField`, and read in the scope their field is read in.
 `configuration_of(resolved, GZIP_CODEC)` is a read configuration typed as
@@ -44,8 +44,8 @@ nothing of the keys it does not declare, and a member typed
 `canonicalize(field, kind, scope)` gives a field without problems in its
 simplest equivalent spelling: each nested field in its own simplest
 spelling, then its definition's own `canonical`, and the envelope in the
-fewest words -- the bare name when nothing is configured, and no
-`must_understand`. A field with any problem, an unknown key included, has
+fewest words every reader takes, with no `must_understand`. A field with
+any problem, an unknown key included, has
 none, since a simpler spelling would erase what its author wrote. What
 `canonical` gives is judged again, and one that does not hold is a
 `ValueError`, a fault in the definition.

packages/zarr-metadata/changes/4436.feature.md

@@ -7,7 +7,5 @@ against the definition that claims its name in a scope,
 configuration, and a document holding one, accepted before, is refused.
 So do the `is_*` and `parse_*` beside it and the v3 group validators,
 with each array an inline `consolidated_metadata` holds, and the v3
-model classes' `from_json`, `from_key_value` and `to_key_value` take the
-same `context`, so a model read in a scope is written in it. The
-pydantic types read in `CORE_AND_EXTENSIONS`. A name nothing in scope
-claims is left unjudged.
+model classes' `from_json` and `from_key_value` take the same `context`.
+A name nothing in scope claims is left unjudged.

packages/zarr-metadata/changes/4443.feature.2.md

@@ -6,5 +6,5 @@ struct fill value missing a field are each a problem at `fill_value`,
 where the package accepted them before. `fill_value_problems(data_type,
 value)` judges a fill value against a data type field a scope read, and
 a field that is read keeps the fields it read inside as
-`Resolved.nested`, a `Nested` mapping by location. A data type nothing
+`nested`, a `Nested` mapping by location. A data type nothing
 in scope claims leaves its fill value unjudged.

packages/zarr-metadata/changes/4443.feature.3.md

@@ -1,7 +1,6 @@
 **Breaking:** a v3 array's `chunk_grid` is judged against its `shape`, by
 the grid's definition. A regular grid whose `chunk_shape` does not have
-one length per dimension of the shape, or has a length of 0 for a
-dimension that is not empty, and a rectilinear grid whose `chunk_shapes`
-does not have one entry per dimension, or whose chunk lengths fall short
-of their dimension, each have a problem in `chunk_grid.configuration`,
-where the package accepted them before.
+one length per dimension of the shape, and a rectilinear grid whose
+`chunk_shapes` does not have one entry per dimension, or whose chunk
+lengths fall short of their dimension, each have a problem in
+`chunk_grid.configuration`, where the package accepted them before.

packages/zarr-metadata/changes/4490.feature.md

@@ -0,0 +1,29 @@
+**Breaking:** metadata is read against definitions in a scope, and every
+model is the pair of its document and the scope it was read in. A field
+(a v3 data type, chunk grid, chunk key encoding, codec or storage
+transformer; a v2 dtype or numcodecs codec) is `AcceptedField` by the definition
+in scope that claims its name, `UnclaimedField` when nothing does, or
+`RefusedField` with every problem; `read_array_metadata_v3`,
+`read_group_metadata_v3`, `read_node_metadata_v3` and
+`read_array_metadata_v2` read a document once and return everything the
+read found, the model among it. A model is built from a document in a
+scope (`CORE_AND_EXTENSIONS` or `CORE_V2` by default) and is never
+invalid; two models are equal when their documents mean the same;
+`update` reads new members, given as JSON, in the model's own scope,
+`with_context` and `refined_in` read the document in another, and
+`refines` orders models by information. A scope is of one Zarr format, `ZarrV2Context` or `ZarrV3Context`, and a
+reader given a scope of the other format raises `TypeError`; scopes
+compare, join and report disagreements, a group's consolidated metadata accepts node models,
+number types carry their bounds and problems carry `input` and `ctx`,
+JSON Schemas are exported for a TypedDict, a field and a `zarr.json`,
+and `read_repaired_node_metadata_v3` and
+`read_repaired_consolidated_metadata_v2` undo known writer bugs below
+the strict model. Gone: the dataclass models and `construct`, the
+`...Partial` TypedDicts (now `...Update`), `ZarrV3NamedConfig`,
+`ZarrV3MetadataField`, `Resolution`/`Unread`, the `context` argument of
+`update` and `to_key_value`, and the helpers `zarr_metadata.model`,
+`zarr_metadata.v3.definition` and `zarr_metadata.typed_json` re-exported
+from the core. Refused where accepted before: `must_understand: false`,
+extension names outside `^[a-z][a-z0-9-_.]+$` or a URI, a chunk length
+of 0, `consolidated_metadata: null`, and a structured v2 dtype with
+`fill_value: 0`.