feat(format): record the target data type in JsonIndexDetails

docs/src/format/index/scalar/.pages

@@ -9,3 +9,4 @@ nav:
   - N-gram: ngram.md
   - FM-Index: fmindex.md
   - RTree: rtree.md
+  - JSON: json.md

docs/src/format/index/scalar/json.md

@@ -0,0 +1,75 @@
+# JSON Index
+
+The JSON index indexes a single path within a JSON column. It is a wrapper: it
+decodes the value at that path out of each JSONB document, converts the decoded
+values into one Arrow array, and hands that array to an ordinary scalar index —
+the *target index* — which does all of the storage and searching.
+
+A JSON value at a given path has no fixed type, so the wrapper has to pick one
+Arrow type for the whole column before the target index can be built. The
+variants it can pick are the image of the JSONB type tags, so every JSON value
+type is covered:
+
+| JSONB type tag    | `JsonTargetDataType`                 | Value stored in the target index      |
+|-------------------|--------------------------------------|---------------------------------------|
+| `Boolean`         | `JSON_TARGET_DATA_TYPE_BOOLEAN`      | The decoded boolean                   |
+| `Int64`           | `JSON_TARGET_DATA_TYPE_INT64`        | The decoded integer                   |
+| `Float64`         | `JSON_TARGET_DATA_TYPE_FLOAT64`      | The decoded float                     |
+| `String`          | `JSON_TARGET_DATA_TYPE_UTF8`         | The decoded string, unquoted          |
+| `Array`, `Object` | `JSON_TARGET_DATA_TYPE_LARGE_BINARY` | The subtree, re-serialized as JSONB   |
+| `Null`            | —                                    | Indexed as a null in the target index |
+
+A document whose path is absent is indexed as a null, the same as an explicit
+JSON null.
+
+The chosen type is recorded in the index details as `target_data_type`. When it
+is `JSON_TARGET_DATA_TYPE_UNSPECIFIED` — an index written before the details
+carried the type, which a later compaction cannot always recover — the type must
+be recovered by decoding the data again, reading the type tag of the first
+non-null value at the path and falling back to
+`JSON_TARGET_DATA_TYPE_UTF8` when every value is null. That result depends on
+which rows are read, so it is not guaranteed to reproduce the type the index was
+originally built with.
+
+## Index Details
+
+```protobuf
+%%% proto.message.JsonIndexDetails %%%
+```
+
+The target index is identified by the `type_url` of `target_details`, which is
+read to select the target index implementation before `target_details` itself is
+decoded as that implementation's own details message.
+
+## Storage Layout
+
+The JSON index writes no files of its own. Its files are exactly the files of
+its target index, written with that index's names and schemas into the same
+index directory, and the target's format documentation describes them: for
+example [BTree](btree.md) or [Bitmap](bitmap.md).
+
+Reader navigation therefore has two steps. Decode `JsonIndexDetails` to recover
+`path` and `target_details`, select the target implementation from the
+`type_url` of `target_details`, then navigate the target index exactly as a
+standalone index of that type, using `target_details` as its details.
+
+## Accelerated Queries
+
+Only the typed accessor functions are routed to a JSON index, and only when
+their path argument is a literal equal to the indexed `path`:
+
+| Function          | Value type evaluated as |
+|-------------------|-------------------------|
+| `json_get_bool`   | `BOOLEAN`               |
+| `json_get_int`    | `INT64`                 |
+| `json_get_float`  | `FLOAT64`               |
+| `json_get_string` | `UTF8`                  |
+
+Once a predicate is routed, the query types the index can accelerate, and
+whether the answer is exact, are those of the target index.
+
+`json_extract` is deliberately not routed. It evaluates to serialized JSON text
+while the target index holds decoded native values, so an indexed
+`json_extract` predicate would answer a different question than an unindexed
+one; quoting is also not order-preserving, so even a `UTF8` target cannot serve
+its ranges.

protos/index.proto

@@ -246,9 +246,57 @@ message HnswParameters {
   uint32 max_level = 3;
 }
 
+/* The data type a JSON index extracted its values as, before handing them to
+ * the target index.
+ *
+ * A JSON value has no fixed type, so a JSON index decodes the value at its path
+ * into a single Arrow type and trains the target index on that. The variants
+ * below are the image of the JSONB type tags: a JSON boolean becomes BOOLEAN, a
+ * JSON integer INT64, a JSON float FLOAT64, a JSON string UTF8, and a JSON array
+ * or object LARGE_BINARY holding the re-serialized JSONB value.
+ */
+enum JsonTargetDataType {
+  /* The index was built before this field existed, so the type it was trained
+   * on is not recorded and must be inferred by decoding the data again.
+   *
+   * Inference reads the type tag of the first non-null value at the path and
+   * falls back to UTF8 when every value is null, so it depends on which rows
+   * are read: inferring over a different row set, or over disjoint subsets in
+   * separate workers, can produce a different type than the original index.
+   *
+   * A build that understands this field always records it when training a
+   * new index. An index built before the field existed can still carry
+   * UNSPECIFIED after a compaction, because the type it was trained on is
+   * not always recoverable from the target index.
+   */
+  JSON_TARGET_DATA_TYPE_UNSPECIFIED = 0;
+  JSON_TARGET_DATA_TYPE_BOOLEAN = 1;
+  JSON_TARGET_DATA_TYPE_INT64 = 2;
+  JSON_TARGET_DATA_TYPE_FLOAT64 = 3;
+  JSON_TARGET_DATA_TYPE_UTF8 = 4;
+  JSON_TARGET_DATA_TYPE_LARGE_BINARY = 5;
+}
+
 message JsonIndexDetails {
+  // The JSON path whose value is indexed, e.g. `$.user.id`.
   string path = 1;
+  // Details of the index built over the extracted values.
   google.protobuf.Any target_details = 2;
+  /* The type the extracted values were decoded to, which is also the type the
+   * target index was trained on.
+   *
+   * Recording it makes these details a complete description of the index:
+   * rebuilding from them reproduces the original type instead of re-inferring
+   * it from whatever rows the rebuild happens to see. See
+   * JSON_TARGET_DATA_TYPE_UNSPECIFIED for the behavior when it is absent.
+   *
+   * An implementation that rewrites these details without retraining the target
+   * index -- compacting or updating it -- must copy this value through
+   * unchanged, including a value it does not recognize, rather than resetting
+   * it to JSON_TARGET_DATA_TYPE_UNSPECIFIED. Otherwise a variant added by a
+   * later revision is erased by any older reader that touches the index.
+   */
+  JsonTargetDataType target_data_type = 3;
 }
 message BloomFilterIndexDetails {}
 

python/python/tests/test_scalar_index.py

@@ -6170,7 +6170,9 @@ def test_describe_indices(tmp_path, format_version, expected_format_version):
         "{}",
         "{}",
         "{}",
-        '{"path":"x","target_details":{}}',
+        # `$.x` is absent from every document, so the target type was inferred
+        # as utf8, the fallback for an all-null path.
+        '{"path":"x","target_data_type":"utf8","target_details":{}}',
         "{}",
         "{}",
     ]