Add images of json_documents: save() and load()

[nlohmann]

Part of the stack for the zero-copy view (#5295). This PR adds images to json_document: save() writes a document so that load() can read it back without parsing. Experimental: the image format is versioned, but it may change before it is declared stable.

Summary

An image is a document stored so that loading it needs no parsing:

• std::vector<std::uint8_t> save() const writes the node index, the text, and the decoded strings.
  • An edited document is written in its current state, in document order, without links. Floats that are not finite become null, as in dump().
  • Images are deterministic: the same document gives the same bytes, whatever BasicJsonType it was parsed with.
• static load(image, check = image_check::full) reads an image written by save().
  • The overloads take a pointer and size, a const std::vector<std::uint8_t>& (both borrowed: the image must outlive the document), or a std::vector<std::uint8_t>&& (kept without a copy; owns_source() is then true).
  • The text and the decoded strings stay in the image. The nodes are copied, so that they are aligned and edits can change them. The hash indexes of large objects are rebuilt.
  • A loaded document can be edited and saved again. shrink_to_fit() leaves the decoded strings in the image.
  • If load(std::vector&&) fails, the vector is left empty (documented in load).
• image_check says how much of an image load() trusts:
  • full checks everything the parser guarantees: structure, bounds, strings (valid UTF-8; source strings without quotes, backslashes, or control characters), and numbers (well-formed, and matching the stored values). Reading and serializing a checked image is safe and yields valid JSON.
  • bounds checks structure and bounds only. Reading and serializing stay safe, but a crafted image can yield invalid UTF-8, strings that serialize to invalid JSON, or numbers that differ from their text.
  • none is for images from a trusted source only; a damaged image is undefined behavior.

Layout (little-endian only):

1. a 64-byte header ("NJVI", version 1, the counts, and reserved fields that must be zero);
2. the nodes;
3. the text and a NUL;
4. the decoded strings and a NUL.

The idea comes from zero-copy formats such as FlatBuffers and YaFF, and the check follows the idea of FlatBuffers' Verifier. No code is taken from either.

The full check is linear

The structure and the bounds are checked node by node. The contents of strings and float tokens are checked afterwards, once for each distinct range. Nodes that share a value (copies within a document) share the whole range; ranges that overlap without being identical are rejected. Without this, many nodes pointing at the same long string made the check quadratic in the size of the image. load() releases its list of large objects after building the indexes.

Changes to existing code

• Float conversion. Supporting bounds needed two changes:

  • the general path (float_token_layout) checks the token's grammar and locates the point and the exponent itself, instead of trusting the layout the node records; a token that is not a JSON number yields 0;
  • the exponent loop of the layout path cannot overflow on non-digits and does not read beyond the token.

  Parsed and edited documents only have well-formed tokens, so for them nothing changes.

• Serializer. write_escaped validates non-ASCII bytes. Strings of a document are valid UTF-8, so this only matters for a damaged image loaded with bounds: dump() then throws what basic_json::dump() throws (type_error.316).

• Editable documents do not take over such a string: set, push_back and insert throw type_error.316 when copying it from a view of the loaded document, and leave the document unchanged.

Errors

• A malformed image or a failed check throws the new parse_error.116: "invalid json_document image: …" ("too short", "unknown format", "sizes out of range", or "the check failed").
• Saving a discarded document, or using images on a big-endian target, throws the new type_error.320.
• Images of 4 GiB or more (nodes, text, or decoded strings individually) throw out_of_range.416, as for documents.

Documentation

Pages for save and load (with examples), the image_check member type, an "Images" section in the feature page, the layout and the rule "a change of the node layout raises image_version" in the architecture page, the new exception ids, and a complexity section for load (linear in the image; full adds sorting the ranges of strings and float tokens).

Performance

load() of a document's image vs. json_document::parse() of its text.

Measured on Apple M1 Max (Apple clang -O2), µs, best of 3 rounds of 200 runs:

file	parse	load, full	load, bounds	load, none
canada	991	2325	619	105
citm_catalog	391	587	243	59
twitter	183	318	86	24
poet	729	1115	192	43
twitterescaped	252	362	87	24
update-center	213	435	118	35

• none is 6–17x faster than parsing, and bounds is 1.6–3.8x faster.
• full scans every string and number token as the parser does, on top of the structure check, so it costs 1.4–2.4x a parse (canada, with its many float tokens, is the worst case). It is meant for images from untrusted places, where the alternative is parsing the text again.

Tests

unit-json_view_image.cpp:

• Round trips of small documents, test files, and large objects (including duplicate keys and colliding keys), with every check. The loaded document must dump, materialize, and save exactly as the original. Images are deterministic.
• Edited documents with every kind of edit (strings and keys in the edit arena, positive number_integer values, floats, NaN and infinity, moved and new arrays and objects), saved, loaded, and edited and saved again. The targets of links never reach an image.
• Ownership: borrowed and owned images, read() after load(), and shrink_to_fit() on a loaded document. Lookups work after load(), and loading releases the list of large objects.
• All errors, and one corruption for every rejection branch of the check (kinds, flags, bounds, structure, strings, number grammar, values, and ranges). Nodes that share a range are accepted; partially overlapping ranges are rejected.
• Invalid UTF-8 from a bounds or none image is rejected when it is copied into an editable document (type_error.316), as a value, as a key, and nested.
• 6,000 seeded random corruptions of two documents, each loaded with full and with bounds. Each load must either be rejected or be read safely under ASan/UBSan; with the full check, the dump must also parse to the materialized value.
• The big-endian error is tested on big-endian targets. The tests avoid GCC's -Wuseless-cast.

fuzzer-json_view_image.cpp (documented in tests/fuzzing.md) uses each input both as an image and as a JSON text. As an image, load() must reject it with parse_error.116 or read it safely (with full, the dump must parse to the materialized value and save() must give the input back). As a text, the image of the parsed document must load with every check and dump to the same text.

Public API

No breaking changes. Images are new:

• new members basic_json_document::save, load (three overloads), and the member type image_check (full, bounds, none);
• new exception ids parse_error.116 and type_error.320, and out_of_range.416 is also thrown for images of 4 GiB or more;
• the image format is experimental and tied to the library build that wrote it; load() rejects other format versions (parse_error.116, "unknown format");
• load(std::vector&&) leaves the vector empty if it throws;
• dump() of a document loaded with bounds or none can throw type_error.316 if the image was damaged. This cannot happen for parsed documents or for images checked with full.

Nothing in json.hpp changes.

---

Written by Claude Code.

🤖 Generated with Claude Code

[nlohmann]

CI fixes for this PR (commit 58679a7, plus the lower fixes merged in). CI had not run yet; these were found locally with gcc:16 (linux/amd64) and the CI flags:

• ci_test_gcc: in unit-json_view_image.cpp, static_cast<std::size_t>(header_field(image, 8)) was a useless cast on Linux x86-64, and rng() returned std::mt19937::result_type as std::uint32_t (-Werror=conversion). Both now cast named variables.
• ci_test_noexceptions: "json_view images: errors", "check" and "damaged images" test the exceptions of load()/save() and catch outside a CHECK_THROWS, which aborts with JSON_NOEXCEPTION. Those tests and their helpers (exception_of, check_failed, load_result, corrupted) are now compiled only with exceptions.
• clang 3.6: const json_document empty; is now value-initialized.
• tests/src/fuzzer-json_view_image.cpp was reformatted with the pinned astyle. The "check" job runs astyle over tests/.

Verified: unit-json_view_image builds and passes (C++11/C++20 with the CI flags, and a JSON_NOEXCEPTION build run as --no-throw). #5633 got only a propagation merge.

— posted by Claude Code on behalf of @nlohmann

[github-actions]

🔴 Amalgamation check failed! 🔴

The source code has not been amalgamated and/or formatted correctly, or BUILD.bazel is out of date.

📎 A ready-to-apply patch is attached to the failed workflow run as the amalgamation-patch artifact. Download it, then apply it locally from the repository root with:

git apply amalgamation.patch

This does not require installing astyle yourself.