Skip to content

Add editable json_documents: set, push_back, insert, and erase - #5630

Open
nlohmann wants to merge 19 commits into
json-view/16-view-simdfrom
json-view/19-edit-set
Open

nlohmann wants to merge 19 commits into
json-view/16-view-simdfrom
json-view/19-edit-set

Conversation

@nlohmann

@nlohmann nlohmann commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Adds editable documents to the view: set, push_back, insert and erase on a new Editable variant of basic_json_document; part of the zero-copy view stack for #5295, on top of #5628.

Summary

basic_json_document and basic_json_view get a second template parameter, Editable (false by default). The new aliases json_editable_document, json_editable_view, ordered_json_editable_document and ordered_json_editable_view can change the document they hold:

  • set(view, value) replaces a value;
  • set(object, key, value) assigns a member or adds it, and a null becomes an object. With duplicate keys, the first member is assigned and the others are removed;
  • set(array, index, value) assigns an element;
  • set(json_pointer, value) sets the member or element that a pointer names. "-" or the size of the array appends. Below a null parent, a last token of "-" or digits creates an array (padded with nulls up to the index), as basic_json::operator[] does; any other token creates an object;
  • push_back(array, value) appends, and a null becomes an array;
  • insert(array, index, value) inserts before an element (index <= size()) and returns a view of the new element;
  • erase(object, key) removes all members with the key and returns their number;
  • erase(array, index) removes an element;
  • erase(json_pointer) removes the member or element a pointer names and returns the number of removed values.

A value can be a view (of any document; it is copied), a BasicJsonType value, or anything BasicJsonType can be constructed from. A view of an erased value keeps its last value; views of other values keep referring to them when elements move. Views of editable and read-only documents compare with each other.

How edits are stored

  • The source text is never written.
  • The parsed index never moves. New values and element sequences go to storage that the document owns and that never moves (detail/view/edit_storage.hpp). So views stay valid, and a view keeps referring to its value: after an assignment, it sees the new value.
  • An array or object whose elements change gets its elements in a separate sequence. Entries of that sequence link to the values, which stay where they are. Iterators taken over such a container before its first edit are invalidated.
  • Replacing a container by a scalar does not walk from the root once the parent is linked (linked flag), so loops of set(view, ...) are linear.
  • Copying a nested value into the document is iterative, so a deeply nested value cannot overflow the stack. set encodes the new value into the document's storage before it touches anything reachable from the root; assign() does not leave a half-assigned container if set_moved() throws.
  • The edit arena can grow to 4 GiB - 1 bytes.
  • Views of read-only documents walk the plain node array through detail::view::navigation<false> and compile without any of this. That policy does not change behavior.

Errors

  • They are those of basic_json where the operation corresponds: type_error.305/307/308/309, out_of_range.401/403/405, parse_error.106/109.
  • A view of another document is invalid_iterator.202.
  • Strings and keys are checked for UTF-8 when they enter the document, including those copied from another document, with the same type_error.316 message as basic_json::dump() for the same string. So an editable document only ever holds valid UTF-8, and dump() never throws.
  • Binary values cannot be stored: this is the new type_error.319 (listed in exceptions.md, with insert among the functions that throw it).
  • Edits of 4 GiB or more end with out_of_range.416, as documents of that size do.

Docs and other touched areas

New pages for set, push_back, insert, erase and the four aliases; an "Editing a document" section in the view article; the architecture page describes the edited layout; the banner of json_view.hpp mentions editable documents; .github/labeler.yml and BUILD.bazel list the new files. The lookup, iterator, serializer and number code in detail/view/ is templated on Editable.

Performance

Read-only documents are within about 1% of the previous PR for parse, traversal and dump(): the plain paths do not carry the edit bookkeeping.

The core library (json::parse / dump) is not affected: this PR only adds code to the view.

Tests

unit-json_view_edit.cpp holds a seeded differential test that applies random assignments, member and element changes, inserts, erases (directly and through JSON pointers), copies within and between documents, and pushes, both to an ordered_json_editable_document and to the equivalent ordered_json value, and compares serialized, materialized, compared and read-back results after every edit.

Further tests cover:

  • the errors; views and strings that stay valid while the edit arena grows; the size of a text arena;
  • numbers (NaN, infinities, extremes, number_format::source);
  • nulls that become containers, a replaced root, and JSON pointers below null values;
  • deeply nested values from read-only views, editable views, edited documents and basic_json values;
  • strings of other documents being checked for UTF-8;
  • replacing arrays and objects by scalars;
  • duplicate keys (parsed layout, moved layout, large objects with an index, objects in moved arrays and in new values, values of other documents);
  • documents reused with read().

Public API

No breaking changes. The view API is new in this stack, so nothing here changes a released interface.

  • basic_json_document and basic_json_view get a defaulted template parameter. Code that names them with one argument, or uses the existing aliases, is unaffected.
  • New aliases json_editable_document, json_editable_view, ordered_json_editable_document, ordered_json_editable_view; set, push_back, insert and erase exist only on editable documents (calling them on a read-only document is a static_assert).
  • New exception id type_error.319 (binary values cannot be stored). The basic_json error ids listed above are also thrown by the view's edits; out_of_range.416 now also covers edits of 4 GiB or more.
  • Decisions: set(object, key, ...) with duplicate keys assigns the first member and removes the others, so lookups, iteration and materialize() agree; iterators over a container become invalid when its elements are first edited.

Written by Claude Code.

🤖 Generated with Claude Code

@nlohmann
nlohmann added this pull request to stack #5636 September 29, 2026 14:20
@nlohmann nlohmann changed the title json view/19 edit set Add editable documents to json_view Sep 29, 2026
@nlohmann

Copy link
Copy Markdown
Owner Author

CI fixes for this PR (commit f72fe8d, plus the lower fixes merged in). CI had not run yet; these were found locally with gcc:16 (linux/amd64) and the CI flags:

  • ci_test_gcc: in unit-json_view_edit.cpp, std::mt19937::result_type is std::uint_fast32_t (unsigned long on Linux), so returning it as std::uint32_t failed -Werror=conversion. It is now converted explicitly through a named variable. static_cast<std::uint64_t>(18446744073709551615u) and static_cast<std::int64_t>(-9223372036854775807 - 1) were useless casts there and are now std::numeric_limits<...>::max()/min().
  • ci_test_noexceptions: exception_of_call() catches outside a CHECK_THROWS, so the invalid-UTF-8 checks aborted with JSON_NOEXCEPTION. They are now compiled only with exceptions.

Verified: unit-json_view_edit builds and passes (C++11/C++20 with the CI flags, and a JSON_NOEXCEPTION build run as --no-throw). #5631 got only a propagation merge.

— posted by Claude Code on behalf of @nlohmann

@nlohmann
nlohmann removed this pull request from stack #5636 September 30, 2026 13:19
@nlohmann
nlohmann force-pushed the json-view/19-edit-set branch from 711ddc4 to c675bcd Compare September 30, 2026 13:20
@nlohmann
nlohmann added this pull request to stack #5739 September 30, 2026 13:21
@nlohmann
nlohmann force-pushed the json-view/19-edit-set branch 3 times, most recently from 05b6ce8 to d52db33 Compare September 30, 2026 18:06
@nlohmann nlohmann added the review needed It would be great if someone could review the proposed changes. label Sep 30, 2026
@nlohmann
nlohmann marked this pull request as ready for review September 30, 2026 18:16
@github-actions

Copy link
Copy Markdown

🔴 Amalgamation check failed! 🔴

The source code has not been amalgamated and/or formatted correctly, or BUILD.bazel is out of date.

@nlohmann
nlohmann force-pushed the json-view/19-edit-set branch from d52db33 to 467ccfd Compare September 30, 2026 19:19
Comment thread include/nlohmann/detail/view/edit_storage.hpp Dismissed
Comment thread include/nlohmann/detail/view/edit_storage.hpp Dismissed
Comment thread include/nlohmann/detail/view/edit_storage.hpp Fixed
Comment thread include/nlohmann/detail/view/materialize.hpp Fixed
Comment thread include/nlohmann/detail/view/materialize.hpp Fixed
Comment thread include/nlohmann/detail/view/materialize.hpp Fixed
Comment thread include/nlohmann/detail/view/materialize.hpp Fixed
Comment thread include/nlohmann/detail/view/node.hpp Dismissed
Comment thread include/nlohmann/detail/view/node.hpp Dismissed
Comment thread include/nlohmann/detail/view/materialize.hpp Fixed
@nlohmann
nlohmann removed this pull request from stack #5739 October 6, 2026 09:26
@nlohmann
nlohmann changed the base branch from json-view/18-view-object-index to json-view/16-view-simd October 6, 2026 09:26
@nlohmann
nlohmann force-pushed the json-view/19-edit-set branch from e0c29a1 to a0b9832 Compare October 6, 2026 09:28
@nlohmann nlohmann changed the title Add editable documents to json_view Add editable json_documents: set, push_back, insert, and erase Oct 6, 2026
@nlohmann
nlohmann added this pull request to stack #5768 October 6, 2026 09:29
@nlohmann
nlohmann force-pushed the json-view/19-edit-set branch from a0b9832 to c6ee5a6 Compare October 7, 2026 14:44
basic_json_document gets a second template parameter, Editable
(false by default), plus the aliases json_editable_document,
json_editable_view, ordered_json_editable_document and
ordered_json_editable_view.

Editable documents can change values and structure without
rewriting the source text: set()/push_back() on values, keys,
array indices and JSON pointers; insert() before an array
element; erase() of an object key, array index or JSON pointer.

New values and element sequences go into edit storage that the
document owns and never moves, so views keep referring to their
value across edits and a parsed node never moves. Read-only
documents walk the plain node array and are unaffected.

Strings are checked for UTF-8 on entry, so dump() of an editable
document never throws type_error.316. Binary values cannot be
stored (type_error.319).

A seeded differential test applies random edits to an editable
document and to the equivalent ordered_json and compares both
after every step.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Value-initialize the const std::less in find_parent (clang 3.4/3.6 do not
implement DR 253), and test the Editable template argument through a
function to avoid MSVC C4127.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Counting and copying the nodes of a view or a basic_json value into an
editable document recursed once per nesting level, so that a deeply
nested value overflowed the stack. The four functions now walk the value
with an explicit stack, as materialize() does.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
assign() rewrote the kind, length and extent of a slot before set_moved()
registered its new element sequence. set_moved() can throw
(std::bad_alloc from reserving the bookkeeping vectors) for a slot that
is not moved yet, which left a container without the moved flag that
showed its old children. The bookkeeping is now reserved first
(reserve_moved()), so that nothing after the first write to the slot can
throw. block_of() reserves before it marks anything for the same reason.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
An editable document only holds valid UTF-8, but copy_scalar() copied the
strings and keys of a view of another document unchecked. A document of
a weaker check (or a borrowed text that changed after parsing) could
therefore bring ill-formed UTF-8 into it. The copy is checked now, with
the error that dump() reports for the string; copies within the same
document stay unchecked.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
append_text() doubled the capacity of the arena and rejected the result
if it exceeded 4 GiB, so that an arena of more than 2 GiB could not grow
even though the 32-bit offsets of nodes address 4 GiB - 1 bytes. The
capacity is now clamped to that limit (text_capacity()), and an append
is rejected only if the bytes themselves do not fit.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
set(json_pointer, value) turned a null parent into an object for every
last reference token, so "/a/0" and "/a/-" below {"a":null} made
{"a":{"0":1}}, where basic_json's operator[](json_pointer) makes an
array. A null parent now becomes an array for "-" and for digit tokens
(filled with nulls up to the index) and an object otherwise. An invalid
index is reported before the parent changes.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Replacing an array or object that spans several nodes by a scalar looks
up its parent from the root (find_parent), which is linear in the size
of the document: setting every element of a 20,000 element array took
more than half a second. The parent only has to switch to links once;
afterward the extent of the replaced value no longer matters. Nodes that
an entry of a moved sequence links to are now marked (node_flags::linked),
and assign() skips the lookup for them.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The banner still described a read-only view, and the documentation of
type_error.319 listed set and push_back only, although insert rejects
binary values as well.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
An editable document reads the last member of a repeated key (as the
read-only document does), but set(object, key, value) assigned the first
one: a view taken from object[key] before the call did not show the new
value. It now assigns the last member's value, keeps the key at the
position of its first occurrence (where materialize() puts it), and drops
the other members.

The seeded differential test now also edits documents whose objects
repeat keys and compares every lookup (operator[], at, find, value,
contains, count, and JSON pointers) with the parsed basic_json value.
A targeted test covers duplicates before and after edits that move the
object, in large objects with an index, in moved arrays, and in values
copied from other documents.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Lookups find the first member of a repeated key again, so set(object, key,
value) assigns that member (keeping the key where it is) and drops the
others, as documented before, and a view taken from object[key] shows the
new value. This undoes the code and documentation changes of 5cbb8e6.
Lookups in edited objects (navigation of editable documents) stop at the
first match, like those of parsed objects.

The tests that commit added expect the first member from lookups now. In the
seeded differential test, the documents with repeated keys repeat them after
the real members; the reference holds the first members, which the edits
address and the lookups are compared with, while materialize() is compared
with what parse() makes of the document's dump.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Silence bugprone-casting-through-void for the SSE loads (a reinterpret_cast
would trip -Wcast-align=strict), use auto for a cast initialiser, add
parentheses to a mixed expression, and name bugprone-std-namespace-modification
in the NOLINTs of the tuple_size/tuple_element specialisations.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
(cherry picked from commit 0effbaa)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), and the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), plus <version> for
std::nullptr_t, and removing <cstddef> and edit_storage.hpp.

- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
  json_view.hpp only includes json.hpp, so including them here would duplicate
  their definitions
- drop edit_storage.hpp, which edit.hpp includes

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
clang-tidy 22 checks the headers with -std=gnu++17 as well, where it asks
for a transparent comparator and std::make_unique. The regions map uses its
default comparator, and the allocation of the edit state keeps new (the
library is C++11) with a NOLINT.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann force-pushed the json-view/19-edit-set branch from f870ade to a312ebe Compare October 18, 2026 02:27

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI CMake documentation L review needed It would be great if someone could review the proposed changes. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants