Repository navigation
Diff deeply nested values without recursing per nesting level - #5548
Conversation
f85ff73 to
f9247cd
Compare
f9247cd to
ced82e8
Compare
ced82e8 to
f4d0822
Compare
f4d0822 to
c4d7239
Compare
c4d7239 to
fd39377
Compare
diff() descended into both values once per nesting level, and compared them with operator== on every level on the way, which recurses as well. Values nested deeply enough - 25,000 levels on an 8 MiB stack - exhausted the call stack and terminated the process, although parse() accepts them without complaint. On such a chain the per-level comparisons and path strings also made diff() quadratic in time and memory. Both the recursion and operator== only descend as far as the source is nested. So diff() first checks, recursing at most diff_depth_limit() (128) levels, whether the source is nested more deeply than that. If not - all but a vanishing minority of values - the recursive algorithm diffs it exactly as before, now as diff_recursively(). Otherwise diff_iteratively() walks the two values on an explicit stack, emitting the same operations in the same order. It does not compare arrays and objects with operator== up front (equal ones yield no operations anyway), keeps the path in one buffer instead of a new string per level, and hands every subtree that is not nested too deeply back to diff_recursively(), so equal parts are still skipped quickly. The check costs one pass over the source. On a 3,000-object document that is about 30% of diffing two equal values (which is just an operator== call), about 10% of diffing values that differ in a few places, and noise when arrays change length. Once operator== no longer recurses (#5390), the check can go. Tests check that the patch reproduces the target at every depth up to 300, for json and ordered_json, including reordered members. They also check the exact operation for a difference deep inside, and diff values nested 100,000 levels deep. Fixes #5393 for diff(). Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC's -Weffc++ (an error in CI) asks a class with pointer members, a user constructor and a non-trivial destructor to declare its copy constructor and copy assignment; diff_frame's vector and basic_json members make its destructor non-trivial. Declare all five as defaulted, which also satisfies clang-tidy's special-member-functions check. Leave their exception specifications implicit: GCC 4.8 rejects an explicit one that differs from the implicit one, as it does for flatten_task in #5517. The converting constructor cannot throw, and is now declared noexcept for GCC's -Wnoexcept, which flags the emplace_back() under C++26 otherwise. The struct also moves from diff_iteratively() into the class, like dump_frame in the serializer. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
diff_depth_limit() is gone in favor of detail::recursion_depth_limit(). Signed-off-by: Niels Lohmann <mail@nlohmann.me>
fd39377 to
08e30ec
Compare
Checking every depth up to 300 made test-json_patch exceed the 1500 s ctest timeout in ci_test_valgrind. Check the depths up to 16, those around the recursion limit of 128, and 300 instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The braces are kept for GCC's -Weffc++, as in json_sax.hpp. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Now that operator== no longer recurses (#5390), diff() can keep its per-level equality shortcut all the way down. It diffs recursively for the first detail::recursion_depth_limit() levels, as merge_patch() does, and hands anything deeper to diff_iteratively(). The nesting_exceeds() scan, which cost about 30% on equal documents, is gone, and diff() is on par with develop again. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
diff_recursively built every patch operation in place from initializer lists. Unoptimized builds give each of those temporaries its own stack slot, so every level of the bounded descent cost kilobytes of stack (about 6 KB with clang -O0), and the 128 recursive levels overflowed the 1 MB stack of MSVC Debug in the "deeply nested values" test. The operations and the key comparison of two objects are now built by separate functions, which diff_iteratively shares, and both diff functions append to one result instead of returning a patch per level that the caller copies. With clang -O0, diffing values nested 300 levels deep now peaks at about 190 KB of stack instead of 880 KB. Since diff() now owns the only returned value, clang's -Wnrvo no longer reports the returns of diff_recursively, which alternated between the local patch and diff_iteratively's result. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The loop in diff_iteratively held a reference to the top frame, which enter() invalidates when it pushes and the end of the loop invalidates when it pops. Nothing used it afterwards, but a later change could. As in the other iterative walks, the members the loop reads are now copied out as constants and the ones it advances are changed through stack.back(). The frame as a whole is not copied: it holds the common keys and the "add" operations of an object. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
#5639 found that This branch has the same condition (currently at - if (common_keys_source_order == common_keys_target_order && new_keys_form_suffix)
+ // Only an object type that keeps its members in insertion order,
+ // such as nlohmann::ordered_map, can need reordering: patch() appends
+ // a new member at the end of such an object. Any other object type
+ // places its members itself - std::map in key order, a hash map in an
+ // order its operator== ignores - so a member-by-member diff always
+ // reproduces target there.
+ if (!detail::is_ordered_map<object_t>::value
+ || (common_keys_source_order == common_keys_target_order && new_keys_form_suffix))
{
// fast path: order of common keys already matches (or the
// object_t's iteration order does not depend onThis comment was written by Claude Code on behalf of @nlohmann. |
Conflicts: - include/nlohmann/json.hpp: kept the PR's diff_recursively()/ diff_object_keys() split and re-applied develop's #5691 (fast path for every object type except an insertion-ordered one; target key order only tracked for such types) inside diff_object_keys(), so both the recursive and the iterative walk get it - single_include/nlohmann/json.hpp: regenerated with make amalgamate - tests/src/unit-json_patch.cpp: kept both new test cases (deeply nested diff from the PR, #5639 regression from develop) Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
Merged develop (795ce4e). Conflict in — posted by Claude Code on behalf of @nlohmann |
* Flatten deeply nested values without recursing per nesting level json_pointer::flatten() called itself once per nesting level, so flatten() on a value nested deeply enough exhausted the call stack. #5547 and #5548 fixed merge_patch() and diff() from #5393, but flatten() was left out. flatten() now walks the value with an explicit stack and keeps the path in one buffer that grows and shrinks with it. It has a single code path and no depth limit: the old version built a new path string per child, so the iterative one is no slower on shallow values and much faster on deep ones. The output, including the order of an ordered_json result, is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Construct flatten frames in place Give the frame a constructor so both call sites can use emplace_back, as suggested in the review; index starts at 0 for every frame. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
What & why
diff()descends into both values once per nesting level, and compares them withoperator==on every level on the way, which recurses as well. Values nested deeply enough terminate the process withSIGSEGV: 25,000 levels on an 8 MiB stack.parse()accepts such values without complaint. On such a chain the per-level comparisons and per-level path strings also madediff()quadratic in time and memory. Reported in #5393.Part of the same series as #5546, see the table there.
The change
Like
merge_patch()anddump(),diff()now bounds its descent with a depth count:detail::recursion_depth_limit()(128, shared with the other operations) levels: the recursive algorithm diffs them exactly as before, now nameddiff_recursively()and given a depth parameter. It keeps its per-leveloperator==shortcut, which is safe since Fix stack overflow and exponential runtime when comparing nested values #5390 made comparisons non-recursive.diff_iteratively()walks the rest on an explicit stack, emitting the same operations in the same order. It doesn't compare arrays and objects withoperator==up front, since doing that on every level would be quadratic in the depth and equal ones yield no operations anyway. It keeps the path in one buffer instead of a new string per level.An earlier version of this PR scanned the source first (
nesting_exceeds) becauseoperator==still recursed; that scan and its cost are gone.Cost
None measurable. Median of 15 runs, clang
-O2:develop(The scan-based version took 0.24 / 0.38 / 0.86 ms on the equal documents.)
Verification
developon 12,042 diffs (8,524 non-empty) is identical, in C++11 and C++20. It covers randomjsonandordered_jsonpairs with NaN, discarded values,1vs1.0, binary values, reordered members, arrays of different lengths, and a custom root path with escaping, plus chains crossing the bound up to 400 levels deep.developon 800 randomjsonandordered_jsonpairs with chains 100–180 levels deep, 110 of whose operations lie deeper than 128 levels: byte-identical patches, and every patch reproduces its target.developcrashes at 100,000.jsonandordered_json, including reordered members). A difference deep inside must be exactly one replace operation, with the right path.make amalgamate+make check-amalgamation; no new findings from the clang-tidy checks CI runs, or-Weverything.Public API impact
No breaking changes.
diff()'s signature and results are unchanged; the helpers are private.make amalgamate.🤖 Generated with Claude Code