Skip to content

Add benchmarks for SAX parsing, value operations, JSON Pointer, and conversions - #5845

Merged
nlohmann merged 1 commit into
developfrom
benchmarks/value-operations
Oct 21, 2026
Merged

nlohmann merged 1 commit into
developfrom
benchmarks/value-operations

Conversation

@nlohmann

@nlohmann nlohmann commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

The benchmark suite in tests/benchmarks covered only parsing, serialization and the binary formats. This adds 51 benchmarks for the rest of the everyday API, all on the existing test files:

benchmark what it measures inputs
SaxParse sax_parse() with a handler that only counts events; the gap to ParseString is the cost of building the DOM canada, citm_catalog, twitter
Copy, CopyAssign, CopyRing copying a parsed document with the destruction of a copy timed as well: a temporary copy, an assignment over an earlier copy, and replacing the oldest of eight live copies canada, citm_catalog, twitter
CopyOnly, DestroyOnly only the copy or only the destruction, the other half with the timer paused; for telling the halves apart, not for comparing versions (see below) canada, citm_catalog, twitter
Equal == between two separate copies of a parsed document canada, citm_catalog, twitter
Iterate visiting every value with range-based for loops canada, citm_catalog, twitter
Flatten, Unflatten flatten() and unflatten() canada, citm_catalog, twitter
FlattenOrdered flatten() of an ordered_json (canada is left out: it takes minutes with 3.12.0) citm_catalog, twitter
Diff, Patch diff() against the document with every eighth scalar changed, and applying that patch canada, citm_catalog, twitter
JsonPointerParse, JsonPointerToString, JsonPointerAt, JsonPointerContains the JSON Pointer of every scalar: parsed from a string, turned back into a string, looked up with at() and contains() citm_catalog, twitter
GetVectorInt, GetVectorDouble, GetMap get<std::vector<std::int64_t>>(), get<std::vector<double>>(), get<std::map<std::string, json>>() (on the flattened document) signed_ints, floats, citm_catalog
ToJsonStrings/copy, ToJsonStrings/move creating a json array from a std::vector of 100,000 strings, copied or moved, and destroying it again (timed, for the reason given below) generated

The benchmarks use only API that 3.12.0 also has, so releases can be compared on them. While preparing the 3.13.0 performance post, they found the unflatten(), operator==, get<std::map>() and destroy regressions against 3.12.0 that are now fixed in stack #5841.

Copies (and the arrays ToJsonStrings creates) are timed together with a destruction because how a document is freed shapes the heap the next copy allocates from. Since the faster destruction (#5762/#5767), part of the work it saves shows up in the next copy's allocations, so timing the copy alone made develop look slower than 3.12.0 (on glibc, citm_catalog 0.6× in a standalone loop) although copying and destroying together got faster. Medians against 3.12.0, develop header:

macOS (Apple Clang) Linux (GCC 13)
Copy 1.09–1.27× 1.09–1.52×
CopyAssign 1.05–1.26× 1.08–1.58×
CopyRing 1.06–1.27× 1.11–1.57×
CopyOnly 0.95–1.03× 0.83–0.99×
DestroyOnly 1.26–1.55× 1.28–2.12×

tests/benchmarks/README.md lists the new benchmarks.

There is no Accept benchmark here, because the json_view stack (#5627 and below) adds one. Merging this with that stack, #5828 or #5833 conflicts only in added #include lines, benchmark definitions and README rows; keep both sides.

Breaking changes

None. Only the benchmark suite changes; the library and its public API are untouched.

Testing

Built and run against the 3.12.0 header and against develop (Apple Clang, -O3 -flto, C++11), and as part of the 3.13.0 release candidate. The copy benchmarks were also run on Linux (GCC 13, x86-64) against 3.12.0, develop and the release candidate.

This PR was written by Claude Code on behalf of @nlohmann.

🤖 Generated with Claude Code

…onversions

Squashed onto develop cd6b428 (CI fixes from #5860). Original commits:

- Add benchmarks for SAX parsing, value operations, JSON Pointer, and conversions
- Time copies together with their destruction in the benchmarks
- Time destroying the json in ToJsonStrings

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann force-pushed the benchmarks/value-operations branch from 091c314 to c9c933a Compare October 21, 2026 08:41
@nlohmann nlohmann added this to the Release 3.13.0 milestone Oct 21, 2026
@nlohmann
nlohmann merged commit 898e6ee into develop Oct 21, 2026
4 of 167 checks passed
@nlohmann
nlohmann deleted the benchmarks/value-operations branch October 21, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant