Skip to content

Decode \u escapes with a table in the lexer - #5619

Closed
nlohmann wants to merge 4 commits into
json-view/03-string-scanfrom
json-view/04-unicode-escapes
Closed

nlohmann wants to merge 4 commits into
json-view/03-string-scanfrom
json-view/04-unicode-escapes

Conversation

@nlohmann

@nlohmann nlohmann commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Part of the stack for the zero-copy view (#5295). This change speeds up json::parse on its own.

Summary

get_codepoint() read the four hex digits of a \uXXXX escape with four calls to get(). Each call went through a chain of range comparisons.

For contiguous input, the four digits are now decoded with one table lookup per byte (after yyjson's read_hex_u16). The table maps a byte to its value, or to 0xFF for anything that is not a hex digit, so an invalid digit shows in the OR of the four values. The lexer then skips the four bytes and updates the position counters as four get() calls would.

If a digit is invalid, or fewer than four bytes are left, nothing changes and the existing loop runs. Error messages and positions are therefore exactly as before.

Tests

  • The contiguous and the streaming path give the same value, or the same exception message, for:
    • valid escapes and surrogate pairs;
    • truncated input at every distance from the end;
    • an invalid digit at each of the four positions;
    • 3,000 seeded random escapes.
  • The existing lexer, parser, and Unicode tests pass unchanged.

Benchmarks

json::parse in a separate process, best of 5, Apple M1 Max:

file change
twitterescaped.json (every non-ASCII character as \u) −13.6%
citm_catalog, twitter, mesh.pretty, poet, github_events, update-center within ±0.3%

This PR originally also compared literals (true/false/null) with memcmp. That made citm_catalog, twitter, and mesh.pretty 2–3% slower, so it was dropped.

Public API

No change.


Written by Claude Code.

🤖 Generated with Claude Code

@nlohmann
nlohmann added this pull request to stack #5636 September 29, 2026 14:20
@nlohmann
nlohmann removed this pull request from stack #5636 September 30, 2026 13:19
@nlohmann
nlohmann force-pushed the json-view/04-unicode-escapes branch from 1372829 to 6c32ac9 Compare September 30, 2026 13:20
@nlohmann
nlohmann force-pushed the json-view/03-string-scan branch from 046ad7e to 54c0c07 Compare September 30, 2026 13:20
@nlohmann
nlohmann added this pull request to stack #5739 September 30, 2026 13:21
@nlohmann
nlohmann force-pushed the json-view/03-string-scan branch from 54c0c07 to 93b2c99 Compare September 30, 2026 15:34
@nlohmann
nlohmann force-pushed the json-view/04-unicode-escapes branch 3 times, most recently from 14c9a28 to b9fdf44 Compare September 30, 2026 18:06
@nlohmann
nlohmann marked this pull request as ready for review September 30, 2026 18:18
get_codepoint() read the four hex digits of a \u escape with four calls
to get(), each classified by a chain of range comparisons. For contiguous
input, get_codepoint_bulk() now decodes them with one lookup per byte
(hex_codepoint() in string_scan.hpp, after yyjson's read_hex_u16): a
256-entry table maps a byte to its value, or 0xFF for anything else, and
an invalid digit shows in the OR of the four values. It then skips the
four bytes and updates the position counters as four get() calls would.
If a digit is invalid or fewer than four bytes are left, it changes
nothing and the existing loop runs, so errors are reported with the same
message and position as before.

json::parse, best of 5 runs in separate processes (M1 Max): the escaped
twitter.json (every non-ASCII character as \u) -13.6%, all other files
within 0.3%.

Tests compare the contiguous and the streaming path (value or exception
message) for valid escapes, surrogate pairs, truncated and invalid digits
at every position, and 3,000 seeded random escapes.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann force-pushed the json-view/04-unicode-escapes branch from b9fdf44 to d23803f Compare September 30, 2026 18:19
…apes

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
…apes

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann removed this pull request from stack #5739 October 6, 2026 09:26
@nlohmann

nlohmann commented Oct 6, 2026

Copy link
Copy Markdown
Owner Author

The zero-copy view stack (#5295) was rebased onto develop and regrouped from 20 PRs into 10, one squashed commit each. The changes of this PR are now in #5738; its review comments stay here for reference.


Written by Claude Code.

@nlohmann nlohmann closed this Oct 6, 2026
@nlohmann
nlohmann deleted the json-view/04-unicode-escapes branch October 12, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant