Recover from parse errors when parse_error() returns true

[nlohmann]

Addresses #3989 and supersedes #4522.

Problem

The return value of json_sax::parse_error() was documented both as "must return false" (parse_error) and as "whether the parsing should continue" (parse exceptions). The code just passed the value on (return sax->parse_error(...)), so a handler returning true got broken behavior:

• JSON text: parsing stopped anyway, but sax_parse() returned true for invalid input such as [1,2,3,].
• Binary formats: the readers kept reading after the error. CBOR 0x9F (an indefinite-length array) followed by the end of the input looped forever, calling parse_error() each time. A truncated MessagePack array32 made about 4 billion calls.

#4522 let the parser carry on after the error without repairing anything, which produced meaningless events (e.g. key(1) for {1}).

Change

parse_error() returning false stops parsing, as before. Returning true now recovers from the error.

JSON text: repair and continue

Every error is repaired with the smallest local edit:

Mistake	Repair	Example	Result
missing , or :	inserted	[1 2], {"a" 1}	[1,2], {"a":1}
missing value	null for an object key or between commas in an array	{"a":}, [1,,2]	{"a":null}, [1,null,2]
trailing comma	removed	[1,2,]	[1,2]
broken string	an unknown escape stands for the character (as in JavaScript); an invalid \u escape, a lone surrogate, or ill-formed UTF-8 becomes U+FFFD	["a\qb"]	["aqb"]
broken number	the longest valid beginning is kept	[1., 2e+]	[1,2]
unreadable value	null	[1, NaN, tru]	[1,null,null]
number too large	passed as infinity, together with its text	[1e999]	number_float(inf, "1e999")
stray :	removed	["a":1]	["a",1]
member without a key	skipped up to the next , or }	{1:2, "b":3}	{"b":3}
wrong closing bracket	closes the innermost array or object	{"a":[1,2}, "b":3}	{"a":[1,2],"b":3}
input ends too early	all open arrays and objects are closed	{"a":[1,2	{"a":[1,2]}
text before the value	skipped	)]}'{"a":1}	{"a":1}

Guarantees while recovering:

• The SAX events stay balanced, and every key() is followed by exactly one value. A DOM-building handler therefore always gets a complete value.
• Parsing always ends.
• Each token is reported at most once, and the handler can stop at any error by returning false.
• The first error's message and position are exactly those reported without recovery.

Binary formats: repair what is complete, complete what was read

The binary formats have no delimiters to find the next value by, so what can be repaired depends on whether the end of the item with the error is known. RFC 8949, Section 5.3 makes the same distinction for CBOR, and names the two choices a decoder has for an item that is complete but invalid: replace it with an error marker and continue, or stop. That is parse_error() returning true or false.

The item is complete, but cannot be passed on as it is: it is replaced, and reading continues after it.

Mistake	Formats	Repair
tag (with the default cbor_tag_handler_t::error)	CBOR	ignored
simple value other than false, true, null, e.g. undefined	CBOR	null
number too large for a custom number_float_t, like float	all	infinity
char (C) that is not ASCII	BJData, UBJSON	U+FFFD
invalid high-precision number (H)	BJData, UBJSON	longest valid beginning, as in JSON text, or null
high-precision number too large	BJData, UBJSON	infinity, with its text
object key that is not a string	BON8, CBOR, MessagePack	member skipped, like a member without a key in JSON text
element of a type the library does not read (ObjectId, datetime, decimal128, regex, ...)	BSON	null
string without its terminator; document whose size does not match	BSON	kept

• Strings that are not valid UTF-8 are no error for the binary formats: since Follow each binary format's UTF-8 rule: strict writers (CBOR/UBJSON/BJData/BSON), lenient readers #5741 the readers pass them on unchanged by default, and sax_parse uses that default. If a reader is set to reject them (error_handler_t::strict, Add an error_handler parameter for UTF-8 to the binary readers and writers #5746), recovering repairs them as error_handler_t::replace does.
• Tags and simple values are handled as RFC 8949, Section 6.1 suggests for converting CBOR to JSON. With sax_parse's default cbor_tag_handler_t::error (Allow SAX parsing of tagged CBOR #5740), every tag is an error; when recovering, it is ignored.
• Non-string keys are skipped, not turned into strings, so Support for integer keys in the SAX interface when parsing CBOR #2766 and msgpack parser failed to parse null as Map key #3381 stay as decided.

The end of the item is unknown (the input ended, a byte is not a valid type marker, or a size cannot be right):

• Reading stops, and the value read so far is completed: null for a key that waits for its value, and end events for all open containers.
• BSON is the exception, because every document stores its size. The element gets null, the rest of its document is skipped, and reading continues after it.

sax_parse() result

sax_parse() now returns true only if the input was parsed without errors and no event returned false.

Comparison with VS Code's jsonc-parser

The design is modeled on the error-tolerant parser in microsoft/node-jsonc-parser (source checked):

• Taken from it: the lenient string and number handling.
• Differs, deliberately:
  • A missing : is inserted; jsonc-parser drops the member.
  • Missing values become null instead of silently disappearing, which would shift array indices and leave keys without values.
  • Brackets are matched; jsonc-parser has none, so {"a":[1,2}, "b":3} gives {"a":[1,2,"b"]} there.
  • Garbage before the top-level value is skipped; jsonc-parser gives up.
  • Every mistake is reported once, and the handler can stop at any error; jsonc-parser's onError returns void.

Implementation notes

• Parser: sax_parse_internal takes a bool AllowRecovery. parse() and accept() use false, and their report_error returns std::false_type, so all recovery code is dead there even when the call isn't inlined. A parse-only program's __text is 260 bytes smaller than before, and none of the recovery functions are in it.
• Lexer: two new functions, recover_token() (keeps the readable part of a rejected token) and unget_token() (lets a repair hand , ] } or the end of input back to the state evaluation). The hot path of scan_string is unchanged; three surrogate error paths now record how to resume.
• Binary reader: binary_reader gets a bool AllowRecovery template parameter, too.
  • The public sax_parse uses true; from_*() use false, and their repair code is dead through the same std::false_type overloads as in the parser.
  • Each of the 41 error sites calls one of three functions:
    • report_error: the end of the item is unknown, so it stops.
    • report_repairable_error: the item is complete.
    • report_error_repairable_if: it depends on the byte, and the error is still reported in one place when recovery is not compiled.
  • The value read before a final error is completed by the reader itself from its container_stack, as the JSON parser does from its states. A key waiting for its value is noted where reading a value fails; the object and array of a BJData ndarray are tracked by a small counter.
  • skip_items skips a member whose key is not a string without passing events. It uses a heap stack of counts and a head reader per format for CBOR, MessagePack, and BON8.
  • replace_invalid_utf8 (in string_utils.hpp) substitutes maximal subparts (Unicode, Section 3.9).
• DOM parsers: json_sax_dom_parser / json_sax_dom_callback_parser get a default InputAdapterType. The documented json_sax_dom_parser<json> has not compiled since json start/end position implementation #4517. json_sax_dom_parser is also exported from the C++ module, so that the documented way to recover into a json value works.

Performance

Median of 61 runs, before → after. Measured on an array of 40 000 small objects ("objects") and an array of 20 000 nested numeric arrays ("arrays"). The sax_parse handler is a do-nothing virtual counter, so the numbers show the parser loop itself.

	clang (Apple Silicon, native)	GCC 14 (x86-64, emulated)
parse (objects)	31.8 → 31.9 ms	510 → 509 ms
parse (arrays)	14.5 → 14.5 ms	185 → 185 ms
accept (objects)	12.6 → 12.5 ms	166 → 167 ms
sax_parse (objects)	13.1 → 13.5 ms	171 → 162 ms
sax_parse (arrays)	4.6 → 4.8 ms	63 → 53 ms

parse() and accept() are unchanged. sax_parse() with a handler that does nothing moves by a few percent, slower with clang and faster with GCC, so that part is code layout.

Binary formats, clang (Apple Silicon), median of 3×41 runs, develop → this PR, on the same 20 000 objects:

	CBOR	MessagePack	UBJSON	BSON	BON8
from_*()	16.39 → 16.22 ms	15.48 → 15.34 ms	19.26 → 19.21 ms	20.44 → 20.09 ms	17.84 → 17.96 ms
sax_parse	4.91 → 4.89 ms	4.29 → 4.38 ms	5.67 → 5.81 ms	6.73 → 6.67 ms	5.35 → 5.17 ms

A program that only calls from_*() for all six formats is 136 bytes (0.14%) larger in __text.

In unoptimized builds, parse(), accept(), and from_*() do not instantiate the recovery code: the stubs that stand in for it are always inlined, and the parser reaches its recovery helpers only through the recovering instantiation.

Tests

• unit-class_parser.cpp:
  • about 80 repair cases, plus comments, ignore_trailing_commas, strict and non-strict parsing, and NUL bytes;
  • event-level checks and error messages;
  • stopping in the middle of a repair;
  • the same inputs through contiguous, iterator, and stream input adapters;
  • 100 000-error inputs, to guard against quadratic behavior;
  • a sweep over every truncation, deletion, and insertion of []{},:"x\ NUL 0xFF in several documents, checking that events are balanced, error counts are bounded, and valid input is unaffected.
• unit-sax_parse.cpp (new): every sax_parse call instantiates the parser and binary reader that recover from errors, which pushed unit-regression2.cpp and unit-regression3.cpp past the MinGW linker's section limit (32767 COFF sections). So the tests that call sax_parse (these, sax_parse cannot read CBOR that contains tags, including binary values with a subtype written by to_cbor #5676's tagged-CBOR test from unit-regression3.cpp, and the MessagePack SAX tests from unit-msgpack.cpp) have a file of their own, and CONTRIBUTING.md points there.
  • Every binary format truncated at every length, plus a trailing byte, must give exactly one error and a well-formed prefix. The handler stops after 100 errors, so the old CBOR hang now shows up as a failure instead of a timeout.
  • A table of the repairs above, numbers out of range, and errors that stop.
  • A sweep over eight encodings: every byte is replaced by ten values or removed. Events must be balanced, errors bounded, an error reported exactly when from_*() fails, and the first error must be the one from_*() throws.
• Fuzzers: all seven run a recovering, event-checking handler (fuzzer-recovering_checker.hpp) from memory and from a stream. The binary ones also check that it reports an error exactly when from_*() fails. Locally, with ASan/UBSan: 3.6 + 1.4 million runs for JSON text and 21 million for the binary formats found nothing.
• Configurations run locally: C++11/C++20, ASan/UBSan, JSON_STRICT_NUL_HANDLING=1, JSON_DIAGNOSTICS=1, JSON_DIAGNOSTIC_POSITIONS=1; the CI's clang -Weverything and GCC 16 warning flags; the binary format suites (with stubbed test data); make check-amalgamation; docs example output and style check.

Docs

• New page features/parsing/error_recovery.md with the rules, binary behavior, limitations, and an example.
• Updated parse_error, sax_parse, the SAX interface and parse-exceptions pages, and the README.

Breaking changes

• No signatures change.
• Behavior changes only for SAX handlers whose parse_error() returns true, which was documented as "must return false":
  • JSON text is now repaired instead of parsing stopping.
  • The binary readers now repair complete items and read on. Where the end of an item is unknown, they stop instead of reading past the error, and complete the value read so far.
  • sax_parse() returns false after any error; before, it could return true for invalid input.
• For all other handlers, and for parse(), accept(), and from_*(), events, error messages, and positions are unchanged.
• Additive only:
  • a default template argument for detail::json_sax_dom_parser and detail::json_sax_dom_callback_parser;
  • a defaulted bool AllowRecovery template parameter for detail::binary_reader;
  • the module export of json_sax_dom_parser.

🤖 Generated with Claude Code

[nlohmann]

Fixed the two red jobs of run 36536773620 in 792853d:

ci_static_analysis_clang (ci_clang_tidy): clang-tidy reported four errors in code added by this PR:

• binary_reader.hpp:4540 and lexer.hpp:2522: nested conditional operators (readability-avoid-nested-conditional-operator), in the BON8 skip code and in remove_incomplete_utf8_sequence(). Replaced with if chains.
• lexer.hpp:2704: two branches with the same body in recover_string() (bugprone-branch-clone). Merged them into one condition. The short-circuit order is unchanged, so remove_incomplete_utf8_sequence() still runs only for the ill-formed UTF-8 case.

ci_cmake_options (ci_test_noexceptions): test-regression2_* aborted (SIGABRT) in "binary formats repair items whose end is known". The #3989 test compares the first recovered error with the message of the exception that from_cbor() and the other readers throw. Under JSON_NOEXCEPTION that call aborts instead of throwing. I guarded binary_error_message() and the checks that use it with #if !defined(JSON_NOEXCEPTION), as other tests in the suite do. The recovery checks themselves (values, error counts, balanced events) still run in that configuration.

Verification: local clang-tidy on tests/abi/diag/diag_on.cpp reports exactly these four diagnostics before the change and none after it (and nothing new on the changed lines). unit-regression2 passes when built with -DJSON_NOEXCEPTION and run with --no-throw, like the CI target does. unit-regression2, unit-class_parser, unit-class_lexer, and unit-alt-string pass in a normal C++17 build. I regenerated the amalgamation with make amalgamate.

— posted by Claude Code on behalf of @nlohmann