Skip to content

Test that the contiguous fast paths never read outside the input - #5826

Open
nlohmann wants to merge 6 commits into
developfrom
guard-page-tests
Open

nlohmann wants to merge 6 commits into
developfrom
guard-page-tests

Conversation

@nlohmann

@nlohmann nlohmann commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

This release cycle added fast paths for contiguous input that load several bytes at a time: the bulk string scanner and SWAR searches in string_scan.hpp, the UTF-8 prefix validator, scan_number_bulk_contiguous(), the 8-digit loops of parse_float_significand(), the 8-byte skip_indentation() (#5810), and the BSON/BON8 bulk string readers. The loops look correctly bounded, but nothing proved it: ASan misses over-reads into the slack of the same heap allocation, and fuzzers rarely end an input exactly at a page boundary.

What the test does

tests/src/unit-guard-page.cpp maps three pages and makes the outer two inaccessible (mmap/mprotect on POSIX, VirtualAlloc/VirtualProtect on Windows). Each input is copied so that its last byte is the last readable byte, and again so that its first byte is the first readable byte (scan_number_bulk_contiguous() reads the byte before the cursor). Reading even one byte outside the input then crashes, with or without sanitizers. On other platforms the test compiles to nothing.

It covers:

  • the string_scan.hpp functions directly, for every length 0..64 and several byte patterns (find_string_delimiter only with JSON_USE_SIMDUTF and C++17, where it exists)
  • parse_float_native() (float and double) directly, for every prefix of several long numbers that is a complete token, including halfway cases that go through digit_comparison(). The lexer converts floats from its token buffer, so json::parse never puts a float token at the page boundary.
  • json::parse (with and without ignore_comments), json::accept, and json::sax_parse for every prefix of seeds that end inside ASCII runs, escapes, \u12, 2/3/4-byte and invalid UTF-8, numbers of 8, 16 and 19+ digits, after ./e/e-, literals, comments, and indentation runs of 0..20 spaces with and without the closing bracket
  • from_bson, from_bon8, from_cbor, from_msgpack, from_ubjson, and from_bjdata for every prefix of a document with long strings and keys (so also BSON keys and BON8 strings without their terminator)

Each result (value or exception message) must match parsing the same bytes from a std::string. The test runs in about 1 s with ASan/UBSan.

Validation

  • Passes with clang (C++11/17/20, ASan+UBSan, -Weverything with the CI exclusions), GCC 16 with the flags from cmake/gcc_flags.cmake (Linux, Docker), and clang-tidy.
  • Mutation check: relaxing any one of the bounds by one byte (skip_indentation, find_ascii_copyable_run, find_string_special, both 8-digit loops of parse_float_significand()), or reading two bytes before the cursor in scan_number_bulk_contiguous(), makes the test crash.
  • No over-read was found on develop.

The Windows path is untested locally and is left to CI.

Follow-up

The json_view SIMD scanners (detail/view/simd.hpp, 16-byte loads) should get the same treatment once that stack is merged. The serializer escape table (#5822) writes rather than reads past the end, so it is out of scope.

Breaking changes

None. This only adds a test.


🤖 Generated with Claude Code

Comment thread tests/src/unit-guard-page.cpp Fixed
Comment thread tests/src/unit-guard-page.cpp Fixed
@github-actions

Copy link
Copy Markdown

🔴 Amalgamation check failed! 🔴

The source code has not been amalgamated and/or formatted correctly, or BUILD.bazel, nlohmann_json.natvis, or the macro_builder tables in macro_scope.hpp are out of date. make amalgamate BUILD.bazel natvis fixes all but the tables (see tools/macro_builder/README.md).

📎 A ready-to-apply patch is attached to the failed workflow run as the amalgamation-patch artifact. Download it, then apply it locally from the repository root with:

git apply amalgamation.patch

This does not require installing astyle yourself.

The bulk string scanner, the SWAR searches, the UTF-8 prefix validator,
the 8-digit number loop, the 8-byte indentation skip, and the BSON/BON8
bulk string readers load several bytes at a time from contiguous input.
ASan misses over-reads into the slack of the same heap allocation, and
fuzzers rarely end an input exactly at a page boundary.

The new test copies each input directly in front of, and directly
behind, an inaccessible page (mmap/mprotect on POSIX,
VirtualAlloc/VirtualProtect on Windows), so reading even one byte
outside the input crashes. It covers the string-scan functions directly
and json::parse, accept, sax_parse (with and without comments) and all
binary readers through the public API, with inputs ending at every byte
of strings, escapes, UTF-8 sequences, numbers, literals, indentation
runs, and binary documents. Each result must match parsing the same
input from a std::string.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Under JSON_NOEXCEPTION, the error paths abort instead of throwing, so the
outcomes are now computed with allow_exceptions=false there. Copying the
input with std::copy from char to unsigned char tripped Clang's implicit
conversion sanitizer; std::memcpy avoids the conversion.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
parse_float_eisel_lemire() was removed by #5738, so the test no longer
compiled. The lexer converts floats from its token buffer, so the JSON
section never places a float token at the page boundary; the section now
calls parse_float_native() for float and double on every complete number
token directly and compares the result bit for bit with converting the
same token from a std::string.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
{
static_assert(sizeof(FloatType) <= sizeof(std::uint64_t), "unsupported floating-point width");
std::uint64_t bits = 0;
std::memcpy(&bits, &value, sizeof(value));
The clang sanitizer job (-fsanitize=integer, x86-64 where char is signed)
reported the implicit char -> unsigned char conversions of std::copy() into
the guarded buffer and of the std::string built from the binary inputs.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants