Repository navigation
Conversation
🔴 Amalgamation check failed! 🔴The source code has not been amalgamated and/or formatted correctly, or 📎 A ready-to-apply patch is attached to the failed workflow run as the git apply amalgamation.patchThis does not require installing astyle yourself. |
nlohmann
force-pushed
the
guard-page-tests
branch
from
October 16, 2026 17:27
b6a9ad4 to
732fc11
Compare
The bulk string scanner, the SWAR searches, the UTF-8 prefix validator, the 8-digit number loop, the 8-byte indentation skip, and the BSON/BON8 bulk string readers load several bytes at a time from contiguous input. ASan misses over-reads into the slack of the same heap allocation, and fuzzers rarely end an input exactly at a page boundary. The new test copies each input directly in front of, and directly behind, an inaccessible page (mmap/mprotect on POSIX, VirtualAlloc/VirtualProtect on Windows), so reading even one byte outside the input crashes. It covers the string-scan functions directly and json::parse, accept, sax_parse (with and without comments) and all binary readers through the public API, with inputs ending at every byte of strings, escapes, UTF-8 sequences, numbers, literals, indentation runs, and binary documents. Each result must match parsing the same input from a std::string. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Under JSON_NOEXCEPTION, the error paths abort instead of throwing, so the outcomes are now computed with allow_exceptions=false there. Copying the input with std::copy from char to unsigned char tripped Clang's implicit conversion sanitizer; std::memcpy avoids the conversion. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
parse_float_eisel_lemire() was removed by #5738, so the test no longer compiled. The lexer converts floats from its token buffer, so the JSON section never places a float token at the page boundary; the section now calls parse_float_native() for float and double on every complete number token directly and compares the result bit for bit with converting the same token from a std::string. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
nlohmann
force-pushed
the
guard-page-tests
branch
from
October 17, 2026 06:29
1b74214 to
ef29747
Compare
| { | ||
| static_assert(sizeof(FloatType) <= sizeof(std::uint64_t), "unsupported floating-point width"); | ||
| std::uint64_t bits = 0; | ||
| std::memcpy(&bits, &value, sizeof(value)); |
The clang sanitizer job (-fsanitize=integer, x86-64 where char is signed) reported the implicit char -> unsigned char conversions of std::copy() into the guarded buffer and of the std::string built from the binary inputs. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This release cycle added fast paths for contiguous input that load several bytes at a time: the bulk string scanner and SWAR searches in
string_scan.hpp, the UTF-8 prefix validator,scan_number_bulk_contiguous(), the 8-digit loops ofparse_float_significand(), the 8-byteskip_indentation()(#5810), and the BSON/BON8 bulk string readers. The loops look correctly bounded, but nothing proved it: ASan misses over-reads into the slack of the same heap allocation, and fuzzers rarely end an input exactly at a page boundary.What the test does
tests/src/unit-guard-page.cppmaps three pages and makes the outer two inaccessible (mmap/mprotecton POSIX,VirtualAlloc/VirtualProtecton Windows). Each input is copied so that its last byte is the last readable byte, and again so that its first byte is the first readable byte (scan_number_bulk_contiguous()reads the byte before the cursor). Reading even one byte outside the input then crashes, with or without sanitizers. On other platforms the test compiles to nothing.It covers:
string_scan.hppfunctions directly, for every length 0..64 and several byte patterns (find_string_delimiteronly withJSON_USE_SIMDUTFand C++17, where it exists)parse_float_native()(floatanddouble) directly, for every prefix of several long numbers that is a complete token, including halfway cases that go throughdigit_comparison(). The lexer converts floats from its token buffer, sojson::parsenever puts a float token at the page boundary.json::parse(with and withoutignore_comments),json::accept, andjson::sax_parsefor every prefix of seeds that end inside ASCII runs, escapes,\u12, 2/3/4-byte and invalid UTF-8, numbers of 8, 16 and 19+ digits, after./e/e-, literals, comments, and indentation runs of 0..20 spaces with and without the closing bracketfrom_bson,from_bon8,from_cbor,from_msgpack,from_ubjson, andfrom_bjdatafor every prefix of a document with long strings and keys (so also BSON keys and BON8 strings without their terminator)Each result (value or exception message) must match parsing the same bytes from a
std::string. The test runs in about 1 s with ASan/UBSan.Validation
-Weverythingwith the CI exclusions), GCC 16 with the flags fromcmake/gcc_flags.cmake(Linux, Docker), and clang-tidy.skip_indentation,find_ascii_copyable_run,find_string_special, both 8-digit loops ofparse_float_significand()), or reading two bytes before the cursor inscan_number_bulk_contiguous(), makes the test crash.develop.The Windows path is untested locally and is left to CI.
Follow-up
The
json_viewSIMD scanners (detail/view/simd.hpp, 16-byte loads) should get the same treatment once that stack is merged. The serializer escape table (#5822) writes rather than reads past the end, so it is out of scope.Breaking changes
None. This only adds a test.
🤖 Generated with Claude Code