1.2.0: fix streaming edge cases, linear-time scanner, packaging (no behaviour changes for valid input) - #10
Merged
Merged
Conversation
Behaviour-preserving release. Every input 1.1.0 parsed successfully parses to the same value; tests/test_compat_1_1_0.py runs the frozen 1.1.0 parser next to the new one over every prefix of a corpus, and tests/fuzz_compat_1_1_0.py does the same for random documents. Fixed - Exponent numbers (1e5, 2.5E-3) inside an incomplete container raised. - Strict mode returned "" for an unterminated string ending in an incomplete escape, losing already-streamed text (issue #8). - Strict mode returned a lone surrogate when cut inside an escaped surrogate pair. - JSON5 mode raised on partial literals and exponents, and mis-handled a comment that had only streamed its first '/'. - JSON5 string decoding depended on whether the optional json5 package was installed. - bytes input crashed the fallback parser; a leading BOM raised. - JSONParser.strict / on_extra_token / last_parse_reminding are readable and assignable again; 0.x method names are callable again. Changed - Index-based scanner: parse time is linear in input size (900 KB partial document: ~1 s -> ~60 ms). - Literals that are not a prefix of true/false/null now raise like json.loads instead of being accepted. - _JSON5Parser subclasses _JSONParser instead of duplicating it. - pyproject.toml, requires-python >= 3.8, classifiers, py.typed, full type hints; CI on 3.8-3.14 with and without json5. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A backward-compatible 1.2.0. Every input that 1.1.0 parsed successfully parses to the same value here. That is not a promise but a test:
tests/legacy_1_1_0_parser.pyis the 1.1.0 parser frozen verbatim,tests/test_compat_1_1_0.pyruns it next to the new parser over every prefix of a document corpus in both strict modes, andtests/fuzz_compat_1_1_0.pydoes the same over random documents (1.08 million prefixes checked locally, zero unexplained differences). The only allowed differences are the fixes below.Fixed
[1e5,{"a": 2.5E-3raisedJSONDecodeError. An exponent that has no digits yet (1e,1e-) is held back until it does.{"a":"foo\u00returned{"a": ""}; now{"a": "foo"}. Only the unfinished escape is held back (Throws when in middle of escaped token #8).\ud83dand\ude00, the result raisedUnicodeEncodeErroron encode. The high half is now held back until its partner arrives.{"a": tr,[fals,[Inf) and exponents raised, and a comment that had only streamed its first/was treated as a stray token.json5was installed. Escapes now decode identically either way; the optional package is only a fast path for complete documents.bytes/bytearrayinput crashed the fallback parser; a leading BOM raised.JSONParser.strict,.on_extra_token,.last_parse_remindingare readable and assignable again (assigningstricton 1.x was silently ignored), and the 0.x method names (parse_string,parse_number, ...) are callable again.Changed
parse()in 1.1.0, about 60 ms here. Structure and all quirks of the old control flow are preserved on purpose.true/false/nullnow raise likejson.loads([trap]returned[True]in 1.1.0). Prefixes such as[t,[trustill parse. In valid JSON nothing buttruecan start witht, so returningTruefor a prefix is safe; the only change is that garbage is rejected instead of guessed._JSON5Parsersubclasses_JSONParserinstead of duplicating it.pyproject.tomlreplacessetup.py:requires-python >= 3.8, classifiers,py.typed, full type hints (mypy--strictclean).json5extra.on_extra_token.Deliberately not changed
on_extra_tokenstill prints to stdout. Worth moving towarningsin a 2.0.test_string_relaxed_returns_raw_content_when_incompletepins this.Verification
json5.--strictclean.py.typedand the compatibility oracle ship.Suggested release steps
v1.2.0, publish. Nothing in the public API changed, so a minor bump is right.🤖 Generated with Claude Code