Repository navigation
feat(tests): comprehensive RFC 3987 §2.2 grammar test data - #42
Closed
jkowalleck with Copilot wants to merge 13 commits into
Closed
jkowalleck with Copilot wants to merge 13 commits into
jkowalleck with Copilot wants to merge 13 commits into
Conversation
- Populate all 30+ previously empty arrays in valid_syntax.json with RFC-driven test cases covering every grammar production in RFC 3987 §2.2: ucschar, iunreserved, sub_delims, iuserinfo, h16, dec_octet, ls32, ipv6address (all 9 forms), unreserved, ipvfuture, ip_literal, ireg_name, ihost, port, iauthority, ipchar, isegment, ipath_abempty, isegment_nz, ipath_absolute, ipath_rootless, ipath_empty, ihier_part, iprivate, iquery, gen_delims, ifragment, isegment_nz_nc, ipath_noscheme, ipath, irelative_part, irelative_ref, reserved, scheme, pct_encoded, ipv4address - Add invalid test cases to 5 previously empty sections in invalid_syntax.json: isegment_nz_nc, ipath_noscheme, ipath, irelative_part, irelative_ref - Remove RFC-wrong entry: a new IRI entry added by the script using literal ASCII asterisks '******example.com:...' was invalid (asterisk is not a valid scheme character; scheme must start with ALPHA). Removed. - All existing entries preserved (no deduplication). - Source of truth: RFC 3987 §2.2 grammar exclusively. - No tests were run per task instructions.
…t data Re-serialize both JSON files with ensure_ascii=True so all non-ASCII characters (especially invisible private-use area chars like U+E000 and ucschar boundary values like U+00A0) appear as explicit \\uXXXX escape sequences. This makes the intent unambiguous in JSON source review.
The previous ensure_ascii=True commit introduced an ambiguous \u10ffFF representation for the existing iprivate test entry (U+10FF followed by literal 'FF'). Revert to raw UTF-8 (ensure_ascii=False) which matches the original file style and avoids this ambiguity. The 'reason' field for each entry explicitly identifies codepoints by name/number.
In isegment_nz_nc, ipath_noscheme and ipath invalid entries, update reason text to say 'ASCII space (U+0020)' rather than just 'space', to avoid confusion with U+00A0 NO-BREAK SPACE which IS valid as a ucschar within iunreserved and thus allowed in path segments.
Use explicit \uXXXX notation for U+00A0 (NO-BREAK SPACE) in value fields so they are visually distinct from ASCII space (U+0020). Also escape all private-use area characters (BMP E000-F8FF, Plane 15 F0000-FFFFD, Plane 16 100000-10FFFD) for clarity. Visible characters (CJK, Arabic, Hebrew etc.) remain as raw UTF-8.
Normalise lowercase \uXXXX to uppercase \uXXXX throughout both JSON files for consistency with the uppercase codepoint notation used in reason strings (e.g. "U+00A0").
Add "NO-BREAK SPACE" name to U+00A0 references in the ipath_noscheme and ipath invalid-space entry reason strings, matching the explicit style used elsewhere in the file.
Copilot
AI
changed the title
[WIP] Update RFC3987 syntax test-data JSON files for grammar compliance
feat(tests): comprehensive RFC 3987 §2.2 grammar test data
Jul 20, 2026
jkowalleck
requested changes
Jul 20, 2026
| yield pytest.param(term, example, id=f"{term}-{example['value']}") | ||
|
|
||
|
|
||
| def test_fixture_json_files_are_ascii_only() -> None: |
Collaborator
There was a problem hiding this comment.
@copilot remove this test. we dont wnat this.
futhermore, add .gitattributes and .editor conf file in that _data dir, and make sire that the files in there are treated as binady by git, and that we wnat the JSON files to be ascii
add a comemnt descibing that this is for compatibility purposes here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both JSON test-data files were sparse —
valid_syntax.jsonhad only 3 of 39 sections populated, andinvalid_syntax.jsonhad 5 empty sections. RFC 3987 §2.2 is the sole source of truth; no alignment with repo implementation was attempted.valid_syntax.json (3 → 39 populated sections, 352 total entries)
s:, boundary alpha/digit/+/-/.charsv1.testIPvFuture%xA0,%xD7FF,%xF900,%xFDCF,%xFDF0,%xFFEF, plane ranges)%00,%FFinvalid_syntax.json (5 empty sections filled, 545 total entries)
a:b); bare colon; space?; bare#(reserved delimiters not allowed in path)//before authority; invalid authority chars#; query-only without?; space in pathEncoding
E000–F8FF,F0000–FFFFD,100000–10FFFD) stored as explicit\uXXXXescapes (uppercase) to prevent visual ambiguity with ASCII space and other invisible characters.