Fix StringLiteral source spans covering only the closing quote - #3
Merged
Conversation
Every StringLiteral's position spanned a single `"` instead of the whole literal, and so did any node wrapping one -- a PositionalArgument, a vector element. A string is matched by SEVERAL lexer rules (opening quote, content runs, escape sequences, closing quote), and YY_USER_ACTION calls updateLocation() on every one of them, so tokenStart always describes the most recent match. Building the token's location from currentTokenLoc() in the closing-quote rule therefore described just that character. STRING is the only multi-rule token, which is why nothing else was affected. ParserDriver::stringStart now records the opening quote's position when that rule fires, and stringTokenLoc() spans from there to the current end. This mattered downstream: a consumer slicing source by these offsets could land inside string CONTENT. BelfrySCAD's reformatter rewrote a comma inside "h,l,height,length" in BOSL2's isosurface.scad, silently changing the string's value -- found because the reformat was checked for AST-shape preservation across the corpus rather than eyeballed. Tests cover the plain case, empty strings, embedded escaped quotes, escape sequences, commas inside strings, the spans of enclosing nodes (how the bug actually did damage), strings in vectors, and a multi-line string still reporting its OPENING line rather than its closing one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every
StringLiteral's position spanned a single"instead of the whole literal — and so did any node wrapping one:Cause
A string is matched by several lexer rules — opening quote, content runs, escape sequences, closing quote — and
YY_USER_ACTIONcallsupdateLocation()on every one of them, sotokenStartalways describes the most recent match. Building the token's location fromcurrentTokenLoc()in the closing-quote rule therefore described just that one character.STRINGis the only multi-rule token, which is why nothing else was affected.Fix
ParserDriver::stringStartrecords the opening quote's position when that rule fires;stringTokenLoc()spans from there to the current end.Why it mattered
A consumer slicing source by these offsets could land inside string content. BelfrySCAD's reformatter rewrote a comma inside
"h,l,height,length"in BOSL2'sisosurface.scad, silently changing the string's value.That was found only because the reformat was checked for AST-shape preservation across the whole corpus rather than eyeballed — the output looked entirely plausible.
Tests
Four new (630 total): the plain case, empty strings, embedded escaped quotes, escape sequences and commas inside strings; the spans of enclosing nodes, which is how the bug actually did damage; strings inside vectors; and a multi-line string still reporting its opening line rather than its closing one.
🤖 Generated with Claude Code