Skip to content

Fix StringLiteral source spans covering only the closing quote - #3

Merged
revarbat merged 1 commit into
mainfrom
fix-string-literal-span
Aug 6, 2026
Merged

Fix StringLiteral source spans covering only the closing quote#3
revarbat merged 1 commit into
mainfrom
fix-string-literal-span

Conversation

@revarbat

@revarbat revarbat commented Aug 6, 2026

Copy link
Copy Markdown
Member

Every StringLiteral's position spanned a single " instead of the whole literal — and so did any node wrapping one:

a = "txt";          StringLiteral      span='"'      (should be '"txt"')
f("x,y", b);        PositionalArgument span='"'      (should be '"x,y"')

Cause

A string is matched by several lexer rules — opening quote, content runs, escape sequences, closing quote — and YY_USER_ACTION calls updateLocation() on every one of them, so tokenStart always describes the most recent match. Building the token's location from currentTokenLoc() in the closing-quote rule therefore described just that one character.

STRING is the only multi-rule token, which is why nothing else was affected.

Fix

ParserDriver::stringStart records the opening quote's position when that rule fires; stringTokenLoc() spans from there to the current end.

a = "txt";                    -> "txt"
a = "";                       -> ""
a = "with \"quote\" in it";   -> "with \"quote\" in it"
x = ["a,b","c"];              -> "a,b"  /  "c"

Why it mattered

A consumer slicing source by these offsets could land inside string content. BelfrySCAD's reformatter rewrote a comma inside "h,l,height,length" in BOSL2's isosurface.scad, silently changing the string's value.

That was found only because the reformat was checked for AST-shape preservation across the whole corpus rather than eyeballed — the output looked entirely plausible.

Tests

Four new (630 total): the plain case, empty strings, embedded escaped quotes, escape sequences and commas inside strings; the spans of enclosing nodes, which is how the bug actually did damage; strings inside vectors; and a multi-line string still reporting its opening line rather than its closing one.

🤖 Generated with Claude Code

Every StringLiteral's position spanned a single `"` instead of the whole
literal, and so did any node wrapping one -- a PositionalArgument, a
vector element.

A string is matched by SEVERAL lexer rules (opening quote, content runs,
escape sequences, closing quote), and YY_USER_ACTION calls
updateLocation() on every one of them, so tokenStart always describes the
most recent match. Building the token's location from currentTokenLoc()
in the closing-quote rule therefore described just that character.
STRING is the only multi-rule token, which is why nothing else was
affected.

ParserDriver::stringStart now records the opening quote's position when
that rule fires, and stringTokenLoc() spans from there to the current
end.

This mattered downstream: a consumer slicing source by these offsets
could land inside string CONTENT. BelfrySCAD's reformatter rewrote a
comma inside "h,l,height,length" in BOSL2's isosurface.scad, silently
changing the string's value -- found because the reformat was checked
for AST-shape preservation across the corpus rather than eyeballed.

Tests cover the plain case, empty strings, embedded escaped quotes,
escape sequences, commas inside strings, the spans of enclosing nodes
(how the bug actually did damage), strings in vectors, and a multi-line
string still reporting its OPENING line rather than its closing one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@revarbat
revarbat merged commit 0789702 into main Aug 6, 2026
11 of 24 checks passed
@revarbat
revarbat deleted the fix-string-literal-span branch August 6, 2026 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant