Resolve escape sequences in string literals - #86
Merged
Conversation
Nothing did. The parser hands back a literal's source text with its backslashes intact, deliberately -- that is what lets a StringLiteral reproduce the source it came from, and what pretty-printing round-trips -- but no one cooked it afterwards, so every escape reached scripts raw. "a\nb" was four characters, a backslash and an 'n' among them, rather than three. The same for \t, \r, \" and \\. unescapeStringLiteral() resolves them where the literal is evaluated, which is both evaluation paths: the tree walk in expr_eval and the constant pool in bytecode_compiler. It returns the input untouched when there is no backslash in it, which is nearly every string, so the common case allocates nothing. A backslash escaping the end of a line contributes nothing at all: the string carries on with no break in it. A CRLF goes whole, unlike the reference implementation, which drops only the LF and leaves the CR in the value -- a stray control character in any string continued in a file written on Windows. A lone CR is still an ordinary escaped character. An unknown escape keeps its character and drops the backslash, matching the reference. Also picks up the parser's fix for the same feature, where a backslash before a newline matched no lexer rule and was printed to stdout by flex's default ECHO. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Windows CI turned red once escapes started being resolved: 43 import, surface and DXF/SVG tests build a script by interpolating a temp path into a string literal, and std::filesystem::path::string() hands back backslashes there. `C:\temp\x.stl` was fine while backslashes passed through untouched; now \t is a tab, which is exactly right and exactly what breaks those paths. generic_string() gives forward slashes, which Windows accepts, and this is what a script has to do for real: a backslash in an OpenSCAD string is an escape, so a Windows path in an import() is written with forward slashes or doubled backslashes. The tests were relying on the bug. Only the sites that interpolate into a quoted string literal changed. `use <path>` and `include <path>` are lexed in a separate state that does not process escapes at all, so those keep string(), and none of them failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing did.
The parser hands back a literal's source text with its backslashes intact — deliberately, since that is what lets a
StringLiteralreproduce the source it came from, and what pretty-printing round-trips. But nothing cooked it afterwards, so every escape reached scripts raw:"a\nb"[97, 92, 110, 98][97, 10, 98]"a\tb"[97, 92, 116, 98][97, 9, 98]"a\\b"[97, 92, 92, 98][97, 92, 98]"a\"b"[97, 92, 34, 98][97, 34, 98]"a\qb"[97, 92, 113, 98][97, 113, 98]unescapeStringLiteral()resolves them where the literal is evaluated — which is both evaluation paths: the tree walk inexpr_evaland the constant pool inbytecode_compiler. A first attempt reached only one of them, so the tests exercise a literal at top level and one inside a function body.It returns the input untouched when there is no backslash in it, which is nearly every string, so the common case allocates nothing.
Line continuations
A backslash escaping the end of a line contributes nothing at all; the string carries on with no break in it.
A CRLF goes whole — unlike the reference implementation, which drops only the LF and leaves the CR in the value, putting a stray control character into any string continued in a file written on Windows. A lone CR is still an ordinary escaped character.
This also picks up the parser's fix for the same feature (BelfrySCAD/openscad_cpp_parser#4), where a backslash before a newline matched no lexer rule and was printed to stdout by flex's default ECHO.
Testing
808 tests pass. Negative-controlled twice: making the unescaper a no-op, and making a continuation keep its newline, each fail their own tests.
🤖 Generated with Claude Code