SQLite: a single non-ASCII byte in a leading comment corrupts star-expanded query text (byte/rune offset drift, pre-meyer-rewrite ANTLR parser) — already fixed on main, unreleased #4665

Version

v1.27.0 (also reproduced on v1.31.1 / @latest)

What happened?

sqlc generate corrupts (or outright fails to generate) the SQL text embedded in a generated query constant, for the SQLite engine, whenever a star-expanded query (SELECT *) is preceded anywhere in the file by a comment containing even a single non-ASCII (multi-byte UTF-8) character — e.g. an em dash (—).

Depending on exact file layout this shows up as either:

• a sqlc generate failure: edited query syntax is invalid: mismatched input 'SELECid' expecting {...}, or
• successful generation with a silently corrupted SQL string constant in the generated .go file (truncated/garbled text).

Root cause (confirmed by bisection): this is a byte-offset vs. rune/character-offset drift in the legacy ANTLR-based SQLite parser (internal/engine/sqlite, pre-rewrite). The star-expansion step (internal/compiler/expand.go's source.Edit{Location: res.Location - raw.StmtLocation, ...}) computes an edit position that's correct when counted in bytes, but the parser/lexer appears to report Location/statement-extent positions counted in Unicode code points (or some other non-byte unit) for at least one of the inputs it derives them from. Every multi-byte UTF-8 character occurring before the edit point (e.g. a 3-byte em dash counted as 1 "unit") introduces a few bytes of drift. For a short query this doesn't move the edit far enough to land somewhere that breaks visibly; for a query preceded by a longer or richer leading comment the accumulated drift moves the edit into the wrong byte range of the query text (sometimes landing mid-identifier, e.g. splitting SELECT into SELEC + the inserted expansion + id, producing SELECid), corrupting the query text or making it fail to re-parse.

This only affects the SQLite engine. I did not find the equivalent problem in the PostgreSQL/MySQL analyzers.

Minimal reproduction

sqlc.yaml:

version: "2"
sql:
  - engine: "sqlite"
    schema: "sqlite/schema.sql"
    queries: "sqlite/query.sql"
    gen:
      go:
        package: "querytest"
        out: "sqlite"

sqlite/schema.sql:

CREATE TABLE authors (
  id   INTEGER PRIMARY KEY,
  name TEXT NOT NULL,
  bio  TEXT
);

sqlite/query.sql (note the single em dash — in the comment — this is the entire trigger):

-- A leading comment block that contains one non-ASCII character — right here.

/* name: GetAuthor :one */
SELECT * FROM authors
WHERE id = ? LIMIT 1;

Running sqlc generate on v1.27.0 and on @latest (v1.31.1) both produce:

line 4:0 mismatched input 'SELECid' expecting {<EOF>, ';', ALTER_, ANALYZE_, ATTACH_, BEGIN_, COMMIT_, CREATE_, DELETE_, DETACH_, DROP_, END_, EXPLAIN_, INSERT_, PRAGMA_, REINDEX_, RELEASE_, REPLACE_, ROLLBACK_, SAVEPOINT_, SELECT_, UPDATE_, VACUUM_, VALUES_, WITH_}
sqlite/query.sql:1:1: edited query syntax is invalid: mismatched input 'SELECid' expecting {...}

Replacing the em dash with a plain ASCII hyphen (-), with everything else byte-identical, generates cleanly. This isolates the trigger precisely to the presence of a multi-byte UTF-8 character anywhere before the query's star-expansion edit point — not query length, not comment style (-- vs /* */), not number of queries in the file.

Already fixed, unreleased

I checked this against current main (3ad8b20, after the SQLite engine's parser was rewritten from the ANTLR grammar to the new github.com/sqlc-dev/meyer-based parser in #4567/#4580/#4606/#4610 and related commits) — the exact same failing input generates cleanly on main. So this looks already fixed by the parser rewrite, just not yet in a tagged release. I'm filing this mainly so:

1. Anyone hitting it on a current release (v1.27.0 through at least v1.31.1) has a precise trigger to work around (avoid non-ASCII bytes in comments preceding a SELECT * query in SQLite query files) while waiting for the next release.
2. It's confirmed/tracked in case the main fix was coincidental rather than a deliberate fix for this exact class of bug, since I couldn't find an existing issue or changelog entry describing this specific byte/rune drift.

Happy to help verify against a specific commit/release candidate if useful. I didn't open a PR since the affected code (the ANTLR-based internal/engine/sqlite parser) has already been replaced wholesale on main, so a point-fix against it would just be throwaway work on code that's going away.

Database schema

CREATE TABLE authors (
  id   INTEGER PRIMARY KEY,
  name TEXT NOT NULL,
  bio  TEXT
);

Queries

-- A leading comment block that contains one non-ASCII character — right here.

/* name: GetAuthor :one */
SELECT * FROM authors
WHERE id = ? LIMIT 1;

Configuration

version: "2"
sql:
  - engine: "sqlite"
    schema: "sqlite/schema.sql"
    queries: "sqlite/query.sql"
    gen:
      go:
        package: "querytest"
        out: "sqlite"

Playground link

No response

What OS are you using?

macOS (also reproduced via Linux-based go run toolchain)

What database engines are you using?

SQLite