fix: apply max_rows to comment-prefixed and CTE queries - #400
Conversation
`SQLRowLimiter.isSelectQuery` classified a statement with
`sql.trim().toLowerCase().startsWith("select")`, so `max_rows` was
silently a no-op for every query that does not open with a bare
`SELECT`. No LIMIT was appended at all — the cap simply did not exist
for these, and nothing surfaced that:
-- reporting job\nSELECT * FROM t -> returned unchanged
/* tag */ SELECT * FROM t -> returned unchanged
WITH x AS (SELECT ...) SELECT * FROM x-> returned unchanged
(SELECT a) UNION (SELECT b) -> returned unchanged
Leading comments are common (query tags/attribution) and CTEs are the
normal shape of an analytical query, so in practice the row cap covered
much less than it appeared to.
Classify on the comment/string-blanked text instead, and accept `WITH`
as row-returning. The SQL handed to the server is never rewritten by the
classification, so an attribution comment survives verbatim. A
data-modifying CTE (`WITH x AS (DELETE ... RETURNING *) SELECT ...`) is
deliberately still not limited: a LIMIT would cap the rows handed back
while the write ran in full, which reads as a cap that isn't one. That
check reuses the read-only classifier's keyword heuristic, now shared as
`hasMutatingKeyword`.
Enabling CTEs exposed a second problem: the LIMIT/TOP helpers matched
the first clause found textually, which on a CTE is the *inner* one.
`WITH x AS (SELECT ... LIMIT 5) SELECT * FROM x JOIN big ON true` would
have had the CTE's own cap rewritten while the statement stayed
uncapped. Every helper now scans with parenthesis-depth tracking (the
mechanism already used for `hasSetOperator` and the ORDER BY hoist) and
reasons only about the statement's own clause; a nested LIMIT/TOP caps
only its branch, so the statement still gets a cap appended. This also
fixes the same pre-existing hole for plain subqueries
(`SELECT * FROM (SELECT * FROM t LIMIT 5) s`).
On SQL Server, TOP now lands on the statement's own SELECT rather than
the first one textually (for a CTE, the final SELECT), and a leading CTE
is kept outside the derived table when a set operation forces a wrap —
T-SQL has no `SELECT ... FROM (WITH ...) AS subq` form.
`SELECT ... UNION ALL SELECT ...` keeps its existing behaviour: it
starts with `select`, gets a trailing LIMIT, and on PostgreSQL that
binds to the whole set operation. Covered by a regression test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
for_stockly_main is the integration branch: upstream main plus one merge per open PR, so the PR branches stay single-commit and reviewable on their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
for_stockly_main is the integration branch: upstream main plus one merge per open PR, so the PR branches stay single-commit and reviewable on their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes max_rows enforcement gaps by correctly classifying comment-prefixed and CTE (WITH) queries as row-returning, and by ensuring LIMIT/TOP detection targets the statement’s own top-level clauses (not nested CTE/subquery clauses). It also updates integration tests to reflect the corrected behavior across connectors, including SQL Server set-operation handling related to #387.
Changes:
- Improve row-returning classification to handle leading comments, leading parentheses, and CTEs while avoiding data-modifying CTEs.
- Rework LIMIT/TOP/ORDER BY detection to ignore nested clauses via top-level parenthesis-depth scanning, and adjust SQL Server wrapping behavior for set operations + CTE prefixes.
- Update/expand unit + integration tests to cover the corrected max_rows behavior for comments, CTEs, nesting, and SQL Server cases.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/utils/sql-row-limiter.ts | Refactors classification and top-level clause detection; updates LIMIT/TOP application logic including SQL Server set-op wrapping. |
| src/utils/allowed-keywords.ts | Extracts hasMutatingKeyword to share mutating detection logic across components. |
| src/utils/tests/sql-row-limiter.test.ts | Adds extensive unit coverage for comment-prefixed queries, CTEs, nesting, and SQL Server TOP behavior. |
| src/connectors/tests/sqlite.integration.test.ts | Updates SQLite integration test to expect maxRows applied to CTE queries. |
| src/connectors/tests/postgres.integration.test.ts | Updates Postgres integration test expectations and adds integration coverage for comment-prefixed and CTE-specific cases. |
| src/connectors/tests/mysql.integration.test.ts | Updates MySQL integration test to expect maxRows applied to CTE queries. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // Splice at the clause's own position rather than replacing the first | ||
| // LIMIT found textually, which on a CTE would rewrite the CTE's cap and | ||
| // leave the statement itself uncapped. | ||
| const effectiveLimit = Math.min(limit.value, maxRows); | ||
| return `${sql.slice(0, limit.index)}LIMIT ${effectiveLimit}${sql.slice(limit.index + limit.length)}`; |
|
Hey @tianzhou 👋 I saw you closed this PR, can I fix the issue raised and re-open or do you not want this? Thanks 🙏 |
Hey 👋 This looks like a bug but if it's wanted behavior, feel free to drop 😊
SQLRowLimiter.isSelectQuerydecided whether to cap a statement with:so anything not opening with a bare
SELECTgot no LIMIT appended at all. Not a wrong limit —no limit, and nothing anywhere says so. Verified by executing the shipped v1.2.0 module
(
applyMaxRows(sql, 100)):-- reporting job\nSELECT * FROM t/* tag: report */ SELECT * FROM tWITH x AS (SELECT * FROM orders) SELECT * FROM x(SELECT id FROM a) UNION (SELECT id FROM b)SELECT * FROM (SELECT * FROM t LIMIT 5) sSELECT * FROM tSELECT * FROM t\nLIMIT 100✅This is not an exotic corner. Leading comments are how queries get attributed (
-- app=…,/* trace-id */), and a CTE is the ordinary shape of an analytical query — which is exactly thekind of query a row cap exists to contain. In practice
max_rowscovered far less than it appearedto. We hit this running DBHub against ~24 production PostgreSQL read replicas with agents driving
execute_sql.The existing suite pinned the gap rather than catching it: three integration tests were named
should not apply maxRows to CTE queries (WITH clause). This PR inverts them.To be clear about what those tests were pinning:
git log -Sshows thestartsWith('select')guardand those three tests both arrive in the same commit —
4fa886e feat: support max-row limit, thecommit that introduced
max_rows. The CTE exclusion was never a separate, deliberate carve-out; itis a consequence of that one-line classifier, and the tests recorded whatever it happened to do.
(One of the assertions is commented "not limited anymore", which reads as though a decision had been
reversed — the history shows there was no earlier behaviour to reverse.) If the exclusion is
wanted, it deserves to be an explicit rule rather than a side effect of the guard, and this PR
should be redirected to documenting it instead.
Fix
Classify on the comment/string-blanked text, and treat a leading
WITHas row-returning. The SQLsent to the server is never rewritten by the classification, so an attribution comment survives
verbatim — only the classifier sees the blanked form.
A data-modifying CTE (
WITH x AS (DELETE … RETURNING *) SELECT …) is deliberately still notlimited: a LIMIT would cap the rows handed back while the write ran in full, which reads as a cap
that isn't one. That check reuses the read-only classifier's own keyword heuristic, extracted as
hasMutatingKeyword, so both layers agree on what counts as a write hidden inside aWITH. A falsepositive there only means the statement keeps today's behaviour.
Enabling CTEs exposed a second bug, which is why the diff is larger than "add
withto a list".Every LIMIT/TOP helper matched the first clause found textually, which on a CTE is the inner one:
Naively enabling CTEs would have rewritten the CTE's own
LIMIT 5and left the statement — the partthat can return millions of rows — uncapped. So the helpers now scan with parenthesis-depth tracking
(the mechanism already in this file for
hasSetOperatorand the ORDER BY hoist) and reason onlyabout the statement's own clause; a nested LIMIT/TOP caps only its branch, so the statement still
gets a cap appended. This also fixes the same pre-existing hole for plain subqueries
(
SELECT * FROM (SELECT * FROM t LIMIT 5) s, last row of the table above).On SQL Server,
TOPnow lands on the statement's ownSELECTrather than the first one textually(for a CTE, the final
SELECT), and a leading CTE is kept outside the derived table when a setoperation forces the
#387wrap — T-SQL has noSELECT … FROM (WITH …) AS subqform.Explicitly preserved:
SELECT … UNION ALL SELECT …starts withselect, still gets a trailingLIMIT, and on PostgreSQL that binds to the whole set operation. Pinned by a regression test.Testing
pnpm run test:unit— 981 passed (958 onmain; +23 new).pnpm run test:integration— 334 passed (331 onmain; +3) across PostgreSQL, MySQL, MariaDB,SQL Server and SQLite containers.
pnpm run build:backend— clean.New coverage: leading
--//* *//mixed comments; CTE; CTE with an inner LIMIT; CTE with its ownLIMIT; CTE with a parameterized LIMIT; data-modifying CTE (all three of INSERT/UPDATE/DELETE);
parenthesised set operation; nested-subquery LIMIT;
UNION ALLregression; and SQL Server CTE/TOPcases.
Notes for reviewers
npx tsc --noEmitreports 132 pre-existing errors onmain; this branch reports the same 132,none in the changed files.
independent and share no files but one integration test file.