Skip to content

Harden validator scope analysis and access-policy enforcement - #25

Merged
nickklos10 merged 6 commits into
mainfrom
fix/validator-scope-and-access-policy
Oct 21, 2026
Merged

nickklos10 merged 6 commits into
mainfrom
fix/validator-scope-and-access-policy

Conversation

@nickklos10

Copy link
Copy Markdown
Owner

Summary

In v0.1.1, row filters, masking, and max_rows could be bypassed with ordinary SQL. The validator also rejected common read-only queries (CTEs, UNION, ORDER BY on aliases), which would make LLM-generated SQL fail often. This PR rebuilds column resolution on sqlglot's scope analysis and fixes each bypass.

Bypasses reproduced on 0.1.1 (filter customers: tenant_id = 42, 50 of 500 customers in tenant 42):

Query 0.1.1 This PR
SELECT (SELECT COUNT(*) FROM customers) AS n FROM customers LIMIT 1 500 50
SELECT COUNT(*) FROM (SELECT id, 42 AS tenant_id FROM customers) 500 50
... FROM orders o WHERE o.customer_id IN (SELECT c.id FROM customers c) filter bound to orders.tenant_id; 2000 rows 203 (correct)
SELECT email AS contact FROM customers with email masked raw email [REDACTED]
SELECT id FROM orders LIMIT -1 with max_rows=100 2000 rows 100

Changes

  • Scope-aware validation. Columns resolve through sqlglot.optimizer.scope, so CTEs (including recursive CTEs and CTE column lists), UNION/INTERSECT/EXCEPT, derived-table qualifiers, ORDER BY/HAVING on select aliases, and correlated or IN subqueries now validate. Unknown and ambiguous columns still fail closed. ValidationResult.tables lists only physical tables, not CTE names.
  • Row filters wrap each table reference. Each reference becomes (SELECT * FROM t WHERE <filter>) AS t, so the filter applies at every scope and cannot bind to another table's column or a spoofed alias. LEFT JOIN semantics are kept, which a per-scope WHERE would break. ⚠️ Rewritten SQL for filtered queries changes shape; the two tests asserting the old form are updated.
  • Lineage-based masking. The new ValidationResult.output_masks records a masking mode per output position. Aliases, expressions, CASE, string and json functions, CTEs, derived tables, scalar subqueries, and set-op branches are masked. COUNT(masked_col) is left unmasked. If a row's shape doesn't match the analyzed projection (for example, duplicate column names, or a USING join's star), every value is masked.
  • LIMIT cap. Anything other than a non-negative integer literal within max_rows is replaced: LIMIT -1, expressions, subqueries, and FETCH.
  • Read-only. SELECT ... INTO is rejected. SQLite is opened with mode=ro, matching the Postgres READ ONLY transaction.
  • Missing fixture. A missing SQLite file raises FileNotFoundError instead of silently creating an empty database. eval run exits early with Fixture database not found: <path>.
  • Dependency floor. sqlglot>=25.20. The old >=25.0 floor was already broken: 0.1.1's validator uses exp.Alter, which doesn't exist before roughly 25.20.

Testing

  • pytest passes locally (354 passed). The 40 new or updated tests fail on main's src/.
  • New behavior has a unit test
  • Validator/safety changes include a safety-suite case (blocks_select_into)
  • User-facing CLI changes have a CLI test (missing fixture DB)
  • CHANGELOG.md updated under [Unreleased]
  • Eval baseline unchanged: smoke is 3/3, safety is 8/8, and eval check passes against .eval/baseline.json

Also tested:

  • Full suite on sqlglot 25.20.0, 27.0.0, 30.0.0, and 30.24.0. One upstream issue: sqlglot before 30.24 drops a CTE column list (r(n)) from SQLite output, so such queries fail at execution. That fails closed.
  • Adversarial probes for shadowed CTE/table names, schema-qualified and quoted names, self joins, USING/NATURAL star joins, table-valued functions, and sqlite_master. No leaks; the unsupported forms are rejected.

Not covered:

  • The Postgres connector was not run against a live database.
  • A masked column used only in a predicate (WHERE email LIKE 'a%') can still allow inference. That behavior is unchanged from 0.1.1.
  • ruff check src tests reports 44 findings on main with the unpinned ruff 0.17. This branch adds none.

🤖 Generated with Claude Code

nickInceptaAi and others added 4 commits October 20, 2026 20:23
Row filters, masking, and max_rows could be bypassed in 0.1.1, and the
validator rejected common read-only SQL (CTEs, UNION, alias references).

- Resolve columns through sqlglot scope analysis: CTEs (incl. recursive
  and column lists), set operations, derived tables, ORDER BY/HAVING on
  select aliases, and correlated/IN subqueries now validate correctly.
- Apply row filters by wrapping every reference to a filtered table as
  (SELECT * FROM t WHERE <filter>) AS t, so they hold in subqueries, CTEs,
  and set operations, cannot bind to another table's column or a spoofed
  alias, and keep outer-join semantics.
- Mask by output-column lineage (ValidationResult.output_masks) instead of
  result key names; mask every value if the row shape does not match.
- Replace any LIMIT that is not a non-negative literal within policy.
- Reject SELECT ... INTO; open SQLite read-only (mode=ro).
- Raise on a missing SQLite file instead of creating it; eval run exits
  with "Fixture database not found" before running cases.
- Raise sqlglot floor to 25.20 (earlier versions lack exp.Alter).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd-access-policy

# Conflicts:
#	CHANGELOG.md
#	src/querypilot/cli.py
Unpinned ruff resolved to 0.17.1, whose expanded default rules report
49 findings on main as well as this branch. 0.15.22 passes both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nickInceptaAi and others added 2 commits October 20, 2026 21:47
…nd-access-policy

# Conflicts:
#	.github/workflows/eval.yml
`SELECT strftime('%Y-%m', d) AS month ... GROUP BY month` was rejected with
"Unknown column: month", which made 4 SaaSPulse correctness golds
unwinnable. An unqualified GROUP BY name now resolves to a select alias
when no source column has that name (source columns still win, as in
SQLite). Adds a test that every SaaSPulse gold passes the validator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nickklos10
nickklos10 merged commit b94d35c into main Oct 21, 2026
5 checks passed
@nickklos10
nickklos10 deleted the fix/validator-scope-and-access-policy branch October 21, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants