[nickklos10]

Summary

This PR makes QueryPilot easier to adopt on any database, LLM provider, agent framework or MCP client. The work is prioritised by a research pass over about 250 sourced developer pain points (Vanna, WrenAI, DB-GPT, LangChain/LlamaIndex SQL tooling, MCP database servers, Spider/BIRD literature). See docs/research/developer-needs.md and the ranked gap analysis in docs/roadmap.md.

• Quickstart:
  • pip install querypilot, then querypilot demo, schema, sql, ask and init.
  • A querypilot.yaml project config, read by the CLI, the servers and QueryPilot.from_config().
  • querypilot mcp-config <claude-code|claude-desktop|cursor|vscode|windsurf> prints a paste-ready config. It uses an absolute uvx path and passes QUERYPILOT_CONFIG.
• Any SQLAlchemy database:
  • The dialect is inferred from the URL or an Engine. On main, connect("postgresql://…") silently built a SQLite connector.
  • A new SQLAlchemyConnector covers DuckDB, MySQL/MariaDB, SQL Server, Snowflake, BigQuery, Redshift, Trino, Oracle, Databricks and ClickHouse, each with its own pip extra.
  • Introspection now includes keys, foreign keys, comments, views and schemas, and the schema is cached.
  • Read-only enforcement is layered for each engine, and dangerous functions are blocked per dialect.
• One agent toolkit:
  • Tools: list_tables, describe_table, search_schema, validate_sql, run_sql and ask_database.
  • Results are compact JSON with a truncated flag. Errors come back as payloads with "did you mean" hints instead of exceptions.
  • Exposed for OpenAI (Chat and Responses), Anthropic, MCP (readOnlyHint; mcp 1.x and 2.x), LangChain (optional drop-in sql_db_* names), LlamaIndex, the OpenAI Agents SDK, the Claude Agent SDK and plain functions.
• Better generated SQL (+14.5 points on BIRD):
  • Prompts carry compact DDL with PKs and FK join paths, comments, policy-safe sample values (every value for small enums), and business hints=/descriptions=.
  • Large schemas are pruned only above a token budget, keeping FK bridge tables and join partners.
  • The repair loop also handles database errors and empty results, with up to 3 attempts by default. ask() and evals share one loop.
  • The prompt asks for exactly the columns the question names.
• Benchmarks:
  • A natural-phrasing SaaSPulse variant, plus a wide-schema generator that adds 300 distractor tables.
  • A runner that uses the local claude CLI as the model.
  • CI now also validates the SaaSPulse golds and smoke-tests the quickstart CLI.

Bugs on main fixed along the way

• querypilot mcp crashed on mcp 2.x (mcp.server.fastmcp no longer exists), which is what mcp>=1.0 installs today.
• The MCP registry entry ran uvx querypilot, which doesn't install the mcp extra.
• QueryPilot.connect("postgresql://…") built a SQLite connector.

Policy hardening from code review

• Database error text is redacted, so masked values can't leak through errors such as a failed CAST(email AS INTEGER).
• Policy errors no longer name hidden columns.
• Row-filter predicates are no longer echoed to agents.
• The HTTP /schema endpoint honours the access policy.
• The schema cache is shared across per-request copies.

Behaviour changes

These are listed in the CHANGELOG:

• The default dialect is inferred instead of sqlite.
• Tool results return columns plus row arrays.
• handle_anthropic_tool_call returns error payloads instead of raising.
• run_sql is advertised instead of execute_sql (the old name still works).
• PyYAML is now a core dependency.

Quality measurement

Full write-up in benchmarks/results/2026-10-20-agent-dx.md. All runs use Claude Haiku via the claude CLI. "Before" is #25's head; "after" is this branch merged with #25.

BIRD dev, 220-question stratified sample (11 real databases, 2 runs each). This is the headline result:

Variant	Accuracy	Simple	Moderate	Challenging
Before	42.5%	49%	39%	33%
Grounded context and repair	47.7%	56%	42%	38%
Plus the exact-columns prompt rule (this PR)	57.0%	65%	51%	50%

About half of the misses that remained after grounding were unrequested extra columns. The exact-columns rule targets those, and part of its gain reflects BIRD's strict column scoring.

SaaSPulse. This benchmark is saturated for this model; it confirms there is no accuracy regression and shows the prompt-size change:

Suite	Before	After	Prompt tokens per question
Core (3 runs)	194/195	195/195	426 → 907
Natural phrasing (3 runs)	195/195	195/195	426 → 907
Wide, 308 tables (2 runs)	130/130	130/130	13,323 → 4,884 (−63%)

The runs caught two regressions in this branch, both fixed: pruning dropped join partners, and top-3 enum samples misled the model. A 64-thread concurrency check (400 mixed toolkit calls on BIRD's 459 MB database) produced no errors beyond the expected structured rejections.

Relationship to #25

This PR doesn't touch validation/validator.py. A test merge with #25 conflicts only on import lines in connectors/sqlite.py and core/client.py: keep both sides. 803 tests pass on the merged tree. Validator gaps found while adding dialects are listed in the session notes (e.g. DuckDB SELECT * FROM 'file.csv'); none were fixed here.

Test plan

• ruff check src tests (0.17.1)
• pytest: 694 passed
• BIRD dev sample (220 questions) and SaaSPulse before/after with Claude Haiku
• python benchmarks/validate_golds.py
• Smoke querypilot eval run / eval check from eval.yml, plus suites/safety.yaml (7/7)
• MCP over stdio with a real mcp client (list_tables, run_sql, structured errors)
• examples/01 and examples/07 run
• Live Postgres, MySQL and warehouse runs (only SQLite and DuckDB were exercised against real engines; other read-only session statements are checked against fakes)
• A full agent loop through the OpenAI Agents and Claude Agent SDKs (tools were invoked directly)

🤖 Generated with Claude Code