[nickklos10]

What

Adds querypilot trace and querypilot hooks, which observe and improve the SQL that agents run through MCP and connector tools QueryPilot can't proxy. The first target is Palantir Foundry's mcp__palantir-mcp__run_sql_query_on_foundry_dataset in Claude Code. Any other tool can be mapped by name and SQL input field: --tool NAME=FIELD[:DIALECT[:ROW_LIMIT_FIELD]].

• Capture. trace import pairs each tool_use in a Claude Code transcript with its tool_result by id. It writes the SQL, outcome, failure class, latency, session and order to the JSONL audit log, and never writes result rows. hooks install adds PostToolUse and PostToolUseFailure hooks that log calls live. Hooks never fail a tool call.

• Guard (opt-in, --guard). A PreToolUse hook validates the SQL with SQLValidator (Spark dialect) against a schema cache. The cache is built from get_foundry_dataset_schema results and single-dataset SELECT * results. The hook denies unknown columns with "did you mean" text, blocks epoch-ms arithmetic on DATE columns, and enforces --max-rows. --require-schema also blocks queries on datasets whose schema hasn't been fetched yet.

  • If the cache is stale, the agent can resend the identical query and it goes through; when that query succeeds, the post hook drops the stale entry.

• Doctor. trace doctor reports:

  • the failure rate by class;
  • retry chains;
  • latency p50/p90 and stalls;
  • a guard replay over the history;
  • findings tagged as a tool, prompt or data fix, with ready-to-paste hints: and descriptions: blocks.

  --export-replay / --suite turn recovered retry chains into an eval replay regression suite.

• QueryAuditRecord gains optional tool_name, tool_call_id and details fields.

Results on the real Foundry history

The report is in docs/trace-doctor-foundry-2026-10-21.md. It is redacted: dataset RIDs and paths become labels, literals become '?', and no result rows appear.

• Calls: 5,163 Foundry SQL calls across 71 sessions (224 transcripts), 11.8% failed. Latency was p50 3.8s and p90 11.6s, with 84 calls over 60s.
• Failure classes:
  • Unresolved column: 191
  • Result too large: 165
  • Upstream/network: 107
  • Unknown table: 57
  • Type mismatch: 47
• Guard replay: the guard would have stopped 98 of the unresolved-column failures and 24 of the type mismatches before they ran. It would also have blocked 38 of 3,099 successful queries it could check, mostly on datasets rebuilt with new columns after their schema was cached.

docs/trace.md compares these numbers with the earlier quick scan.

Testing

• tests/test_trace.py (41 tests) covers the classifier, the transcript parser, the schema cache and the guard. It also covers the doctor and the replay export, and checks the hooks by replaying a synthetic sample transcript through them, including a subprocess round trip.
• Full suite: 915 passed, 28 skipped.
• ruff check src tests (0.17.1) is clean.

Known gaps

• The validator reads Spark struct fields (alias.field) and backticked column names that contain dots as table qualifiers. The guard skips those queries instead of judging them.
• Only Claude Code's transcript format is read.

🤖 Generated with Claude Code