spec: sqlflow serve, named SQL over HTTP - #275
Merged
Merged
Conversation
A pipeline ends at a sink and the rows need an API. This designs a serve command: a separate config with commands and datasets, bearer tokens as identities, typed params bound through prepared statements, a row cap and a timeout, and a reserved rate-limit policy that is refused until enforced. Verified on DuckDB 1.5.2 that the postgres extension pushes filters and projections down but not GROUP BY, and that a filter against a Postgres view pushes into the view. The example config is shaped by that.
The spec left binding and cancellation to the plan and asserted that $from fails to parse. A probe against DuckDB 1.5.2 and arrow-adbc 1.6.0 settled all three, and two of the answers were not the ones the spec planned for: - DuckDB binds by position. The parameter schema names fields 0 and 1, and a bound record's field names are ignored. The server now rewrites $name to $N itself and checks DuckDB's parameter count against its own. - The Go driver manager ignores the context and exposes no cancel. A timeout returns 504 while the query runs on and holds the lock. - $from prepares fine. The CLI's EXECUTE call syntax rejected it. The reserved-word rule is gone. Also from the probe: SetSqlQuery binds eagerly, so a missing view fails at startup; Arrow renders HUGEINT through a float, so decimals encode as exact strings; results stream, so the row cap stops the query. Serve examples move out of dev/config/examples, where two sweeps decode every file as a pipeline. --metrics is dropped until serve records an instrument.
Owner
Author
|
Superseded by #278, which carries this spec's commits and the implementation. |
turbolytics
added a commit
that referenced
this pull request
Sep 13, 2026
#275 squash-merged this branch's first two commits, so both sides added docs/superpowers/specs/2026-09-12-serve-design.md. main's copy is byte-identical to ac97dbc. This branch's copy is that plus 40ec437 and 00432fe, which record what implementation changed and the 500 body that no longer carries DuckDB's error, so the conflict resolves to this branch's copy. #277's README, example and manager changes merged cleanly. The cli, config, schema, validate, managers and serve packages pass under -race after the merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A design spec for
sqlflow serve: a request/response mode that exposes only the datasets a config declares, as JSON over HTTP, from the same config machineryrunuses.docs/superpowers/specs/2026-09-12-serve-design.mdThe shape
commands:plus aserve:block. Thecommands:block is the source, soATTACH … (TYPE POSTGRES, READ_ONLY)reaches the data the same way a pipeline does.string,integer,timestamp) bound through prepared statements. A dataset has onesqlor agrainsmap of them. Nothing from a request becomes SQL text.rate_limitpolicy, global and per dataset, parsed and refused until it is enforced.sqlflow validatepicks the schema by top-level key.What was measured
On DuckDB 1.5.2 the postgres extension pushes filters and projections down but runs
GROUP BYitself. A filter against a Postgres view pushes into the view. A 30-day daily rollup: 2.2 s aggregated in DuckDB, 0.12 s through a view. The example config is shaped by that.Settled by a Go probe
Probed against DuckDB 1.5.2 through arrow-adbc 1.6.0, and folded into the spec in ac97dbc:
0,1, and a bound record's field names are ignored. The server numbers$nameplaceholders itself and checks its count against DuckDB's at startup.504, and the query holds the connection until it finishes.$fromprepares fine. The earlier claim that it is a parse error was wrong: the DuckDB CLI'sEXECUTE q(from := …)call syntax rejected it, not the placeholder. No reserved-word rule exists.dev/config/serve/, because two sweeps decode everything underdev/config/examples/as a pipeline.--metricsis dropped until serve records instruments.First audience is the practitioner who already runs a pipeline and wants programmatic access to what it wrote. The first consumer is the bluesky demo's page; that repo's spec codes against this contract.