update ggsql version and add new writers

.github/workflows/MainDistributionPipeline.yml

@@ -14,19 +14,19 @@ concurrency:
 jobs:
   duckdb-stable-build:
     name: Build extension binaries
-    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5.4
+    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5.6
     with:
-      duckdb_version: v1.5.4
-      ci_tools_version: v1.5.4
+      duckdb_version: v1.5.6
+      ci_tools_version: v1.5.6
       extension_name: ggsql
       extra_toolchains: 'rust'
       exclude_archs: 'wasm_mvp;wasm_eh;wasm_threads'
 
   code-quality-check:
     name: Code Quality Check
-    uses: duckdb/extension-ci-tools/.github/workflows/_extension_code_quality.yml@v1.5.4
+    uses: duckdb/extension-ci-tools/.github/workflows/_extension_code_quality.yml@v1.5.6
     with:
-      duckdb_version: v1.5.4
-      ci_tools_version: v1.5.4
+      duckdb_version: v1.5.6
+      ci_tools_version: v1.5.6
       extension_name: ggsql
       format_checks: 'format;tidy'

.gitignore

@@ -9,3 +9,4 @@ rust/target
 .cache
 .claude
 .format-tools
+tools/hep-viewer/node_modules

CLAUDE.md

@@ -1,6 +1,6 @@
 # ggsql-duckdb
 
-DuckDB extension that routes `VISUALISE`/`VISUALIZE` statements through the Rust [ggsql](https://ggsql.org) engine, rendering vega-lite charts. Built on the DuckDB extension template.
+DuckDB extension that routes `VISUALISE`/`VISUALIZE` statements through the Rust [ggsql](https://ggsql.org) engine, rendering with ggsql's native hephaestus renderer (vega-lite remains only as the `spec` escape hatch). Built on the DuckDB extension template.
 
 ## Layout
 
@@ -14,11 +14,11 @@ src/                    C++ extension: ParserExtension, scalar+table funcs, FFI
 rust/                   Rust staticlib linked into the extension
   src/lib.rs            `ggsql_execute` C entrypoint; dispatches on output mode
   src/reader.rs         `CallbackReader` impls ggsql::Reader via the C++ bridge
-  src/server.rs         tiny_http singleton serving the SPA + vega assets
+  src/server.rs         tiny_http singleton serving the SPA + hep viewer assets + .hep documents
   src/ffi.rs            C ABI types (ByteBuffer, ReaderBridge)
   src/dialect.rs        DuckDbDialect, inlined from ggsql (see comment below)
   include/ggsql_ext_rs.h  Hand-rolled C header; must track ffi.rs
-  assets/               Vendored vega/vega-lite/vega-embed bundles + SPA shell
+  assets/               Vendored hep viewer bundle + wasm/font assets (built by tools/hep-viewer) + SPA shell
 test/sql/ggsql.test     SQL logic tests — run with GGSQL_NO_OPEN_BROWSER=1
 duckdb/, extension-ci-tools/  Submodules; versions bumped per release (see docs/UPDATING.md)
 ```
@@ -37,12 +37,13 @@ Two entry points, both funnel into the same Rust `ggsql_execute`:
 
 - **ParserExtension** — any statement containing a top-level `VISUALISE`/`VISUALIZE` keyword is claimed and planned as a call to the `ggsql_run` table function. The scanner in `ggsql_parser.cpp` is hand-rolled: it skips `'...'`, `"..."`, `--` line comments and `/* */` block comments, and matches only at word boundaries. A trailing `;` is stripped because ggsql's tree-sitter grammar rejects it.
 - **Scalar** — `SELECT ggsql('<query>')` runs the same pipeline with the string as input.
+- **Save scalar** — `SELECT ggsql_save('<query>', '<path>')` infers the writer from the file extension (.svg/.pdf/.hep/.html/.json), applies `ggsql_writer_options`, ignores `ggsql_output`, writes the file in C++ (`std::ofstream`), and returns the path.
 
-Output mode is session-scoped via the `ggsql_output` setting (`silent` default / `url` / `spec` / `html`). The result column is always named `plot` — don't rename it. Unknown values throw at bind time. `silent` emits zero rows; `url` emits one; `spec`/`html` return the bytes and do not start the HTTP server or open a browser.
+Output mode is session-scoped via the `ggsql_output` setting (`silent` default / `url` / `spec` / `html` / `svg` / `pdf` / `hep`). The result column is always named `plot` — don't rename it. Unknown values throw at bind time. `silent` emits zero rows; `url` emits one; `spec`/`html` return the bytes and do not start the HTTP server or open a browser. The browser display (`silent`/`url`) renders a .hep document with the vendored hephaestus-svg-wasm viewer — the same renderer as the `svg`/`pdf`/`hep` writers, so display and file output match. `svg` returns VARCHAR, `pdf`/`hep` return binary — `ggsql_run` types their column as BLOB, the scalar stays VARCHAR. `ggsql_writer_options` is a `key=value;…` string forwarded verbatim to ggsql's `WriterOptions` (shared keys: width/height/units/dpi/background); only `spec` rejects options — the Rust side errors otherwise.
 
 ## FFI contract (C++ ↔ Rust)
 
-- `ggsql_execute(query, len, bridge, mode, out) -> int32` — 0 ok, 1 error (payload in `out`), 2 panic. `out` is a `ggsql_byte_buffer_t` allocated by Rust; **caller must free via `ggsql_free_buffer`**.
+- `ggsql_execute(query, len, bridge, writer, writer_len, options, options_len, out) -> int32` — 0 ok, 1 error (payload in `out`), 2 panic. `writer` names the output path (`silent`/`url`/`html`/`spec`/`svg`/`pdf`/`hep`); `options` is the raw writer-options string. `out` is a `ggsql_byte_buffer_t` allocated by Rust; **caller must free via `ggsql_free_buffer`**.
 - `ReaderBridge` has two callbacks, both implemented in `ggsql_bridge.cpp`:
   - `exec_sql` — runs SQL on the inner `Connection`, returns Arrow stream via C Data Interface.
   - `free_buffer` — C++ frees a buffer it previously populated (for error messages).
@@ -66,7 +67,7 @@ The inner `Connection` is created lazily and **persists for the whole `ggsql_exe
 - Spec registry is a `HashMap<uuid, json>` in memory; unbounded (grows for the life of the process).
 - SPA URL form is `http://.../#plot/<uuid>` — the hash matters: browsers treat tabs with different fragments as the same URL for `open::that`, so repeat queries focus the existing tab instead of spawning new ones.
 - `/api/latest` doubles as a liveness heartbeat: if it was polled within `TAB_ALIVE_WINDOW` (5s), `register_spec` sets `should_open=false` and the poll loop in the SPA picks up the new plot via `history.pushState` — no second window. `GGSQL_NO_OPEN_BROWSER` overrides regardless.
-- Vega/vega-lite/vega-embed bundles are `include_str!`'d from `rust/assets/` so plots render offline. `html` mode inlines the same bundles into a single self-contained document. `</` in the spec is escaped to avoid `</script>` breakout.
+- The hep viewer (`hep-viewer.js`, IIFE) and wasm/Roboto assets (`hep-assets.js`, gzip+base64) are built by `tools/hep-viewer/build.mjs` from the `hephaestus-svg-wasm` npm package and `include_str!`'d from `rust/assets/`, so plots render offline. **The npm package version must track the hephaestus crate ggsql links** — the .hep document format version is compared for equality at load time. `html` mode inlines the same two assets plus the base64'd .hep document into a single self-contained file; base64 can't contain `</script>`, so no breakout escaping is needed.
 
 ## Inlined `DuckDbDialect`
 

CMakeLists.txt

@@ -24,7 +24,7 @@ build_static_extension(${TARGET_NAME} ${EXTENSION_SOURCES})
 build_loadable_extension(${TARGET_NAME} " " ${EXTENSION_SOURCES})
 
 # -----------------------------------------------------------------------------
-# Rust side (ggsql_ext_rs staticlib — embeds ggsql + the vega-lite HTTP server)
+# Rust side (ggsql_ext_rs staticlib — embeds ggsql + the plot HTTP server)
 # -----------------------------------------------------------------------------
 set(GGSQL_RUST_DIR ${CMAKE_CURRENT_SOURCE_DIR}/rust)
 set(GGSQL_RUST_BUILD_DIR ${CMAKE_CURRENT_BINARY_DIR}/rust-target)
@@ -81,14 +81,16 @@ add_custom_target(ggsql_rust_lib DEPENDS ${GGSQL_RUST_LIB})
 # Rust's `staticlib` emits cargo:rustc-link-lib directives for the system libs
 # its std + deps need, but those don't propagate to a C++ consumer's linker —
 # we have to replicate the platform list manually.
-#  - macOS: CoreFoundation/Security/SystemConfiguration for tls + tiny_http.
+#  - macOS: CoreFoundation/Security/SystemConfiguration for tls + tiny_http;
+#    CoreText/Foundation for fontique's system-font enumeration (the native
+#    svg/pdf/hep writers).
 #  - Unix:  standard pthread/dl/m.
 #  - Windows: ntdll (NtCreateNamedPipeFile, used by std::process child pipes),
 #    userenv (GetUserProfileDirectoryW), and bcrypt (BCryptGenRandom, used by
 #    getrandom — pulled in via uuid/rand via arrow). DuckDB's own link list
 #    already covers ws2_32/advapi32/kernel32.
 if(APPLE)
-  set(GGSQL_RUST_SYSDEPS "-framework CoreFoundation" "-framework Security" "-framework SystemConfiguration")
+  set(GGSQL_RUST_SYSDEPS "-framework CoreFoundation" "-framework Security" "-framework SystemConfiguration" "-framework CoreText" "-framework Foundation")
 elseif(WIN32)
   set(GGSQL_RUST_SYSDEPS ntdll userenv bcrypt)
 elseif(UNIX)

README.md

@@ -1,6 +1,6 @@
 # ggsql DuckDB extension
 
-A DuckDB extension that routes `VISUALISE`/`VISUALIZE` statements through the [ggsql](https://ggsql.org) engine and renders vega-lite charts. The chart is served from an in-process HTTP server and opened in your default browser.
+A DuckDB extension that routes `VISUALISE`/`VISUALIZE` statements through the [ggsql](https://ggsql.org) engine and renders plots with ggsql's native hephaestus renderer. The plot is served from an in-process HTTP server and opened in your default browser — the same renderer that produces the `svg`/`pdf`/`hep` output, so what you see is what you save.
 
 ## Building
 
@@ -48,7 +48,22 @@ Use the session setting `ggsql_output` to choose what a query produces:
 | `silent` *(default)* | Opens the default browser; the `VISUALISE` statement produces **no result set at all**. Good for interactive use — you see the plot, the shell doesn't spam a URL at you. |
 | `url` | Opens the browser and returns the plot URL in a 1×1 result. Good for scripts that want the URL. |
 | `spec` | Returns the raw vega-lite JSON as VARCHAR. No HTTP server, no browser. Good for piping to other tools. |
-| `html` | Returns a self-contained HTML document (~830 KB — vega + vega-lite + vega-embed inlined from the vendored bundles, plus the spec). No HTTP server, no browser. Good for saving a shareable snapshot: `COPY (SELECT ggsql('…')) TO 'plot.html'`. |
+| `html` | Returns a self-contained HTML document (~1.9 MB — the .hep plot document plus the hephaestus wasm viewer and Roboto faces inlined). No HTTP server, no browser. Same rendering as the interactive display, so a saved snapshot looks identical: `COPY (SELECT ggsql('…')) TO 'plot.html'`. |
+| `svg` | Returns the plot as SVG text (VARCHAR), rendered natively by ggsql — no vega-lite, no server, no browser. |
+| `pdf` | Returns the plot as PDF bytes. Use the table form `ggsql_run('…')`, which types the column as BLOB. |
+| `hep` | Returns the plot as a `.hep` plot document (ggsql's native format). BLOB from `ggsql_run('…')`. |
+
+The native writers (`svg`/`pdf`/`hep`) — and the `html` and browser display modes, which render through the same pipeline — take per-session options via `ggsql_writer_options`, a semicolon-separated `key=value` string forwarded to ggsql's writer. Shared keys are `width`, `height`, `units`, `dpi`, and `background`, and each writer adds its own (e.g. `embed-fonts` for `svg`). Unknown keys are rejected with an error naming them. Only `spec` mode rejects options outright (it's raw vega-lite JSON, a different backend kept as an escape hatch).
+
+### Saving to a file (`ggsql_save`)
+
+`ggsql_save(query, path)` renders a query straight to a file; the writer is inferred from the extension (`.svg`, `.pdf`, `.hep`, `.html`, `.json` for the raw vega-lite spec), `ggsql_writer_options` applies, and the `ggsql_output` mode is ignored. It returns the path.
+
+```sql
+SET ggsql_writer_options = 'width=800;height=600';
+SELECT ggsql_save('SELECT * FROM range(10) t(x) VISUALISE x, x*x AS y DRAW line', 'plot.svg');
+SELECT ggsql_save('SELECT * FROM range(10) t(x) VISUALISE x, x*x AS y DRAW line', 'plot.pdf');
+```
 
 ```sql
 -- default: just see the plot, no shell output
@@ -67,10 +82,20 @@ SELECT * FROM range(10) t(x) VISUALISE x, x*x AS y DRAW line;
 SET ggsql_output = 'html';
 COPY (SELECT ggsql('SELECT * FROM range(10) t(x) VISUALISE x, x*x AS y DRAW line')) TO 'plot.html';
 
+-- render natively to SVG, sized via writer options
+SET ggsql_output = 'svg';
+SET ggsql_writer_options = 'width=800;height=600';
+SELECT * FROM range(10) t(x) VISUALISE x, x*x AS y DRAW line;
+
+-- get a PDF back as BLOB (use the table form, which types the column as BLOB)
+SET ggsql_output = 'pdf';
+SET ggsql_writer_options = '';
+SELECT plot FROM ggsql_run('SELECT * FROM range(10) t(x) VISUALISE x, x*x AS y DRAW line');
+
 RESET ggsql_output;  -- back to silent
 ```
 
-In `url`/`spec`/`html` modes the result column is named `plot`, so wrapper queries (`SELECT plot FROM …`) keep working when the mode is toggled.
+In every non-silent mode the result column is named `plot`, so wrapper queries (`SELECT plot FROM …`) keep working when the mode is toggled.
 
 ## Session sharing (current limitation)