Begin migration to DuckDB 2.0 alpha and upgrade ggsql to 0.5.2

.github/workflows/MainDistributionPipeline.yml

@@ -4,29 +4,34 @@
 name: Main Extension Distribution Pipeline
 on:
   push:
+    branches: [main]
+    tags: ['v*']
   pull_request:
   workflow_dispatch:
 
 concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || '' }}-${{ github.base_ref || '' }}-${{ github.ref != 'refs/heads/main' && github.sha || '' }}
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
   cancel-in-progress: true
 
 jobs:
-  duckdb-stable-build:
-    name: Build extension binaries
-    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5.4
+  duckdb-v2-build:
+    name: Build DuckDB 2.0 alpha extension binaries
+    uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@20bad04ce694ea3f4052efb51c1a177dc9891bfd
     with:
-      duckdb_version: v1.5.4
-      ci_tools_version: v1.5.4
+      # Keep these immutable refs aligned with the submodules.
+      duckdb_version: d8a1bd4f4fbccdf3d22d8f1fe27a94c11c805d0a
+      ci_tools_version: 20bad04ce694ea3f4052efb51c1a177dc9891bfd
       extension_name: ggsql
       extra_toolchains: 'rust'
       exclude_archs: 'wasm_mvp;wasm_eh;wasm_threads'
+      test_config: '{"test_env_variables":{"GGSQL_NO_OPEN_BROWSER":"1"}}'
 
   code-quality-check:
     name: Code Quality Check
-    uses: duckdb/extension-ci-tools/.github/workflows/_extension_code_quality.yml@v1.5.4
+    uses: duckdb/extension-ci-tools/.github/workflows/_extension_code_quality.yml@20bad04ce694ea3f4052efb51c1a177dc9891bfd
     with:
-      duckdb_version: v1.5.4
-      ci_tools_version: v1.5.4
+      duckdb_version: d8a1bd4f4fbccdf3d22d8f1fe27a94c11c805d0a
+      ci_tools_version: 20bad04ce694ea3f4052efb51c1a177dc9891bfd
       extension_name: ggsql
+      extra_toolchains: 'rust'
       format_checks: 'format;tidy'

.gitmodules

@@ -1,8 +1,8 @@
 [submodule "duckdb"]
 	path = duckdb
 	url = https://github.com/duckdb/duckdb
-	branch = main
+	branch = v2.0-cyanoptera
 [submodule "extension-ci-tools"]
 	path = extension-ci-tools
 	url = https://github.com/duckdb/extension-ci-tools
-	branch = main
+	branch = main

CLAUDE.md

@@ -20,7 +20,7 @@ rust/                   Rust staticlib linked into the extension
   include/ggsql_ext_rs.h  Hand-rolled C header; must track ffi.rs
   assets/               Vendored vega/vega-lite/vega-embed bundles + SPA shell
 test/sql/ggsql.test     SQL logic tests — run with GGSQL_NO_OPEN_BROWSER=1
-duckdb/, extension-ci-tools/  Submodules; versions bumped per release (see docs/UPDATING.md)
+duckdb/, extension-ci-tools/  Pinned submodules (see docs/DUCKDB_2_MIGRATION.md)
 ```
 
 ## Build / test
@@ -29,13 +29,13 @@ duckdb/, extension-ci-tools/  Submodules; versions bumped per release (see docs/
 - `make test` — runs SQL logic tests. Set `GGSQL_NO_OPEN_BROWSER=1` to stop Rust from launching a browser tab per test.
 - CMake drives `cargo build --release` via a custom command; Rust sources + `assets/*` are listed as dependencies so edits trigger rebuilds.
 - macOS links `CoreFoundation`, `Security`, `SystemConfiguration` (needed by Rust std / tiny_http TLS bits).
-- DuckDB target version is set in three places — keep them in sync: `.github/workflows/MainDistributionPipeline.yml` (`duckdb_version`, `ci_tools_version`, workflow `@` tag) and the `duckdb` + `extension-ci-tools` submodule refs. `docs/UPDATING.md` has the full checklist.
+- DuckDB and CI-tool refs are pinned in `.github/workflows/MainDistributionPipeline.yml` (`duckdb_version`, `ci_tools_version`, workflow `@` refs) and the `duckdb` + `extension-ci-tools` submodules. Keep each workflow ref aligned with its matching submodule. See `docs/DUCKDB_2_MIGRATION.md` for this branch's pins and validation.
 
 ## User-visible surface
 
 Two entry points, both funnel into the same Rust `ggsql_execute`:
 
-- **ParserExtension** — any statement containing a top-level `VISUALISE`/`VISUALIZE` keyword is claimed and planned as a call to the `ggsql_run` table function. The scanner in `ggsql_parser.cpp` is hand-rolled: it skips `'...'`, `"..."`, `--` line comments and `/* */` block comments, and matches only at word boundaries. A trailing `;` is stripped because ggsql's tree-sitter grammar rejects it.
+- **ParserExtension** — DuckDB 2.0 supplies `SimpleToken` values. The callback claims one statement containing a top-level `VISUALISE`/`VISUALIZE` keyword and reports its consumed token count. It reconstructs ggsql source from raw token spellings, preserving namespaced datasets and signed literals, and excludes the trailing terminator. Quoted identifiers, string literals, and nested expressions do not trigger the extension.
 - **Scalar** — `SELECT ggsql('<query>')` runs the same pipeline with the string as input.
 
 Output mode is session-scoped via the `ggsql_output` setting (`silent` default / `url` / `spec` / `html`). The result column is always named `plot` — don't rename it. Unknown values throw at bind time. `silent` emits zero rows; `url` emits one; `spec`/`html` return the bytes and do not start the HTTP server or open a browser.
@@ -70,7 +70,7 @@ The inner `Connection` is created lazily and **persists for the whole `ggsql_exe
 
 ## Inlined `DuckDbDialect`
 
-`rust/src/dialect.rs` carries a verbatim copy of ggsql's `DuckDbDialect` (currently from ggsql 0.4.1's `src/reader/duckdb.rs`, with the `pub(crate)` `wrap_with_column_aliases` helper inlined). We can't enable ggsql's `duckdb` feature because it pulls in `duckdb-rs` with `bundled`, which would statically link a second DuckDB into an extension already loaded inside DuckDB (symbol clashes + binary bloat). If upstream changes the dialect, re-sync manually.
+`rust/src/dialect.rs` carries a copy of ggsql's `DuckDbDialect` (verified against ggsql 0.5.2's `src/reader/duckdb.rs`, with the `pub(crate)` `wrap_with_column_aliases` helper inlined). We can't enable ggsql's `duckdb` feature because it pulls in `duckdb-rs` with `bundled`, which would statically link a second DuckDB into an extension already loaded inside DuckDB (symbol clashes + binary bloat). If upstream changes the dialect, re-sync manually.
 
 ## Conventions
 

CMakeLists.txt

@@ -1,4 +1,4 @@
-cmake_minimum_required(VERSION 3.5)
+cmake_minimum_required(VERSION 3.14...3.29)
 
 # Set extension name here
 set(TARGET_NAME ggsql)
@@ -8,17 +8,10 @@ set(LOADABLE_EXTENSION_NAME ${TARGET_NAME}_loadable_extension)
 
 project(${TARGET_NAME})
 
-include_directories(
-  src/include
-  rust/include
-)
+include_directories(src/include rust/include)
 
-set(EXTENSION_SOURCES
-  src/ggsql_extension.cpp
-  src/ggsql_bridge.cpp
-  src/ggsql_parser.cpp
-  src/ggsql_exec.cpp
-)
+set(EXTENSION_SOURCES src/ggsql_extension.cpp src/ggsql_bridge.cpp
+                      src/ggsql_parser.cpp src/ggsql_exec.cpp)
 
 build_static_extension(${TARGET_NAME} ${EXTENSION_SOURCES})
 build_loadable_extension(${TARGET_NAME} " " ${EXTENSION_SOURCES})
@@ -31,64 +24,67 @@ set(GGSQL_RUST_BUILD_DIR ${CMAKE_CURRENT_BINARY_DIR}/rust-target)
 
 find_program(CARGO_EXECUTABLE cargo REQUIRED)
 
-# extension-ci-tools' Makefile passes -DRust_CARGO_TARGET=<triple> for
-# macOS cross-builds and Windows mingw (matching the Corrosion convention).
-# Honor it: pass --target to cargo and adjust the output path accordingly.
-# Without --target, cargo builds for the host arch and the final link fails on
-# osx_amd64 (arm64 runner → x86_64 binary) with "ignoring file ... found
-# architecture arm64, required architecture x86_64".
+# extension-ci-tools' Makefile passes -DRust_CARGO_TARGET=<triple> for macOS
+# cross-builds and Windows mingw (matching the Corrosion convention). Honor it:
+# pass --target to cargo and adjust the output path accordingly. Without
+# --target, cargo builds for the host arch and the final link fails on osx_amd64
+# (arm64 runner → x86_64 binary) with "ignoring file ... found architecture
+# arm64, required architecture x86_64".
 #
 # Cargo's staticlib filename varies by target: `lib<name>.a` on Unix and
-# Windows-gnu, `<name>.lib` on Windows-MSVC. CMake's per-platform defaults
-# track the C++ target identically, so reuse them.
-set(GGSQL_RUST_LIB_FILENAME "${CMAKE_STATIC_LIBRARY_PREFIX}ggsql_ext_rs${CMAKE_STATIC_LIBRARY_SUFFIX}")
+# Windows-gnu, `<name>.lib` on Windows-MSVC. CMake's per-platform defaults track
+# the C++ target identically, so reuse them.
+set(GGSQL_RUST_LIB_FILENAME
+    "${CMAKE_STATIC_LIBRARY_PREFIX}ggsql_ext_rs${CMAKE_STATIC_LIBRARY_SUFFIX}")
 if(Rust_CARGO_TARGET)
   set(GGSQL_CARGO_TARGET_ARGS --target ${Rust_CARGO_TARGET})
-  set(GGSQL_RUST_LIB ${GGSQL_RUST_BUILD_DIR}/${Rust_CARGO_TARGET}/release/${GGSQL_RUST_LIB_FILENAME})
+  set(GGSQL_RUST_LIB
+      ${GGSQL_RUST_BUILD_DIR}/${Rust_CARGO_TARGET}/release/${GGSQL_RUST_LIB_FILENAME}
+  )
 else()
   set(GGSQL_CARGO_TARGET_ARGS "")
   set(GGSQL_RUST_LIB ${GGSQL_RUST_BUILD_DIR}/release/${GGSQL_RUST_LIB_FILENAME})
 endif()
 
 # Cargo's stamp file is unreliable; list the Rust sources as the dependency set.
-file(GLOB_RECURSE GGSQL_RUST_SOURCES
+file(
+  GLOB_RECURSE
+  GGSQL_RUST_SOURCES
   ${GGSQL_RUST_DIR}/src/*.rs
   ${GGSQL_RUST_DIR}/include/*.h
   ${GGSQL_RUST_DIR}/assets/*.js
   ${GGSQL_RUST_DIR}/assets/*.html
   ${GGSQL_RUST_DIR}/Cargo.toml
-)
+  ${GGSQL_RUST_DIR}/Cargo.lock)
 
 # tree-sitter-ggsql's build.rs re-runs `tree-sitter generate` from grammar.js
-# unless GGSQL_SKIP_GENERATE=1; the published crate already ships the
-# generated src/parser.c, so setting this avoids a hard dep on the tree-sitter
-# CLI in CI and clean builds.
+# unless GGSQL_SKIP_GENERATE=1; the published crate already ships the generated
+# src/parser.c, so setting this avoids a hard dep on the tree-sitter CLI in CI
+# and clean builds.
 add_custom_command(
   OUTPUT ${GGSQL_RUST_LIB}
-  COMMAND ${CMAKE_COMMAND} -E env GGSQL_SKIP_GENERATE=1
-          ${CARGO_EXECUTABLE} build --release
-          --manifest-path ${GGSQL_RUST_DIR}/Cargo.toml
-          --target-dir ${GGSQL_RUST_BUILD_DIR}
-          ${GGSQL_CARGO_TARGET_ARGS}
+  COMMAND
+    ${CMAKE_COMMAND} -E env GGSQL_SKIP_GENERATE=1 ${CARGO_EXECUTABLE} build
+    --release --locked --manifest-path ${GGSQL_RUST_DIR}/Cargo.toml --target-dir
+    ${GGSQL_RUST_BUILD_DIR} ${GGSQL_CARGO_TARGET_ARGS}
   DEPENDS ${GGSQL_RUST_SOURCES}
   WORKING_DIRECTORY ${GGSQL_RUST_DIR}
   COMMENT "Building ggsql_ext_rs (release)"
-  VERBATIM
-)
+  VERBATIM)
 
 add_custom_target(ggsql_rust_lib DEPENDS ${GGSQL_RUST_LIB})
 
 # Rust's `staticlib` emits cargo:rustc-link-lib directives for the system libs
-# its std + deps need, but those don't propagate to a C++ consumer's linker —
-# we have to replicate the platform list manually.
-#  - macOS: CoreFoundation/Security/SystemConfiguration for tls + tiny_http.
-#  - Unix:  standard pthread/dl/m.
-#  - Windows: ntdll (NtCreateNamedPipeFile, used by std::process child pipes),
-#    userenv (GetUserProfileDirectoryW), and bcrypt (BCryptGenRandom, used by
-#    getrandom — pulled in via uuid/rand via arrow). DuckDB's own link list
-#    already covers ws2_32/advapi32/kernel32.
+# its std + deps need, but those don't propagate to a C++ consumer's linker — we
+# have to replicate the platform list manually. - macOS:
+# CoreFoundation/Security/SystemConfiguration for tls + tiny_http. - Unix:
+# standard pthread/dl/m. - Windows: ntdll (NtCreateNamedPipeFile, used by
+# std::process child pipes), userenv (GetUserProfileDirectoryW), and bcrypt
+# (BCryptGenRandom, used by getrandom — pulled in via uuid/rand via arrow).
+# DuckDB's own link list already covers ws2_32/advapi32/kernel32.
 if(APPLE)
-  set(GGSQL_RUST_SYSDEPS "-framework CoreFoundation" "-framework Security" "-framework SystemConfiguration")
+  set(GGSQL_RUST_SYSDEPS "-framework CoreFoundation" "-framework Security"
+                         "-framework SystemConfiguration")
 elseif(WIN32)
   set(GGSQL_RUST_SYSDEPS ntdll userenv bcrypt)
 elseif(UNIX)
@@ -106,5 +102,4 @@ install(
   TARGETS ${EXTENSION_NAME}
   EXPORT "${DUCKDB_EXPORT_SET}"
   LIBRARY DESTINATION "${INSTALL_LIB_DIR}"
-  ARCHIVE DESTINATION "${INSTALL_LIB_DIR}"
-)
+  ARCHIVE DESTINATION "${INSTALL_LIB_DIR}")

Makefile

@@ -6,31 +6,3 @@ EXT_CONFIG=${PROJ_DIR}extension_config.cmake
 
 # Include the Makefile from extension-ci-tools
 include extension-ci-tools/makefiles/duckdb_extension.Makefile
-
-# ---------------------------------------------------------------------------
-# Local formatter shims
-#
-# duckdb/scripts/format.py hard-requires clang-format 11.x and cmake-format.
-# Point it at `clang-format@11` from Homebrew (CI pins the same version), and
-# stub cmake-format since we have no CMake files under src/test to format.
-# ---------------------------------------------------------------------------
-FORMAT_TOOLS_DIR := $(PROJ_DIR).format-tools/bin
-CLANG_FORMAT_11  := $(shell brew --prefix clang-format@11 2>/dev/null)/bin/clang-format-11
-
-.PHONY: format-fix format-check _format-tools
-
-_format-tools:
-	@if [ ! -x "$(CLANG_FORMAT_11)" ]; then \
-	  echo "clang-format 11 not found. Install it with: brew install clang-format@11"; \
-	  exit 1; \
-	fi
-	@mkdir -p $(FORMAT_TOOLS_DIR)
-	@ln -sf $(CLANG_FORMAT_11) $(FORMAT_TOOLS_DIR)/clang-format
-	@printf '#!/bin/sh\necho "cmake-format 0.6.0"\n' > $(FORMAT_TOOLS_DIR)/cmake-format
-	@chmod +x $(FORMAT_TOOLS_DIR)/cmake-format
-
-format-fix: _format-tools
-	PATH="$(FORMAT_TOOLS_DIR):$$PATH" python3 duckdb/scripts/format.py --all --fix --noconfirm --directories src test
-
-format-check: _format-tools
-	PATH="$(FORMAT_TOOLS_DIR):$$PATH" python3 duckdb/scripts/format.py --all --check --directories src test

README.md

@@ -2,9 +2,18 @@
 
 A DuckDB extension that routes `VISUALISE`/`VISUALIZE` statements through the [ggsql](https://ggsql.org) engine and renders vega-lite charts. The chart is served from an in-process HTTP server and opened in your default browser.
 
+## DuckDB 2.0 migration branch
+
+This branch targets a pinned DuckDB 2.0 **alpha** commit and ggsql **0.5.2**.
+It is a development build; it is not the version installed by
+`INSTALL ggsql FROM community`. The native extension must be loaded by a DuckDB
+build with the matching ABI. See [the migration notes](docs/DUCKDB_2_MIGRATION.md)
+for exact dependency refs, compatibility changes, and validation instructions.
+
 ## Building
 
 ```sh
+git submodule update --init --recursive
 make
 ```
 

docs/DUCKDB_2_MIGRATION.md

@@ -0,0 +1,99 @@
+# DuckDB 2.0 migration
+
+This branch starts the migration from DuckDB 1.5.4 and ggsql 0.4.1 to the
+DuckDB 2.0 alpha API and ggsql 0.5.2. It does not claim compatibility with an
+unreleased final DuckDB 2.0 ABI or publish an update to the community repository.
+
+## Reproducible dependencies
+
+| Dependency | Pin |
+| --- | --- |
+| DuckDB | `d8a1bd4f4fbccdf3d22d8f1fe27a94c11c805d0a` from `v2.0-cyanoptera` |
+| extension-ci-tools | `20bad04ce694ea3f4052efb51c1a177dc9891bfd` |
+| ggsql | `=0.5.2`, latest published non-yanked release checked on 2026-09-26 |
+| Arrow Rust | Major 58, compatible with ggsql 0.5.2; exact resolution in `rust/Cargo.lock` |
+
+The submodule commits and CI workflow refs must stay aligned. The CMake build
+uses `cargo build --locked` and rebuilds the Rust library when `Cargo.lock`
+changes. Updating the upstream `v2.0-cyanoptera` branch does not silently change
+this checkout's pinned dependency.
+
+## Compatibility changes
+
+- Migrate the parser callback from SQL text to DuckDB 2.0's `SimpleToken` API,
+  including reporting the exact number of tokens consumed for one statement.
+- Reconstruct source while preserving ggsql's namespaced datasets and signed
+  numeric literals. Native tokens omit whitespace and comments; the scalar
+  `ggsql('<source>')` entry point continues to receive the original query string.
+- Use `Identifier` for table-function output names, retaining the `plot` column.
+- Mark the scalar function fallible and volatile: query errors are ordinary
+  runtime errors, and prepared calls observe current data and output settings.
+- Keep the C++ Arrow-stream bridge and sibling connection model. Session-local
+  temporary tables and registered relations remain unavailable; this migration
+  does not change that documented limitation.
+- Update ggsql and the bundled Vega renderer to agree on the Vega-Lite schema.
+  The inlined DuckDB dialect is checked against the pinned ggsql release.
+- Use the current extension CI tooling and its portable formatter targets.
+  Distribution tests set `GGSQL_NO_OPEN_BROWSER=1`.
+
+Direct `VISUALISE` statements pass through DuckDB's tokenizer. Use SQL `--` or
+`/* ... */` comments and double-quoted identifiers in that form. For ggsql-only
+spellings such as `//` line comments or backtick-quoted identifiers, use the
+scalar `ggsql('<source>')` entry point so the ggsql parser receives the source
+without token normalization.
+
+One upstream ggsql 0.5.2 edge case remains: a SQL string literal containing `--`
+(for example `'semi;colon -- VISUALISE'`) is rejected by the ggsql parser in both
+the direct and scalar entry points. The migration tests cover embedded
+semicolons and keywords separately from that unsupported comment-marker case.
+
+## Build and validation
+
+Prerequisites: a C++ compiler, CMake 3.14 or newer, Make (optionally Ninja),
+Python, and a Rust toolchain capable of building the locked dependencies.
+
+```sh
+git submodule update --init --recursive
+make
+GGSQL_NO_OPEN_BROWSER=1 make test
+GGSQL_SKIP_GENERATE=1 cargo check --locked --manifest-path rust/Cargo.toml
+```
+
+The SQL suites cover output modes, normal error recovery, prepared scalar
+execution, committed in-memory data through the Arrow bridge, CTEs, parser
+token boundaries, quoted text, dataset namespaces, and signed literals.
+CI builds the supported native platforms; WebAssembly remains excluded because
+the existing HTTP server, threading, and browser-launch dependencies require a
+native host.
+
+### Validation performed on 2026-09-26
+
+- Linux x86-64 release build completed for DuckDB, the shell, the test runner,
+  and the loadable extension using GCC 15 and Rust 1.98.1.
+- After the final parser fix, the rebuilt loadable binary passed all three SQL
+  suites: 63 assertions (19 core, 17 embedding, 27 parser). Each fresh runner
+  database checked that ggsql was unloaded, loaded the artifact by its full path,
+  and checked that it was loaded. Temporary copies replaced `require ggsql` with
+  explicit `LOAD` so these checks exercised the final dynamic library.
+- A separate dynamic-load smoke test passed seven assertions.
+- `cargo check --locked` passed; the dependency tree contains no second DuckDB.
+- The actual ggsql line-and-point specification compiled with the bundled Vega
+  stack and rendered to SVG. Its standalone HTML also rendered in a browser
+  without JavaScript warnings or errors.
+- C++/SQL/CMake formatting checks and `git diff --check` passed. GitHub's format
+  and clang-tidy checks passed during the migration; CI reruns on the final push.
+
+Native Windows/macOS builds and the final DuckDB 2.0 release remain unverified.
+
+## Before release
+
+- Rebase the pins onto the intended final DuckDB 2.0 release and rerun all builds.
+- Validate Windows and macOS in CI in addition to local Linux testing.
+- Smoke-test browser rendering and self-contained HTML using the bundled assets.
+- For community alpha testing, submit this extension commit as `repo.ref_next`
+  in the community descriptor; do not replace the stable `repo.ref` prematurely.
+- Publish only after those checks pass. No community submission is made by this
+  migration branch.
+
+References: [DuckDB 2.0 alpha announcement](https://duckdb.org/2026/09/02/try-duckdb-20-alpha),
+[ggsql 0.5.2](https://crates.io/crates/ggsql/0.5.2).