[adubovikov]

Requested by Alex

Problem

SQLiteTransaction opens its own sqlite3* handle and closes it after commit. For an on-disk database in WAL mode, every such close is the last connection, so SQLite runs checkpoint-on-close after every transaction: fsync of the WAL and of the main file, then unlink of -wal/-shm, which the next transaction creates again. On a DuckLake metadata catalog stored in SQLite this means ~5 fsyncs per snapshot. DuckDB 2.0.0 behaves the same way.

Related:

• There is no way to set synchronous. DuckLake forwards META_SYNCHRONOUS, which is rejected with Unsupported parameter for SQLite Attach: synchronous.
• JOURNAL_MODE is spliced into PRAGMA journal_mode=... unquoted, so JOURNAL_MODE 'WAL; PRAGMA synchronous=NORMAL' runs both statements.

Change

1. Opt-in (SET sqlite_keep_database_open = true, read at ATTACH; default false keeps the existing behaviour): SQLiteCatalog keeps one anchor handle open for local on-disk files until DETACH. The handle reads sqlite_master once, because SQLite counts a connection as a WAL participant only after a read. Auto-checkpoint keeps the WAL bounded, and the final checkpoint and -wal removal happen once, at DETACH. The anchor is not used for :memory:, remote paths or VFS-backed paths.
2. New ATTACH option SYNCHRONOUS (OFF|NORMAL|FULL|EXTRA or 0..3, case-insensitive), applied to every handle.
3. JOURNAL_MODE and SYNCHRONOUS are validated against SQLite's keyword lists, which closes the multi-statement injection.

Tests

• New test/sql/storage/attach_wal_lifetime.test, plus new cases in attach_options.test.
• Full suite on main (DuckDB v2.0.0): all tests passed (3251 assertions in 72 test cases, 18 skipped for httpfs / TPCH env).

Benchmark

300 single-row INSERTs into a DuckLake table (300 snapshots), SQLite catalog with META_JOURNAL_MODE 'WAL', local NVMe, strace -f -e trace=fsync,unlink:

build	fsync	-wal unlinks	wall time
stock 1.5.5	1514	303	5.3–6.4 s
stock 2.0.0	1514	303	5.0–5.4 s
1.5.5 + this patch (sqlite_keep_database_open)	618	1 (at DETACH)	4.4–4.6 s
1.5.5 + this patch + META_SYNCHRONOUS 'NORMAL'	19	1	3.5–4.2 s

The same commit applies to v1.5-variegata (it was written there first), and I can open a backport if this is accepted.

👹 Fixed by Goblin

[staticlibs]

Hi, thanks for the PR! I think this change is useful, just with SQLite extension being used widely, I suspect some of its usage depends on the "close-and-reopen for every transaction" behaviour. Can we add an extension option whether to close or not the DB on transaction finish? And by default keep the existing behaviour, making the new logic opt-in.

New ATTACH options look good to me.

Nit: use {TEST_DIR} substitution syntax in tests instead of __TEST_DIR__.

To pass the formatting check run make format with these packages installed.

[adubovikov]

Thanks for the review! Done in 9b392c1:

• Keeping the handle open is now opt-in via a new extension setting sqlite_keep_database_open (default false, read at ATTACH), so the existing close-on-transaction-end behaviour is unchanged by default.
• Tests use {TEST_DIR}; attach_wal_lifetime.test now also checks the default behaviour.
• make format applied.