[xproblematique]

Closes https://github.com/duckdblabs/duckdb-internal/issues/11567

This is the first PR in a series to make DuckLake's metadata catalogs pluggable. The SQLite-, Postgres- and MySQL-specific code in DuckLake moves into the matching extensions, so that DuckLake itself has no catalog-specific logic (duckdb/ducklake#882). Following the direction in duckdb/ducklake#1126, catalogs would register themselves with DuckLake rather than being built into it. We are starting with SQLite.

Part of DuckLake's SQLite code exists only because sqlite_scan pushes no filters into SQLite. The main case is the stats override from duckdb/ducklake#1814. Without pushdown, every filtered DuckLake read copied all of ducklake_file_column_stats into DuckDB, so SQLiteMetadataManager now reads the stats through sqlite_query instead. With this PR, DuckLake's generic stats query is filtered inside SQLite, and that override can go.

Planned next steps:

1. ducklake: bump the sqlite_scanner pin and delete the stats override
2. ducklake: turn the remaining SQLite overrides into a data profile & add a registration call
3. duckdb-sqlite: register the SQLite profile from this extension, so that DuckLake has no SQLite code left

---

Changes

Scans of attached SQLite tables (ATTACH ... (TYPE SQLITE)) now take pushed-down filters. sqlite_scan() and sqlite_attach() are unchanged.

Filters go into the WHERE clause of the SQLite query only where SQLite compares values exactly like DuckDB:

• BIGINT columns and the rowid. The rowid is addressed through the table's rowid name from 60e4170, so a column named rowid does not hide it
• VARCHAR columns with TEXT affinity in a UTF-8 database. They are compared with COLLATE BINARY, so a NOCASE column returns the same rows as it would in DuckDB

The other pushed-down filters are evaluated by DuckDB inside the scan. This uses TableFilterState and ColumnSegment::FilterSelection, as the Parquet reader does. It applies to:

• DOUBLE, DATE, TIMESTAMP and BLOB columns. SQLite has no NaN, and it stores dates and timestamps in several formats
• Strings in columns without TEXT affinity, where SQLite converts a numeric string constant to a number
• Strings in UTF-16 databases, which SQLite orders by their UTF-16 bytes
• Every column under sqlite_all_varchar

Filters with a NUL byte in a string constant also stay in DuckDB as SQLite stops reading the query text at the first NUL.

Pushdown is on by default. SET sqlite_enable_filter_pushdown = false turns it off.

Behaviour change: rows that SQLite filters out are never read. A value that doesn't match its column's declared type no longer raises a type-mismatch error when the filter excludes its row. Rows that pass the filter are checked as before.

Known cost: a column used only in a filter is still selected from SQLite because the scan does not set filter_prune. This can be a follow-up.

Why not supports_pushdown_type

The usual way to keep some columns' filters in DuckDB is TableFunction::supports_pushdown_type. As of v2.0.0 that planner path has two bugs:

• Extra output column. When a filter is declined on a scan with projection_ids, the filter column is added to the scan output and never removed. It then fails with INTERNAL Error: Attempted to access index 3 within vector of size 3 Fixed by Fix volatile filter pushdown, constant_or_null folding, statistics and projection placement duckdb#27757
• Rowid crash. A declined rowid filter crashes in virtual_columns.at when the function has no get_virtual_columns

Evaluating the declined filters inside the scan avoids these bugs. I'll open follow-up issues for the rowid bug.

Testing

• test/sql/storage/attach_filter_pushdown.test covers:

  • every pushed type, with EXPLAIN checks;
  • filters split between SQLite and DuckDB, and chunks that DuckDB filters out completely;
  • the rowid, and a column named rowid;
  • join filters;
  • NOCASE collation and NUMERIC affinity;
  • a UTF-16 database, using the new fixture data/db/utf16.db;
  • NUL bytes, the setting, and sqlite_all_varchar.

  The test fails if any one of these five safeguards is removed: COLLATE BINARY, the TEXT-affinity check, the encoding check, the all_varchar check, and in-scan evaluation.

[staticlibs]

Looks good to me!