Skip to content

Read columns through sqlite3_column_* so parallel scans do not touch unprotected values - #262

Merged
staticlibs merged 2 commits into
duckdb:mainfrom
jfo84:fix/parallel-scan-column-reads
Oct 22, 2026
Merged

staticlibs merged 2 commits into
duckdb:mainfrom
jfo84:fix/parallel-scan-column-reads

Conversation

@jfo84

@jfo84 jfo84 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Parallel sqlite_query scans of one database crash or return wrong results. SqliteScan reads each column with sqlite3_column_value() and then the sqlite3_value_*() accessors. That value is unprotected, and SQLite documents the sqlite3_value_*() accessors on an unprotected value as safe only when the connection is not used from another thread. Parallel scans share one connection, so a type conversion (an integer read as text allocates) runs outside the connection mutex while another thread prepares a statement, and SQLite's heap is corrupted. This change reads through sqlite3_column_*(), which take the mutex.

  • SqliteScan, CheckTypeMatches and CheckTypeIsFloatOrInteger read through sqlite3_column_*(). The error paths also handle a NULL text value.
  • test/sql/scanner/parallel_sqlite_query.test runs sixteen sqlite_query scans of one in-memory database at 12 threads, 100 times.

Validation on macOS arm64 (12 cores, load 3–8), DuckDB 0b272c23, duckdb-sqlite b043eb7b with and without this commit, runs alternated:

without the fix with the fix
the new test, unittest 10 of 10 failed: 5 SIGBUS, 4 SIGSEGV, 1 wrong sum (3199838845, expected 3199839440) 10 of 10 passed
the reproduction below, CLI 16 of 20 crashed: 13 SIGSEGV, 2 SIGABRT, 1 SIGBUS 20 of 20 completed

The full sqlite_scanner suite passes with the fix: 69 test cases, 3,029 assertions, 18 skipped.

Reproduction:

SET threads = 12;
SET preserve_insertion_order = false;
ATTACH ':memory:' AS meta (TYPE sqlite);
CREATE TABLE meta.t AS SELECT i FROM range(20000) r(i);
SELECT sum(v) FROM (
    SELECT i::BIGINT AS v FROM sqlite_query('meta', 'SELECT i FROM t WHERE i >= 0')
    UNION ALL SELECT i::BIGINT FROM sqlite_query('meta', 'SELECT i FROM t WHERE i >= 1')
    -- ... sixteen scans in all, i >= 0 through i >= 15
);

Run the SELECT repeatedly. The expected sum is 3199839440.

@staticlibs

Copy link
Copy Markdown
Member

Hi, thanks for the PR and for the quick update! I wonder if you will be able to rebase the PR on the latest main after #264 is integrated (in 10-15 minutes, I will add another comment when it is ready) - it should make the CI to run much faster as GH runners are currently overloaded.

@staticlibs

Copy link
Copy Markdown
Member

Ready now, please rebase the PR onto main. Also, where possible, please remove Claude from commit authors.

jfo84 added 2 commits October 22, 2026 10:27
… touch unprotected values

SqliteScan read each column with sqlite3_column_value() and then the
sqlite3_value_*() accessors. The value is unprotected: SQLite documents
that sqlite3_value_*() on it is only safe when the connection is not used
from another thread. Parallel sqlite_query scans share one connection, so
a type conversion (an integer read as text allocates) runs without the
connection mutex while another thread prepares a statement, corrupting
SQLite's heap. sqlite3_column_*() take the mutex.

The test runs sixteen sqlite_query scans of one in-memory database at
twelve threads. Without the fix it crashes (SIGSEGV, SIGBUS, SIGABRT) or
returns a wrong sum.
@jfo84
jfo84 force-pushed the fix/parallel-scan-column-reads branch from c333d8e to 8ba69f7 Compare October 22, 2026 14:28
@jfo84

jfo84 commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! Rebased onto the latest main and removed the Claude co-author trailers from both commits. The source patch is unchanged. New head: 8ba69f7.

@staticlibs staticlibs left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR looks good to me.

@staticlibs
staticlibs merged commit c86d3cc into duckdb:main Oct 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants