Repository navigation
Read columns through sqlite3_column_* so parallel scans do not touch unprotected values - #262
Merged
Merged
Conversation
Member
|
Hi, thanks for the PR and for the quick update! I wonder if you will be able to rebase the PR on the latest main after #264 is integrated (in 10-15 minutes, I will add another comment when it is ready) - it should make the CI to run much faster as GH runners are currently overloaded. |
Member
|
Ready now, please rebase the PR onto |
… touch unprotected values SqliteScan read each column with sqlite3_column_value() and then the sqlite3_value_*() accessors. The value is unprotected: SQLite documents that sqlite3_value_*() on it is only safe when the connection is not used from another thread. Parallel sqlite_query scans share one connection, so a type conversion (an integer read as text allocates) runs without the connection mutex while another thread prepares a statement, corrupting SQLite's heap. sqlite3_column_*() take the mutex. The test runs sixteen sqlite_query scans of one in-memory database at twelve threads. Without the fix it crashes (SIGSEGV, SIGBUS, SIGABRT) or returns a wrong sum.
jfo84
force-pushed
the
fix/parallel-scan-column-reads
branch
from
October 22, 2026 14:28
c333d8e to
8ba69f7
Compare
Contributor
Author
|
Thanks! Rebased onto the latest |
This was referenced Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallel
sqlite_queryscans of one database crash or return wrong results.SqliteScanreads each column withsqlite3_column_value()and then thesqlite3_value_*()accessors. That value is unprotected, and SQLite documents thesqlite3_value_*()accessors on an unprotected value as safe only when the connection is not used from another thread. Parallel scans share one connection, so a type conversion (an integer read as text allocates) runs outside the connection mutex while another thread prepares a statement, and SQLite's heap is corrupted. This change reads throughsqlite3_column_*(), which take the mutex.SqliteScan,CheckTypeMatchesandCheckTypeIsFloatOrIntegerread throughsqlite3_column_*(). The error paths also handle a NULL text value.test/sql/scanner/parallel_sqlite_query.testruns sixteensqlite_queryscans of one in-memory database at 12 threads, 100 times.Validation on macOS arm64 (12 cores, load 3–8), DuckDB
0b272c23, duckdb-sqliteb043eb7bwith and without this commit, runs alternated:unittest3199838845, expected3199839440)The full sqlite_scanner suite passes with the fix: 69 test cases, 3,029 assertions, 18 skipped.
Reproduction:
Run the
SELECTrepeatedly. The expected sum is3199839440.