Repository navigation
[Parquet][Python][Dataset] dataset(...).to_table() sometimes returns a truncated table instead of raising when one listed Parquet fragment cannot be opened (freshly written files) #51495
Description
Describe the bug, including details regarding any error messages, version, and platform.
Summary
A pyarrow.dataset built from an explicit list of Parquet files, one of which is not a valid Parquet file, is
expected to raise on to_table(). It usually does. Occasionally it instead returns a table containing only the
rows of the file(s) that sort before the invalid fragment, with no exception. It happens only when the files were
written immediately before the scan (a fresh directory per iteration); reusing the same files never reproduced it.
Seen on Windows and on Linux, threaded and unthreaded.
Environment
| Windows | Linux | |
|---|---|---|
| pyarrow | 23.0.1 | 23.0.1 |
| Python | 3.11.1 | 3.11.15 |
| OS | Windows 11, build 26200 (platform.platform() → Windows-10-10.0.26200-SP0) | Ubuntu 24.04.4 LTS (x86_64; a cloud workspace, the same class on every run) |
| filesystem | NTFS (local disk, the system temp directory) | ext4 on a virtual disk, the system temp directory |
What was checked
- The file list is complete when the scan is short. On a truncated read, the listed paths included the invalid file and the dataset held all 5 fragments (fragment count = number of listed files) — the scan stopped early and returned without error. A fragment-count check before the scan therefore cannot detect it.
- A 0-byte invalid file in the same position raises too (in the same test shape) — the truncation is not an empty file being skipped.
- Threading is not the variable: it reproduces with
use_threads=Falseon both platforms. - Freshness matters: 0 truncations in 3,600 reads of reused files, against the fresh-file rates above. Why fresh files matter is not understood.
- The shape of the failing fragment matters: the 25-byte non-Parquet file truncates; a half-written Parquet file raised every time in 6,000 tries. Not understood.
What remains as candidates: the dataset scanner's handling of a fragment-open error, or something in the filesystem beneath it for files closed moments earlier — the two cannot be told apart from here.
Workaround in use
Opening every listed file's footer independently before the scan (pyarrow.parquet.ParquetFile(path).metadata)
turns the silent truncation into a raise (0 truncations in 2,000 tries on the Windows reproduction); comparing the
rows read with the rows the footers declare backstops the case where a footer opens but its data does not.
Component(s)
Python, Parquet
Activity
[-][Python][Dataset] dataset(...).to_table() sometimes returns a truncated table instead of raising when one listed Parquet fragment cannot be opened (freshly written files)[/-][+][Parquet][Python][Dataset] dataset(...).to_table() sometimes returns a truncated table instead of raising when one listed Parquet fragment cannot be opened (freshly written files)[/+]kita-renji commentedon Sep 26, 2026
I can reproduce this on Linux with pyarrow 23.0.1 and 25.0.1, and I think I've found the cause. It's a race in
MergedGenerator, not the filesystem.On a plain
LocalFileSystemI couldn't hit it in ~15k iterations. Routing the same files through apyarrow.fs.PyFileSystemwrapper (which adds thread handoffs) makes it show up at roughly 1-3% with 8 files and one 25-byte bad file. That's with bothuse_threads=FalseandTrue, and even withfragment_readahead=1.The root cause is in
cpp/src/arrow/util/async_generator.h. When an inner subscription (here: the fragment whoseGetReaderAsyncfails) delivers an error while no caller is waiting,InnerCallbacksetsstate->broken = trueunder the mutex, releases it, and only then stores the error inMarkFinalError()(final_error = err). If the consumer callsoperator()in that window, it seesbrokenwithfinal_errorstill OK and gets a plain end-of-stream. The source node finishes normally, the plan completes OK, andToTable()returns whatever batches were already delivered.OuterCallbackhas the same pattern.This also fits the "shape" observation. The unlocked path is only taken when nobody is waiting, i.e. while the consumer is still processing the previous batch. A 25-byte file fails its footer check almost immediately after that batch is handed off, so the two collide. A larger half-written file fails later, when the consumer is usually already waiting, and that path delivers the error safely.
A standalone C++ loop hits it naturally about 1 in 200k runs. With a 200µs sleep injected before the
final_errorstore it goes to 1990/2000. For pyarrow, I rebuiltlibarrow_dataset.sofrom apache-arrow-25.0.1 and swapped it into the 25.0.1 wheel: 57/4500 truncated with the unmodified source, 0/11.6k with the fix.The fix is small: store
final_errorwhile still holding the lock (and only defer theall_finishedcallback for the case where a caller is already waiting). I'm happy to open a PR with it and a regression test if that sounds right.Until then, the workaround from the issue (opening each footer with
pq.ParquetFile(path).metadatabefore the scan) is a reasonable guard.PepijnBrasseur commentedon Sep 26, 2026
Thank you — this matches what we see. Our Linux rate on a plain local filesystem was ~1 in 16,000 across three runs, and higher on Windows, which fits the extra hand-offs explanation; the "small file fails early, half-written file fails late" shape is what our runs recorded too. A PR with a regression test would be very welcome from our side. We are keeping an application-side row-count check against the footer in the meantime.
kita-renji commentedon Sep 26, 2026
@PepijnBrasseur thanks for confirming. The PR is up in #51498. It stores the error while the lock is still held and adds two C++ regression tests, one for inner (fragment) errors and one for outer errors. The tests use a small test-only hook to pull at the exact point where the error used to get lost, so they fail every time without the fix. Until it's released, your row-count check against the footer is a good safeguard.
apacheGH-51495: [C++] Register waiting error before later terminal pu…
apacheGH-51495: [C++] Complete MergedGenerator futures in order inste…
GH-51495: [C++] Fix race in MergedGenerator that could drop an error …
zanmato1984 commentedon Sep 28, 2026
Issue resolved by pull request 51498
#51498