fix(parquet): stop scalar reads past available input bits

[fallintoplace]

Rationale for this change

BitReader.fillbuffer padded short reads with zeros without recording how many bits were present. Scalar and trailing reads could therefore treat missing input as encoded zero values.

What changes are included in this PR?

• Track the valid bits in each buffer and reject scalar reads that extend beyond them.
• Keep boolean batches on the bulk path, checking availability once per byte instead of once per value.
• Propagate seek and refill failures.
• Validate bulk discards.
• Preserve zero-width reads without requiring input.

Are these changes tested?

Yes. Regression coverage includes empty input, partial scalar batches, reads crossing the 64-bit buffer boundary, oversized discards, zero-width values, and every width from 1 through 64 across input lengths from 0 through 16 bytes. The utility and encoding packages pass normal, noasm, and race tests.

A six-run comparison against current main shows no regression in BenchmarkPlainDecodingBoolean across batch sizes from 1,024 through 65,536 values.

Are there any user-facing changes?

Truncated scalar bit streams now return short-input errors rather than decoding missing data as zeros.

Related changes

This PR covers scalar and trailing buffered reads. #943 covers bulk 32-value unpacking, while #946 fixes short boolean-batch loop progress.

[zeroshade]

Thanks — the fix is correct, and I verified it integrates cleanly with current main (merges onto #943/#946 with no conflicts, compiles, and the parquet/internal/utils + parquet/internal/encoding tests pass). CI is green.

However, a benchmark comparison (main vs this PR head, benchstat -count=6) shows a consistent throughput regression on the per-value boolean bit-read path:

PlainDecodingBoolean/len_8192     +4.25%  (p=0.002)
PlainDecodingBoolean/len_16384    +4.85%  (p=0.002)
PlainDecodingBoolean/len_32768    +4.82%  (p=0.002)
PlainDecodingBoolean/len_65536    +4.89%  (p=0.002)
DecodeDictByteArray               +1.95%  (p=0.002)
DeltaBinaryPackedDecodingInt32    -1.6%..-2.2%  (faster)
ReadInt32Column (end-to-end)      ~  (no change, p=0.13)

The regression comes from the new per-value validBits guard in BitReader.next(): boolean plain-decode calls next(1) once per value, so it hits that branch on every bit, whereas the bulk GetBatch/unpack32 paths (int32 etc.) are unaffected or slightly faster. The guard itself is correct and needed for the fix.

Could we keep the boolean hot path off the per-value route — e.g. decode booleans through the bulk GetBatchBools (already hardened in #946 via ensureBoolBitAvailable) instead of looping next(1)? That should preserve the safety fix with ~0 regression. Happy to re-run the benchmarks once it's updated.

[zeroshade]

This per-value guard is the source of the ~4.8% PlainDecodingBoolean regression: boolean plain-decode calls next(1) once per value, so it hits this check on every bit. The check is correct — the way to recover the throughput is to keep booleans off the per-value path (decode them via the bulk GetBatchBools, hardened in #946 with ensureBoolBitAvailable) rather than looping next(1). Fixed-width/bulk paths (GetBatch/unpack32) don't pay this and are unaffected.

[zeroshade]

Thanks for the update — the boolean over-read I flagged earlier is resolved. nextBool() keeps the bulk GetBatchBools path (no per-value validBits check in the hot loop), and that regression is gone.

However, benchmarking the current head (564e6a6) surfaced a new regression on the dictionary byte-array decode path:

name                 old time/op  new time/op  delta
DecodeDictByteArray   74.3µs ± 1%  80.7µs ± 1%  +8.54%  (p=0.000 n=10+10)

(measured with benchstat, 10 runs each; old = base branch, new = this PR head.)

This looks like fallout from the next() / GetBatchIndex restructuring — the added per-value bounds/validBits handling penalizes the dict-index scalar decode loop. Could the check be hoisted out of the hot per-value path (mirroring how nextBool() keeps its bulk path) so dictionary decode isn't taxed?

The correctness fix itself looks right; I just want to avoid trading the boolean over-read for an ~8.5% hit on dict decode before this merges.