Skip to content

Fix Parquet 0.8.0 type fidelity and selective reads - #482

Merged
perNyfelt merged 4 commits into
mainfrom
parquet-v080-fixes
Oct 18, 2026
Merged

perNyfelt merged 4 commits into
mainfrom
parquet-v080-fixes

Conversation

@perNyfelt

Copy link
Copy Markdown
Member

Implement matrix-parquet/req/v0.8.0-fixes.md in matrix-parquet, keeping 0.8.0-SNAPSHOT and the existing unreleased release section.

  • Correct unsigned integers, INT96 and legacy MAP reads; preserve nested Matrix leaf types through allowlisted matrix.nestedTypes metadata.
  • Use exact residual filters for unrepresentable predicate bounds while retaining safe partial AND pruning. Default all timestamp classes to MICROS truncation, with overflow-safe NANOS opt-in.
  • Fix nested shape inference and error paths, bloom/codec validation, secure stream spooling, dataset names, glob depth/separators and weak-ETag preconditions. Remove dead reader/POJO paths and update documentation.

The 200-column × 50,000-row write measurement improved from 9,590 ms to 3,427 ms with identical output size (one local run per implementation).

Verification passed:

  • ./gradlew :matrix-parquet:codenarcMain :matrix-parquet:codenarcTest --no-daemon --rerun-tasks --console=plain
  • ./gradlew :matrix-parquet:spotlessCheck --console=plain
  • ./gradlew :matrix-parquet:test --console=plain
  • PARQUET_PYTHON=/tmp/matrix-parquet-fixtures/bin/python ./gradlew :matrix-parquet:test :matrix-parquet:compileConsumerApi -PrunSlowTests=true --console=plain — final module run: 156 passed, 6 optional skips.
  • PyArrow 21.0.0 verify_matrix_output.py: SNAPPY/ZSTD and CDC rewrite/readback.
  • ./gradlew :matrix-parquet:test --tests '*ParquetReadBenchmarkTest' -Pparquet.baseline=true -Pparquet.benchmarkRows=10000 --rerun --console=plain — 1 executed, 0 skipped.
  • ./gradlew :matrix-parquet:test --tests '*ParquetLayoutBenchmarkTest' -Pparquet.layoutBenchmark=true --rerun --console=plain — 2 executed, 0 skipped.
  • ./gradlew test -Pheadless=true --console=plain — full repository suite passed, including a final rerun.

The completed requirement file contains the detailed verification record, including benchmark setup and a clean rerun after an internal CodeNarc/Groovy worker error. No tags were created.

Windows parsing rules were unit-tested on macOS; no Windows host run was performed. Weak ETags and legacy MAP layouts were verified with local HTTP/synthetic Parquet fixtures, without production-server or real legacy-file validation.

@perNyfelt
perNyfelt merged commit 6941c78 into main Oct 18, 2026
3 checks passed
@perNyfelt
perNyfelt deleted the parquet-v080-fixes branch October 18, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant