Add optimized Parquet support and selective dataset reads

[perNyfelt]

Hugging Face optimized downloads are standard Parquet, but Matrix previously decoded every field through Group records, could not use caller predicates for page pruning, and mishandled several external logical types. This implements the reviewed plan for matrix-parquet 0.8.0, including migration coverage for existing Matrix files.

• Replace the full-scan Group reader with column converters on an InputFile/ParquetFileReader row-group loop. Add Matrix-owned typed predicates, projection, limits, ordered row-group selection and closeable bounded batches.
• Add ordered shard/directory/glob reads with strict schema checks and bounded parallel workers, HTTP range reads with safe download fallback, and pinned Hub URI resolution with origin-scoped token handling.
• Preserve raw binary and exact decimals, fix naive timestamps, annotate nested strings, validate inferred/concrete column types, and document the 0.8.0 behavior changes. Metadata uses a fixed allowlist. FLOAT16/INT96 fail contextually.
• Share writer tuning across output paths, add deterministic PyArrow CDC/index fixtures and a Python CI lane, and record benchmarks and completed task evidence. Checked-in Parquet fixtures total 738,113 bytes.

Modules touched: matrix-parquet; shared cookbook/tutorial and the release-tools workflow.

Validation passed:

./gradlew :matrix-parquet:codenarcMain :matrix-parquet:codenarcTest --console=plain
./gradlew :matrix-parquet:spotlessCheck --console=plain
PARQUET_PYTHON=/tmp/matrix-parquet-fixtures/bin/python \
  ./gradlew :matrix-parquet:test :matrix-parquet:compileConsumerApi -PrunSlowTests=true --console=plain
./gradlew test -Pheadless=true --console=plain
/tmp/matrix-parquet-fixtures/bin/python matrix-parquet/src/test/python/verify_matrix_output.py
./gradlew :matrix-parquet:test --tests ParquetInteropTest --console=plain
./gradlew :matrix-parquet:codecWithoutHadoop --console=plain
groovy scripts/release-all/test-workflow.groovy .

The Parquet suite reports 137 tests, 132 passed, zero failures and five intentional opt-in benchmark/capture skips; default-size CDC ran. Arrow 21.0.0 verified annotated strings, binary, decimals and metadata-preserving CDC rewrites, then Java reread them. Fixture regeneration and the 12 offline huggingface_hub 2.2.0 URL cases are deterministic. The consumer compile probe excludes parquet-java/Hadoop from its compile classpath.

Local million-row comparisons measured 89.5% lower wide-scan median time and 43.8% lower categorical sampled live heap; mixed/nested time also improved. Independent page-index, bloom and HTTP byte counters demonstrate pruning without timing assertions. See benchmark methods/results and task evidence.

Native CDC writing remains deferred to apache/parquet-java#3818; the documented PyArrow rewrite is verified. Hadoop remains a runtime dependency because the codec-exclusion spike found upstream Hadoop linkage even for UNCOMPRESSED initialization. Original 0.7.x byte[] writing fails with InvalidSchemaException in the pinned checkout; the legacy [B text fixture is explicitly synthetic and no original bytes are claimed recoverable.

Review fixes in 0d71ce5 address all thirteen findings: empty Arrow dictionaries; JSON/ENUM text; bulk ranges, redirect reuse and usable cache bounds; common static URL/Hub/directory routing; atomic file replacement; consistent source/index handling; shared column type descriptors; null map defaults; daemon workers; dot-segment rejection; projection around legacy [B columns; and corrected documentation.

The new regression coverage includes a 1 MiB range served in one data request, repeated-range cache reuse, exact preservation of existing output bytes after mid-write decimal failures, and common index policy across collected and limited shard batches. Ordered module checks, the Python CDC lane and full repository suite passed. Review tasks and verification.

Follow-up review fixes preserve normal umask permissions for new output, retain existing symbolic
links while replacing their resolved targets, document the atomic-move filesystem requirement,
and avoid the extra scratch-buffer allocation for array-backed HTTP range reads. Added POSIX
permission/symlink and sliced/direct ByteBuffer regressions. Re-ran static analysis, formatting,
the full Parquet suite (137 tests: 132 passed, five intentional skips), Python interoperability,
consumer API compilation and ./gradlew test -Pheadless=true successfully.

Dangling symbolic links (relative/absolute and chained) now create their resolved target without
replacing the links; cyclic links fail before temporary output is created. The module guide also
explains leftover temporary siblings after abrupt JVM termination. Static analysis, formatting,
the Parquet suite (137 tests: 132 passed, five intentional skips), consumer compilation and the
full repository suite passed again; the new POSIX regressions ran successfully.

The merge-time Linux CI run exposed a stale BOM pin; matrix-bom/bom.xml now references
matrix-parquet:0.8.0-SNAPSHOT, matching the module. Additional local verification passed:

BOM_VERIFY_REPO=/private/tmp/matrix-pr481-bom-verify RUN_EXTERNAL_TESTS=false RUN_SLOW_TESTS=false GRADLE_OPTS=-Dorg.gradle.workers.max=2 bash matrix-bom/verifyBomApi.sh
groovy scripts/release-all/test-manifest-verifier.groovy . /private/tmp/matrix-pr481-bom-verify

This verified isolated publishing/resolution, 15 module tests, 27 API integration tests and the
manifest verifier's 21 unreleased release-coordinate fixtures with read-only staging.