Add Reference Tables by sungwy · Pull Request #6 · apache/iceberg-verification

sungwy · 2026-09-05T05:21:07Z

Add Reference Tables, with additional guidelines in README.md on best practices in organizing and writing reference tables for consumer subscription.

This fixture was developed using Claude, and is currently in draft mode while being reviewed by human

moomindani · 2026-09-18T22:51:09Z

Exercised all three tables from the consumer side with PyIceberg main (0d584073), via the symlink route.

All three load and scan correctly. v1/partitioned-parquet-data decodes its spec as identity(2) + day(3) and returns 3 rows; v2/nested-types-parquet-data decodes the struct's child ids 5 and 6 with the list and map nulls intact; v2/simple-append-parquet-data returns 4 rows with the one null. Sizes measured 16161, 25967 and 28047 bytes, matching the 16–28 KB in the README. index.json resolved all three metadata paths as documented.

One correction to the Paths section. The substitution route does not work for PyIceberg as described, because the absolute prefix is not only in metadata.json: the manifest lists carry manifest_path and the manifests carry file_path, both absolute (file:/tmp/iceberg-verification/...). Measured:

No symlink, no substitution: StaticTable.from_metadata succeeds and the scan then fails on the manifest list.
Prefix substituted in metadata.json only: the load succeeds, location is the new prefix, and the scan fails one level deeper, on the manifest_path inside the manifest-list Avro.

PyIceberg has no load-time path-rewriting hook — StaticTable.from_metadata takes a metadata path and properties and nothing else — so on Windows the honest recipe is "copy the corpus and rewrite the prefix in the metadata JSON and in the manifest lists and manifests", which is a different operation from substituting while loading. Worth saying that explicitly, since the two routes are currently presented as equally cheap.

Related, and worth a line in the procedure: a broken path setup surfaces at scan, not at load. A surface that asserts only on decoded metadata passes with no symlink at all, so a consumer can report green on a corpus it never actually opened.

sungwy · 2026-09-29T15:08:55Z

Hi @moomindani thanks for verifying all the reference examples from the consumer side.

I agree with the findings regarding the substitution approach. I had initially attempted adding that as an alternative, especially for OS environments where symlink creation mechanics may be different. I'm leaning towards removing this approach for now and only supporting the symlink approach prior to the introduction of relative paths in V4.

The Paths section now documents only the symlink route and states it is not yet verified on Windows. I also added that path setup is verified by a scan, since a load opens only the metadata file and passes without the symlink.

gkalashyan-akv · 2026-10-05T11:50:59Z

+
+A consumer resolves it one of two ways. Symlinking `/tmp/iceberg-verification` at
+the checkout leaves every committed byte untouched. Substituting the prefix while
+loading works where symlinks do not, notably on Windows. Both are valid, because


Following up on the earlier discussion: the prefix substitution path still appears in the current revision. Would prefix-rewriting actually work end to end? My understanding is that scanning would still fail on the absolute manifest_path inside the manifest-list Avro. Could we require a scan step in the setup verification rather than just opening metadata? That would confirm the referenced artifacts are actually accessible.

I decided to remove the prefix writing approach as it was error-prone, and kept just the symlink based approach for now. I think it'll make sense to revisit it if we see reported issues with the symlink approach in different environments

gkalashyan-akv · 2026-10-05T11:50:59Z

@@ -0,0 +1,28 @@
+{
+  "tables": [


Could we add a check that follows each index entry through every retained snapshot, manifest list, manifest, and data file, verifying that live references exist, recorded file lengths match, and all snapshots remain readable? Intentionally pruned files in metadata-log should be excluded. Without automation, there is no ongoing guarantee that a future change does not silently break an older snapshot's artifact chain.

I think this is a good idea, I'd prefer to leave it out of scope of this PR as there's interest in introducing that in a follow up from @med9110

gkalashyan-akv · 2026-10-05T11:50:59Z

+      "written-by": "Iceberg Java 1.11.0",
+      "data-file-format": "parquet",
+      "metadata": "v2/nested-types-parquet-data/metadata/v2.metadata.json",
+      "description": "Struct, list and map, so every nested id space appears. One snapshot, two rows, one of them entirely null."


The nested fixture currently contains populated containers and null containers, but no empty containers, null list elements, or null map values. Could we include those cases? The schema explicitly permits null elements and values, and these exercise different decoding behavior from a null list or map.

added this, thank you for the suggestion

gkalashyan-akv · 2026-10-05T11:50:59Z

+
+This directory hosts whole Iceberg tables reference fixtures.
+Surfaces reference a table by path and hold their own expected
+values, so one table can support several surfaces without being copied.


Keeping shared tables separate from surface-specific expectations makes sense. Could we link a companion surface that consumes one of these tables and defines exact expected values and comparison rules? Successful loading and scanning establish usability, but consumers still need independent expectations to establish conformance.

med9110 · 2026-10-12T15:29:25Z

Hi @sungwy, I was reviewing this PR and noticed the feedback about adding automated integrity checks for the reference tables.

I'd be interested in contributing a follow-up validation utility that checks the referenced snapshots, manifests, data files, and file lengths, with appropriate handling of intentionally pruned metadata.

Would you prefer this as a separate PR after the reference-table structure is merged, or should it be incorporated into the current work?

I'd be happy to take ownership of the validation part if that would be useful.

sungwy · 2026-10-21T20:47:10Z

Hi @sungwy, I was reviewing this PR and noticed the feedback about adding automated integrity checks for the reference tables.

I'd be interested in contributing a follow-up validation utility that checks the referenced snapshots, manifests, data files, and file lengths, with appropriate handling of intentionally pruned metadata.

Would you prefer this as a separate PR after the reference-table structure is merged, or should it be incorporated into the current work?

I'd be happy to take ownership of the validation part if that would be useful.

Hi @med9110 - I've included minimal validation for snapshots and rows a pre-requisite to it being merged.

I think it would be great to get more validations added to the list, and a follow up PR would work well

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

Excluding **/*.avro globally is overly broad and could unintentionally exempt non-fixture Avro… · New
The markdown table syntax here is invalid (double-leading ||), so it won’t render as a table in… · New

What changed in this PR

Adds a set of Iceberg “reference tables” as on-disk fixtures (metadata/manifests/data + expected scan outputs) along with documentation and indexing to support cross-implementation verification.

Changes:

Added multiple v1/v2 Iceberg table fixtures (metadata + expected scan results) covering append, partitioning, and nested types
Added reference-tables/index.json to enumerate fixtures for consumers
Added reference-tables/README.md describing the contract, layout, and consumption procedure; updated RAT excludes for Avro fixtures

File	Description
reference-tables/v2/simple-append-parquet-data/metadata/v3.metadata.json	Adds Iceberg metadata for a v2 unpartitioned append fixture
reference-tables/v2/simple-append-parquet-data/expected/scan.json	Adds expected full-scan rows per snapshot for the v2 append fixture
reference-tables/v2/partitioned-parquet-data/metadata/v3.metadata.json	Adds Iceberg metadata for a v2 partitioned fixture
reference-tables/v2/partitioned-parquet-data/expected/scan.json	Adds expected full-scan rows per snapshot for the v2 partitioned fixture
reference-tables/v2/nested-types-parquet-data/metadata/v2.metadata.json	Adds Iceberg metadata for a v2 nested-types fixture
reference-tables/v2/nested-types-parquet-data/expected/scan.json	Adds expected full-scan rows for the v2 nested-types fixture
reference-tables/v1/partitioned-parquet-data/metadata/v3.metadata.json	Adds Iceberg metadata for a v1 partitioned fixture
reference-tables/v1/partitioned-parquet-data/expected/scan.json	Adds expected full-scan rows per snapshot for the v1 partitioned fixture
reference-tables/index.json	Introduces an index of all reference tables and their entrypoint metadata files
reference-tables/README.md	Documents fixture contract, naming, symlink/path conventions, and expected output format
dev/.rat-excludes	Excludes `.avro` fixture files from RAT header checks

_{Review effort: Lite}

+
+# Avro container files embed their schema as JSON text in the file header, so RAT
+# reads them as text. They are binary fixture data and cannot carry a header either.
+**/*.avro


+| | |
+| -- | -- |
+| Iceberg Java | `new BaseTable(new StaticTableOperations(path, io), name)` |
+| PyIceberg | `StaticTable.from_metadata(path)` |
+| iceberg-rust | `StaticTable::from_metadata_file(path, ident, io)` |
+| iceberg-go | `table.NewFromLocation(..., path, ...)` |
+| iceberg-cpp | `TableMetadataUtil::Read(io, path)`, then `StaticTable::Make(ident, metadata, path, io)` |
+


nssalian

Thanks for the PR @sungwy. In the absence of runners I tested locally at the last commit. I read every table at every snapshot in expected/scan.json with five readers: PyIceberg 0.11.0, Iceberg Java 1.12.0, iceberg-go (d66ab48), iceberg-rust 0.10.1, and a raw fastavro + pyarrow reader that uses no Iceberg library. All 7 snapshots across the 4 tables match expected/scan.json in all five readers. Every manifest list, manifest and data file is reached from the metadata, manifest lengths, file sizes and record counts match what the metadata records, and every Parquet field id is in the table schema. As a sanity check, the comparison catches altered rows, longs written as ints, and dropped or duplicated rows. Removing a data file makes the read fail.

Left some comments to help clean up and get this going. I'll add some v3 specific examples once this is merged.

nssalian · 2026-10-26T16:27:21Z

+## Procedure
+
+Each entry in `snapshots` is checked on its own. The table is scanned at that
+snapshot through the implementation's scan API, with no filter and every column


Which schema does the scan project at an earlier snapshot, the current one or the one named by the snapshot's schema-id? Java's data-table scan and PyIceberg's scan(snapshot_id=...) both use the snapshot's schema, and I couldn't find a spec clause that fixes it either way. Every table here has a single schema, so it doesn't matter today. But a table that adds a column between snapshots, like a v3 default-values table, would get different expected rows depending on the reading. Could the Procedure say "projected with the schema of the snapshot's schema-id"?

nssalian · 2026-10-26T16:34:27Z

+## Paths
+
+Prior to v4 the spec requires every path field to be fully qualified, so each
+table carries an absolute prefix in `location`, in every `manifest-list`, and in


nit: manifest_path in the manifest lists and the metadata-log entries carry the prefix too. Also, only location is spelled file:///. manifest-list, manifest_path, file_path and metadata-log are all file:/tmp/..., so the prefix as written above matches only location.

nssalian · 2026-10-26T16:41:09Z

+
+Size tracks commit count, and almost nothing else. A commit costs roughly 12 to
+14 KB, over half of it the manifest and manifest list it produces. Rows are close
+to free: a Parquet file holding five rows is under 1 KB and is nearly all footer.


nit: Small fix on the numbers: no data file here has five rows, and the 2-row a.parquet in the partitioned tables is 1087 B, with a 719 B footer. Something like "a Parquet file holding a few rows is about 1 KB, roughly two thirds of it footer" would match the files.

nssalian · 2026-10-26T16:42:05Z

+
+Only the current `metadata.json` is committed. Its name carries a version
+counter, not the table's format version: a writer names each metadata commit
+`v<V>.metadata.json` starting at `v1` on create, so `v1/partitioned-parquet-data`


v<V>.metadata.json is the naming the spec gives for File System Tables. Metastore tables use <V>-<uuid>.metadata.json. Maybe scope this to how these tables were written.

nssalian · 2026-10-26T16:43:16Z

@@ -0,0 +1,36 @@
+{
+  "tables": [


I noticed two places where these tables follow the Java writer rather than the spec text. It would help to note them so an implementation of iceberg doesn't assert them. The v2 manifests, including the new v2/partitioned-parquet-data, omit the schema-id key-value entry, which the spec lists as required for v2. The v1 table carries refs and partition field-id, where the spec's v1 column is blank ("should be omitted").

nssalian · 2026-10-26T16:46:09Z

+}
+```
+
+A table is written once and not regenerated, so `written-by` is per table: the


Could the code that wrote these tables be checked in next to them, even if CI never runs it? parquet-testing keeps its generators beside its fixtures (see the JSON and BSON section of data/README.md). written-by gives the release, but not the writes, the table properties, or the steps after the write, like the pruning on line 141. Without that, it's hard to tell which bytes came straight from the writer and what steps were taken

laskoviymishka mentioned this pull request Sep 18, 2026

[Epic] Conformance verification: phased roadmap (tracking) #10

Open

2 of 11 tasks

gkalashyan-akv reviewed Oct 5, 2026

View reviewed changes

This was referenced Oct 12, 2026

Add name-mapping enabled table (any version) #14

Open

Add reference tables for V3 row-lineage enabled table #13

Open

sungwy added 3 commits October 21, 2026 20:55

reference tables

72d61b7

adopt feedback, with minimal validation

2ee5558

Exclude Avro fixtures from RAT

76c0e9c

sungwy force-pushed the reference-tables branch from ba990b3 to 76c0e9c Compare October 21, 2026 20:57

added empty list and map case

c2f639c

sungwy marked this pull request as ready for review October 21, 2026 21:44

Copilot AI balanced review requested due to automatic review settings October 21, 2026 21:44

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Copilot stopped reviewing on behalf of sungwy due to an error October 21, 2026 23:07
The job was not acquired by a runner even after multiple attempts.

sungwy requested review from gkalashyan-akv and nssalian and removed request for gkalashyan-akv October 22, 2026 17:26

nssalian reviewed Oct 26, 2026

View reviewed changes

Conversation

sungwy commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moomindani commented Sep 18, 2026

Uh oh!

sungwy commented Sep 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sungwy Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

med9110 commented Oct 12, 2026

Uh oh!

sungwy commented Oct 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

nssalian left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sungwy commented Sep 5, 2026 •

edited

Loading

sungwy Oct 21, 2026 •

edited

Loading