You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Core: Reject v1 manifests in RewriteManifests.addManifest for v2+ tables - #18502
RewriteManifests.addManifest did not check the format version of the manifest it was given. On a v2+ table, a v1-format manifest was mishandled on both paths:
Reused manifest (snapshot ID null): the commit succeeded, but scan planning later threw an NPE in DeleteFileIndex.forEntry because EXISTING entries had null sequence numbers.
Copied manifest (snapshot ID -1): the commit failed with IllegalStateException: Found unassigned sequence number.
This change rejects v1 manifests up front on v2+ tables with IllegalArgumentException: Cannot add v1 manifest to a v<N> table: <path>. The format version is read from the format-version key in the Avro manifest header (absent means v1). Non-Avro (Parquet) manifests are skipped, since v1 manifests are always Avro.
Tests: added testAddV1ManifestToV2PlusTable and testAddV1CopiedManifestToV2PlusTable in TestRewriteManifests, covering both paths on v2+. Both fail without the fix.
Note: on v2+ tables, each addManifest call now reads the manifest header once.
This fixes #18482, where RewriteManifests.addManifest mishandles v1-format manifests on v2+ tables (NPE during scan planning on the reused path, IllegalStateException on the copied path).
One design question where I'd value your input: I went with rejecting v1 manifests up front on v2+ tables with a clear IllegalArgumentException, rather than copying them and writing the spec default of 0 as explicit sequence numbers (the approach iceberg-cpp takes in apache/iceberg-cpp#774). My reasoning is that it fails fast and avoids silently upgrading the manifest format, but I'm happy to switch to copy-with-0 if that's preferred for Java.
I've rebased onto the latest main and CI is green. Thanks!
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18482
RewriteManifests.addManifestdid not check the format version of the manifest it was given. On a v2+ table, a v1-format manifest was mishandled on both paths:DeleteFileIndex.forEntrybecause EXISTING entries had null sequence numbers.IllegalStateException: Found unassigned sequence number.This change rejects v1 manifests up front on v2+ tables with
IllegalArgumentException: Cannot add v1 manifest to a v<N> table: <path>. The format version is read from theformat-versionkey in the Avro manifest header (absent means v1). Non-Avro (Parquet) manifests are skipped, since v1 manifests are always Avro.Tests: added
testAddV1ManifestToV2PlusTableandtestAddV1CopiedManifestToV2PlusTableinTestRewriteManifests, covering both paths on v2+. Both fail without the fix.Note: on v2+ tables, each
addManifestcall now reads the manifest header once.