Skip to content

Core: Reject v1 manifests in RewriteManifests.addManifest for v2+ tables - #18502

Open
adityaksolves wants to merge 1 commit into
apache:mainfrom
adityaksolves:fix-addmanifest-v1-on-v2
Open

adityaksolves wants to merge 1 commit into
apache:mainfrom
adityaksolves:fix-addmanifest-v1-on-v2

Conversation

@adityaksolves

@adityaksolves adityaksolves commented Oct 14, 2026 •

Copy link
Copy Markdown

Closes #18482

RewriteManifests.addManifest did not check the format version of the manifest it was given. On a v2+ table, a v1-format manifest was mishandled on both paths:

  • Reused manifest (snapshot ID null): the commit succeeded, but scan planning later threw an NPE in DeleteFileIndex.forEntry because EXISTING entries had null sequence numbers.
  • Copied manifest (snapshot ID -1): the commit failed with IllegalStateException: Found unassigned sequence number.

This change rejects v1 manifests up front on v2+ tables with IllegalArgumentException: Cannot add v1 manifest to a v<N> table: <path>. The format version is read from the format-version key in the Avro manifest header (absent means v1). Non-Avro (Parquet) manifests are skipped, since v1 manifests are always Avro.

Tests: added testAddV1ManifestToV2PlusTable and testAddV1CopiedManifestToV2PlusTable in TestRewriteManifests, covering both paths on v2+. Both fail without the fix.

Note: on v2+ tables, each addManifest call now reads the manifest header once.

@github-actions github-actions Bot added the core label Oct 14, 2026
@adityaksolves
adityaksolves force-pushed the fix-addmanifest-v1-on-v2 branch from 0134d27 to 24b1717 Compare October 23, 2026 09:56
@adityaksolves

Copy link
Copy Markdown
Author

Hi @amogh-jahagirdar @RussellSpitzer, could one of you take a look when you get a chance?

This fixes #18482, where RewriteManifests.addManifest mishandles v1-format manifests on v2+ tables (NPE during scan planning on the reused path, IllegalStateException on the copied path).

One design question where I'd value your input: I went with rejecting v1 manifests up front on v2+ tables with a clear IllegalArgumentException, rather than copying them and writing the spec default of 0 as explicit sequence numbers (the approach iceberg-cpp takes in apache/iceberg-cpp#774). My reasoning is that it fails fast and avoids silently upgrading the manifest format, but I'm happy to switch to copy-with-0 if that's preferred for Java.

I've rebased onto the latest main and CI is green. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Core: RewriteManifests.addManifest mishandles v1-format manifests on v2+ tables

1 participant