Skip to content

incremental_append_scan from an expired start fails after expire_snapshots #4193

Open

Description

@RyanLin5967

Apache Iceberg version

0.12.0 (latest release)

Please describe the bug 🐞

Currently, applying RemoveSnapshotsUpdate sets the children's parent_snapshot_id to None, so the lineage check #3512 added for an expired exclusive start never matches and the scan raises. Java keeps the parent id, and #3512's test removes the snapshot by hand with the parent id intact.

import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog

cat = SqlCatalog("c", uri="sqlite:///:memory:", warehouse=f"file://{tempfile.mkdtemp()}")
cat.create_namespace("ns")
t = cat.create_table("ns.t", schema=pa.schema([("x", pa.int64())]))
ids = []
for x in (1, 2, 3):
    t.append(pa.table({"x": [x]}))
    ids.append(t.current_snapshot().snapshot_id)

scan = lambda: t.incremental_append_scan(from_snapshot_id_exclusive=ids[0]).to_arrow()["x"].to_pylist()
print(sorted(scan()))
# [2, 3]
t.maintenance.expire_snapshots().by_id(ids[0]).commit()
print(t.snapshot_by_id(ids[1]).parent_snapshot_id)
# None, was ids[0]
print(sorted(scan()))
# ValueError: Starting snapshot (exclusive) 6945542855238032183 is not a parent ancestor of end snapshot 6485958663478198635
# should be [2, 3]

tested on pyiceberg 0.12.0 and main

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Activity

  1. dragonknightk commented on Oct 30, 2026

    @dragonknightk

    Thanks for reporting this (´。• ᵕ •。`) I've opened #4206 to keep the parent snapshot id when snapshots are removed, matching the Java behavior. I'd be really grateful if you could take a look whenever you have a moment. No rush at all, any feedback is very welcome ʕ•ᴥ•ʔ

  2. RyanLin5967 commented on Oct 31, 2026

    @RyanLin5967
    Author

    Took a look at #4206. Dropping the reset matches what Java does, ancestors_of already stops at a parent id that no longer resolves, and the new test is the same case as the repro here, so the change looks right to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions