You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
One of the missing v2 features in PyIceberg is reading from equality delete files.
I believe PyIceberg needs support for reading these files or else we run into situations where people are querying deleted data. Happy to be disproven on this!
I'm in favor of adding support, @rambleraptor! I've been meaning to open an issue for this but keep getting sidetracked.
This is a gap beyond just v2. Any table with equality deletes is affected, which can come from streaming use cases like Flink or direct usage of the Java core library (v3+ can have equality deletes). We should be able to read them, but I'd avoid writing them in favor of delete vectors.
One of the main benefits would be rewriting equality deletes to positional deletes. I know there was a PR in deltacat to do this in the past: ray-project/deltacat#471
I think a good path forward would be to start by indexing the equality deletes so they can be mapped into file scan tasks. Then follow up with the reads using PyArrow, similar to what was done in #2255. That way we avoid a massive PR.
I think that sounds like a great plan. I did the index part at #3285. It also includes some plumbing to ensure that equality deletes are ignored in the PyArrow reading in order to avoid crashing.
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
Activity
geruh commentedon Apr 24, 2026
I'm in favor of adding support, @rambleraptor! I've been meaning to open an issue for this but keep getting sidetracked.
This is a gap beyond just v2. Any table with equality deletes is affected, which can come from streaming use cases like Flink or direct usage of the Java core library (v3+ can have equality deletes). We should be able to read them, but I'd avoid writing them in favor of delete vectors.
One of the main benefits would be rewriting equality deletes to positional deletes. I know there was a PR in deltacat to do this in the past: ray-project/deltacat#471
I think a good path forward would be to start by indexing the equality deletes so they can be mapped into file scan tasks. Then follow up with the reads using PyArrow, similar to what was done in #2255. That way we avoid a massive PR.
What do you think?
rambleraptor commentedon Apr 24, 2026
I think that sounds like a great plan. I did the index part at #3285. It also includes some plumbing to ensure that equality deletes are ignored in the PyArrow reading in order to avoid crashing.
Add equality-delete apply on PyArrow reads
github-actions commentedon Oct 26, 2026
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.