Apache Iceberg version
1.10.1
Query engine
Starrocks
Please describe the bug 🐞
Summary
ScanTaskIterable (REST scan planning, Iceberg 1.10.1) can leave background PlanTaskWorker tasks running after the scan is abandoned. This happens when:
- The consumer stops iterating (query timeout / cancellation / client disconnect in downstream engines), and
- Only the outer
CloseableIterable is closed, or nothing is closed at all.
In our production integration (StarRocks FE + Iceberg REST catalog), this leads to all planning pool threads blocked in offerWithTimeout() for hours, while new scans block in hasNext() with empty queues.
Affected version / component
- Version: Apache Iceberg 1.10.1
- Class:
org.apache.iceberg.rest.ScanTaskIterable
- Related:
org.apache.iceberg.rest.RESTTableScan
Problem 1: ScanTaskIterable.close() is a no-op
RESTTableScan returns:
return CloseableIterable.whenComplete(
new ScanTaskIterable(...),
this::cancelPlan);whenComplete calls iterable.close() in a finally block after the wrapped iterable is closed.
However, in 1.10.1:
@Override
public void close() throws IOException {} // ScanTaskIterable.close() is emptyShutdown is only set in:
// ScanTasksIterator.close()
shutdown.set(true);
taskQueue.clear();
planTasks.clear();
If the engine:
closes only the outer iterable, or
never closes the iterator because the query timed out / was cancelled during hasNext(),
then:
PlanTaskWorker threads keep running,
producers may block in offerWithTimeout() when taskQueue (capacity 1000) is full,
consumers are gone, so the queue never drains,
workers retry forever because shutdown remains false.
Observed behavior in downstream engine (StarRocks)
FE jstack shows:
Consumers blocked: ScanTasksIterator.hasNext() → LinkedBlockingQueue.poll (empty queue)
Producers blocked: PlanTaskWorker.offerWithTimeout() → LinkedBlockingQueue.offer (full queue)
REST catalog probes still succeed because catalog is healthy; failure is client-side lifecycle/cleanup.
Willingness to contribute
Apache Iceberg version
1.10.1
Query engine
Starrocks
Please describe the bug 🐞
Summary
ScanTaskIterable(REST scan planning, Iceberg 1.10.1) can leave backgroundPlanTaskWorkertasks running after the scan is abandoned. This happens when:CloseableIterableis closed, or nothing is closed at all.In our production integration (StarRocks FE + Iceberg REST catalog), this leads to all planning pool threads blocked in
offerWithTimeout()for hours, while new scans block inhasNext()with empty queues.Affected version / component
org.apache.iceberg.rest.ScanTaskIterableorg.apache.iceberg.rest.RESTTableScanProblem 1:
ScanTaskIterable.close()is a no-opRESTTableScanreturns:whenComplete calls iterable.close() in a finally block after the wrapped iterable is closed.
However, in 1.10.1:
Shutdown is only set in:
If the engine:
closes only the outer iterable, or
never closes the iterator because the query timed out / was cancelled during hasNext(),
then:
PlanTaskWorker threads keep running,
producers may block in offerWithTimeout() when taskQueue (capacity 1000) is full,
consumers are gone, so the queue never drains,
workers retry forever because shutdown remains false.
Observed behavior in downstream engine (StarRocks)
FE jstack shows:
Consumers blocked: ScanTasksIterator.hasNext() → LinkedBlockingQueue.poll (empty queue)
Producers blocked: PlanTaskWorker.offerWithTimeout() → LinkedBlockingQueue.offer (full queue)
REST catalog probes still succeed because catalog is healthy; failure is client-side lifecycle/cleanup.
Willingness to contribute