Describe the bug
In CometExecIterator, memoryLimitPerTask is passed into executePlan. However, the top-level executePlan is always a ShuffleWriter which then calls back into the JVM to execute the child plan via a recursive call to executePlan.
This results in the same per-task memory pool limit being used for both plans. This means that the memory pool limit is actually 2x of available memory.
Steps to reproduce
No response
Expected behavior
No response
Additional context
For a single native plan containing a ShuffleWriterExec and a SortMergeJoinExec, each of these operators would get 50% of the pool each because Comet uses fair_unified pool by default when off-heap memory is enabled. Because Comet currently executes these as two separate plans, they both get their own memory pools with the same per-task limit.
Describe the bug
In CometExecIterator,
memoryLimitPerTaskis passed intoexecutePlan. However, the top-levelexecutePlanis always aShuffleWriterwhich then calls back into the JVM to execute the child plan via a recursive call toexecutePlan.This results in the same per-task memory pool limit being used for both plans. This means that the memory pool limit is actually 2x of available memory.
Steps to reproduce
No response
Expected behavior
No response
Additional context
For a single native plan containing a
ShuffleWriterExecand aSortMergeJoinExec, each of these operators would get 50% of the pool each because Comet usesfair_unifiedpool by default when off-heap memory is enabled. Because Comet currently executes these as two separate plans, they both get their own memory pools with the same per-task limit.