[Feature] Add query-scoped planning limits and cooperative cancellation #435

Parent integration: sirius-db/sirius#2181. Consumer bridge tracking: polardb/duckdb-paimon#139.

Need

A database embedding paimon-cpp needs to admit planning work before materializing an arbitrarily large native plan, and stop submitting work after cooperative cancellation. This is a feature request, not a claim that existing standalone planning is incorrect.

The integration pins 53f9c86d45aabb0a6f1a379271da07d7a9f27a3d. The following API gap also remains in inspected main 242decffdfdefef01dcd0e52e48c9a1e69792479:

• ScanContextBuilder::WithMemoryPool routes pool-backed allocations, but does not bound ordinary STL planning containers or their transitive metadata allocations.
• TableScan::CreatePlan() returns a complete plan; it exposes no planning cancellation/budget contract. SetLimit is a row-result pushdown, not a planning-memory quota: DataTableBatchScan::CreatePlan calls starting_scanner_->Scan before ApplyPushDownLimit.
• FileStoreScan::ReadFileEntries builds a vector of futures for all supplied manifest metadata, then collects complete per-manifest entry vectors. A custom serial executor can prevent concurrent task execution, but does not itself bound those retained containers/results.

Source references at the inspected main:

• Scan context API
• TableScan API
• Batch scan / limit ordering
• Manifest task/results materialization

Proposed Contract To Discuss

Provide optional query-scoped planning controls, preserving existing unlimited behavior by default. The API shape should be agreed with maintainers; a post-plan split-count check is not sufficient.

• Check conservative manifest/entry/split/retained-metadata quotas before the corresponding allocation or task submission; identify which allocations are covered and what remains outside the guarantee.
• Bound outstanding manifest work and retained results, including ordinary containers, rather than only pool-backed buffers.
• Check a cooperative cancellation token before I/O/submission and inside substantial decode/merge loops. Do not promise interruption of an arbitrary blocking filesystem call unless the filesystem supports it.
• Return a distinguishable resource-refusal/cancellation status, never partial success or EOF. Stop new submissions and drain accepted work before returning, including exceptions and submission failure.
• Avoid changing process-global executors, Arrow thread capacity, or allocation resources.

Acceptance tests should use many small manifests and injected allocation/I/O/submission failures. Assert the limit is enforced before excess work is accepted, accepted jobs are drained on every terminal path, and cancellation cannot return a partial successful plan. Include serial/nested executor configurations to guard against deadlock. Keep row LIMIT semantics separate.

This issue does not request another Arrow batch-lifetime implementation; that work already exists in #259/#260. No hard whole-process-memory bound or prompt cancellation of uninterruptible I/O is claimed by the current integration.