Bug triage results: 2026-08-03 #5231

Triage pass over the open requires-triage queue, per the project Bug Triage Guide.

• Date: 2026-08-03
• Total issues processed: 68 (63 triaged, 5 skipped, 0 failed)
• Type counts: 23 bugs, 40 enhancements
• Priority counts applied: priority:critical 2, priority:high 6, priority:medium 12, priority:low 3
• Guide: docs/source/contributor-guide/bug_triage.md

Labels have already been applied. A reviewer should spot-check the calls below and close this issue when satisfied; corrections should be made directly on the affected issue.

Two notes on label availability: the guide lists spark 4 as a pre-existing area indicator, but the repository only has spark 4.0 / spark 4.1 / spark 4.2, so no spark 4 label was applied anywhere in this pass. Where an issue already carried non-guide labels (correctness, performance, EPIC, area:udf, area:Iceberg, temporal expressions, test, user experience, good first issue), those were left untouched and are included in the area lists below for context.

Bugs

priority:critical

• Codegen dispatcher: whole-tree NullIntolerant short-circuit suppresses ANSI errors, plus TIME type gaps between canHandle and the runtime dispatcher (#5218)
  • Area labels: area:expressions, area:udf, correctness
  • Rationale: Finding 1 makes Comet return NULL on a row where Spark raises an ANSI error, with no error or warning to the user, which the guide's decision tree places at priority:critical; this replaces the reporter's initial priority:high, and the blast radius is wide because ~70 built-in expressions route through CometCodegenDispatch.
• [EPIC] cast from string: trim semantics diverge from Spark across all numeric, datetime and boolean targets (#5149)
  • Area labels: area:expressions, correctness, EPIC
  • Rationale: 7 of the 8 supported string-cast targets diverge in both directions, including Comet returning a non-NULL value where Spark returns NULL or throws — silent wrong results across a core expression family.

priority:high

• ListPositionsExpr panics on sliced list input, breaking native posexplode over LIMIT with OFFSET (#5224)
  • Area labels: area:expressions
  • Rationale: Reproducible native panic (Result::unwrap() on Err) on a supported path — native posexplode fed by a limit with a non-zero offset — matching the guide's crash rule; the failure is visible, so it stays off critical.
• [EPIC] Memory pool and accounting audit sweep (#5212)
  • Area labels: area:shuffle, area:ffi, correctness, performance, EPIC
  • Rationale: The two headline findings are defects in the default off-heap pool — a task can never exceed pool_size / num_consumers, and per-task pool entries leak for the executor's lifetime — which is major functional breakage of production memory provisioning rather than a wrong-results bug.
• collect_list/collect_set can fail with "column types must match schema types" on nested-field nullability drift (#5158)
  • Area labels: area:aggregation, area:expressions
  • Rationale: The aggregate hard-fails validating its own output on a supported path (reachable via CreateNamedStruct, whose evaluate() and data_type() types can diverge), so the query dies rather than falling back; visible failure keeps it below critical.
• ShuffleScanExec and ExpandExec assert on nested field nullability instead of normalizing it (#5137)
  • Area labels: area:shuffle, area:ffi
  • Rationale: A single mismatched nested nullable flag hard-fails the operator even though the data itself is fine — major functional breakage at the FFI/shuffle boundary, per the guide's priority:high definition.
• Hashing a CalendarInterval value fails with "Unsupported data type in hasher: Interval(MonthDayNano)" (#5059)
  • Area labels: area:expressions
  • Rationale: The plan-time gate now advertises CalendarIntervalType as supported, so the hash is serialized to native and the query dies with a CometNativeException instead of falling back — a regression from feat: add CalendarIntervalType support #4898 that turns a clean fallback into a hard failure.
• Null CalendarInterval literal throws native exception instead of evaluating to null (#5058)
  • Area labels: area:expressions
  • Rationale: Same shape as Hashing a CalendarInterval value fails with "Unsupported data type in hasher: Interval(MonthDayNano)" #5059 — the type passes isSupportedDataType but create_null_literal has no Interval arm, so a null literal fails the query at execution time instead of falling back.

priority:medium

• Native make_date rejects Spark-valid years outside chrono range (#5208)
  • Area labels: area:expressions
  • Rationale: Comet returns NULL (non-ANSI) or throws where Spark returns a valid date, but only for years beyond chrono's ±262143 range, so the practical surface is negligible; kept at priority:medium with an escalation note below rather than critical.
• spark.comet.exceptionOnDatetimeRebase is dead code: wire it up or remove it (#5195)
  • Area labels: area:scan
  • Rationale: A user-facing, documented config that promises an exception on legacy-calendar data is never read, so users get silence and wrong results instead; a broken feature with a workaround (spark.comet.scan.enabled=false) → priority:medium.
• Native Parquet schema-on-read diverges from Spark for ANSI interval targets (#5188)
  • Area labels: area:scan
  • Rationale: Comet reinterprets an INT column as microseconds for a read Spark rejects outright with PARQUET_COLUMN_DATA_TYPE_MISMATCH; no query Spark itself accepts returns wrong data, so it is a broken-compatibility case rather than critical — see Escalations.
• fix: string-to-timestamp does not trim ISO control characters, and leading '+' returns null under ANSI (#5165)
  • Area labels: area:expressions
  • Rationale: Comet returns NULL where Spark parses successfully (C0 control characters) and misses an ANSI CAST_INVALID_INPUT; the surface is limited to control-character-padded input and one ANSI rejection path, so priority:medium with an escalation note.
• [Bug] Native make_interval overflows or loses precision for valid Spark seconds (#5131)
  • Area labels: area:expressions
  • Rationale: Genuine precision loss and overflow versus Spark, but CometMakeInterval is marked Incompatible by default so the native path requires explicit opt-in — no exposure in the default configuration.
• Native columnar-to-row conversion is much slower than the JVM implementation for small batches (#5112)
  • Area labels: area:ffi, performance
  • Rationale: Reclassified from enhancement to bug: the native converter is enabled by default and measures 3.7x–15.7x slower per row than the JVM path it replaced, which the guide lists as a performance regression at priority:medium (workaround: disable the native converter).
• negative: ANSI overflow check reads null slots and can raise spurious overflow errors (#5093)
  • Area labels: area:expressions
  • Rationale: A MIN value left in a null slot raises a spurious ARITHMETIC_OVERFLOW where Spark returns null — a visible functional bug that requires the null slot's residual bytes to be exactly MIN, so priority:medium rather than high.
• [EPIC] ANSI mode audit follow-ups (vs Spark 4.1.1) (#5078)
  • Area labels: area:expressions, EPIC
  • Rationale: Tracker for a filed set of children; all of its silent-wrong-result children (IntegralDivide returns wrong result for Long.MinValue div -1 under ANSI mode #5065, MakeDecimal ignores fail_on_error: returns NULL where Spark throws under ANSI mode #5066, Remainder on Float/Double returns NaN for zero divisor under ANSI mode instead of throwing #5067, round on LongType with scale <= -19 returns 0 instead of overflowing #5070, DecimalPrecision.promote uses session ANSI flag instead of per-expression eval mode #5075) are already closed, so what remains open is error-fidelity work (ANSI arithmetic overflow errors: wrong error class for Byte/Short, wrong type names, missing try_ suggestions #5071–next_day and make_date ANSI errors surface as CometNativeException instead of Spark exception classes #5073, medium tier) plus hardening and test coverage — the EPIC is set to the highest remaining child tier.
• next_day and make_date ANSI errors surface as CometNativeException instead of Spark exception classes (#5073)
  • Area labels: area:expressions, correctness, temporal expressions
  • Rationale: The throw/NULL decision is already correct; only the exception type, error class, and SQLSTATE differ, so nothing is silent — a functional bug affecting users who match on Spark error conditions.
• Native ANSI errors raised as Arrow errors bypass SparkError conversion (wide decimal, decimal divide, decimal-to-decimal cast) (#5072)
  • Area labels: area:expressions, correctness
  • Rationale: Same class as next_day and make_date ANSI errors surface as CometNativeException instead of Spark exception classes #5073 — the error is raised, but as a generic CometNativeException without the Spark error class, SQLSTATE, or query context; visible failure, so priority:medium.
• ANSI arithmetic overflow errors: wrong error class for Byte/Short, wrong type names, missing try_ suggestions (#5071)
  • Area labels: area:expressions, correctness
  • Rationale: Error-fidelity only: Comet raises ARITHMETIC_OVERFLOW where Spark raises BINARY_ARITHMETIC_OVERFLOW, names Long overflow "integer", and omits the try_ suggestion. The overflow itself is detected correctly.
• Cast from boolean to decimal ignores eval mode: throws where Spark returns NULL in legacy/try mode (#5068)
  • Area labels: area:expressions, correctness
  • Rationale: The pair is marked Compatible() in all eval modes but the native kernel takes no eval_mode, so Comet throws where Spark returns NULL on Spark 3.x, with ANSI off, and for try_cast; the divergence is a visible error rather than a silent wrong value.

priority:low

• Extended explain operator stats miscount reuse wrappers and CometSubqueryBroadcast (#5203)
  • Area labels: none
  • Rationale: Affects only the Comet accelerated N out of M eligible operators line in extended explain output; query results and execution are unaffected — cosmetic/reporting per the guide.
• Expression explain tags leak between unrelated plans via the shared Literal.TrueLiteral singleton (#5229)
  • Area labels: none
  • Rationale: A [COMET-INFO: ...] message is attached to an unrelated operator in an unrelated query; misleading diagnostics only, with no effect on planning decisions or results.
• cast_map_to_map drops entries null buffer and ignores target sorted flag (#5097)
  • Area labels: area:expressions
  • Rationale: The reporter states both defects are latent today — Spark-produced maps carry no null map entries, and the sorted flag is metadata-only — so nothing is observable through normal plans; a minor issue per the guide.

Enhancements

• Replace hand-coded rollup of expression fallback reasons onto operators (#5230)
  • Area labels: none
  • Rationale: Refactor of the FALLBACK_REASONS roll-up mechanism (~200 hand-written call sites) to make the lifting automatic and type-safe; the two historical bugs it cites are already fixed, so this is maintainability work rather than a live defect.
• Deduplicate Throwable cause-chain traversal in Comet tests (#5223)
  • Area labels: none
  • Rationale: Refactor that lifts a duplicated causeChain helper into CometTestBase; test-code cleanup with no behavior change.
• Enable SPARK-57298 collect_set tests when Spark 4.2 SQL diff lands (#5209)
  • Area labels: area:aggregation, spark sql tests
  • Rationale: Test-enablement task blocked on a Spark 4.2 diff and CI job that do not exist yet; nothing is broken today.
• Do not accelerate a query stage that contains a columnar-to-row fallback (enable whole-stage revert by default) (#5207)
  • Area labels: performance, user experience
  • Rationale: Proposes flipping RevertNativeForTransitionHeavyStages on by default and making its strictest setting usable — a default/config change to improve out-of-the-box behavior.
• [EPIC] Planner, serde, and optimizer rule performance audit (#5199)
  • Area labels: performance, EPIC
  • Rationale: Collects driver-side planner and serde optimization opportunities from a profiling pass; performance work, which the guide classifies as enhancement.
• Shuffle performance audit: candidate optimizations needing investigation (#5198)
  • Area labels: area:shuffle, performance
  • Rationale: An explicitly unbenchmarked work list of shuffle optimization candidates — investigation and optimization, not a defect.
• Avoid duplicate CheckOverflow evaluation for decimal division (#5190)
  • Area labels: area:expressions
  • Rationale: Two nested equivalent CheckOverflow wrappers both scan the batch; removing the redundant pass is a performance optimization with identical results.
• Fall back instead of asserting when native scan partition values cannot be serialized (#5189)
  • Area labels: area:scan
  • Rationale: Hardening of an invariant — the reporter states there is no currently known unsupported partition literal that passes the gate, so this prevents a future failure rather than fixing a live one.
• Revisit CometCastSuite assumptions now that Cast has a codegen dispatch fallback (#5186)
  • Area labels: area:expressions
  • Rationale: Test-suite modernization: assumptions written before CodegenDispatchFallback now force working coverage to stay ignored.
• [DISCUSS] Is Comet ready to move to top-level Apache Comet project? (#5184)
  • Area labels: none
  • Rationale: Project-governance discussion thread, not a defect; classified as enhancement because the guide's type labels are exhaustive and nothing is broken.
• Make Comet even friendlier to agentic development (#5178)
  • Area labels: none
  • Rationale: Planned contributor-guide and skill improvements — documentation and tooling additions.
• [EPIC] Optimize native cast expression kernels (#5128)
  • Area labels: area:expressions, performance, EPIC
  • Rationale: Tracks per-row cost removal in the native cast kernels with a bit-identical-output requirement; pure optimization work.
• Support inline and stack generators in GenerateExec (#5125)
  • Area labels: area:expressions
  • Rationale: New generator support (inline, stack) that currently falls back to Spark as designed — added functionality.
• Add native support for CollectMetricsExec (df.observe) to preserve stage fusion (#5124)
  • Area labels: none
  • Rationale: New native operator for an operator that is not yet expected to run natively.
• Extend native Arrow UDF path to grouped-aggregate, window, and applyInArrow Python operators (#5123)
  • Area labels: area:udf
  • Rationale: Extends an existing opt-in native path to additional Python operators that currently fall back by design.
• Accelerate row-level MERGE / UPDATE / DELETE plans (MergeRowsExec, ReplaceDataExec, WriteDeltaExec) (#5122)
  • Area labels: area:writer, area:Iceberg
  • Rationale: New operator support for row-level plans that fall back entirely today.
• Accelerate DataSource V2 writes (AppendDataExec, OverwriteByExpressionExec, OverwritePartitionsDynamicExec) (#5121)
  • Area labels: area:writer, area:Iceberg
  • Rationale: New write-path support; DSv2 writes are not yet expected to be accelerated.
• Optimize JVM columnar-to-row conversion (#5119)
  • Area labels: performance, EPIC
  • Rationale: Removes measured allocation and virtual-dispatch costs from the interpreted JVM C2R path; output is unchanged.
• Use Spark ResourceProfiles to give native and JVM stages independent memory configs (#5116)
  • Area labels: none
  • Rationale: New capability enabled by stage-based fallback (feat: Stage based fallback #4519) — per-stage memory provisioning that does not exist today.
• Add support for SampleExec (DataFrame.sample, TABLESAMPLE, randomSplit) (#5109)
  • Area labels: none
  • Rationale: New operator support; SampleExec currently falls back as designed.
• [EPIC] Replace hand-rolled native code with existing arrow-rs kernels (#5104)
  • Area labels: EPIC, good first issue
  • Rationale: Tracks refactoring duplicated logic onto arrow-rs kernels; every candidate was pre-screened for Spark-semantics equivalence, so behavior is intended to be unchanged.
• Consolidate date/timestamp truncation and xxhash64 with upstream DataFusion implementations (#5103)
  • Area labels: area:expressions
  • Rationale: Upstream consolidation of two hand-rolled implementations; refactor with no behavior change intended.
• rlike: consider arrow regexp_is_match kernel and fix non-StringArray panic (#5102)
  • Area labels: area:expressions
  • Rationale: Primarily a kernel-consolidation proposal with unresolved trade-offs (per-batch regex compilation); the expect-on-non-StringArray hardening it mentions is latent, since no current path feeds Utf8View/LargeUtf8 here.
• list_extract: replace per-row MutableArrayData gather with take and zip (#5100)
  • Area labels: area:expressions
  • Rationale: Replaces a hand-rolled per-row gather with take + zip; identical null propagation, expected performance win.
• size: compute list sizes with the arrow length kernel (#5099)
  • Area labels: area:expressions
  • Rationale: Vectorizes a per-row loop using the arrow length kernel with the same Spark legacy semantics.
• sum_int: use arrow sum kernels and collapse integer type dispatch (#5098)
  • Area labels: area:aggregation
  • Rationale: Collapses six duplicated dispatch blocks onto arrow's sum kernels; refactor plus SIMD win, with the accumulation-order edge case called out.
• Use arrow dictionary casts instead of hand-rolled dictionary handling in cast paths (#5096)
  • Area labels: area:expressions
  • Rationale: Replaces a degenerate hand-built dictionary and a manual unpack branch with arrow's dictionary cast; Spark-owned semantics stay in the value cast.
• Delegate int/float/boolean to decimal cast arms to arrow safe cast (#5095)
  • Area labels: area:expressions
  • Rationale: The arms were verified line-for-line identical to arrow-cast 58.4.0, so delegating is a refactor with bit-identical output.
• decimal_rescale_check: replace fused rescale and precision check with arrow decimal cast (#5094)
  • Area labels: area:expressions
  • Rationale: Arrow's decimal cast already does the rescale, HALF_UP rounding, and precision validation in one pass; replacing the hand-rolled version is a cleanup.
• checked_arithmetic: delegate ANSI integer arithmetic to arrow checked kernels (#5092)
  • Area labels: area:expressions
  • Rationale: Calls arrow's checked kernels via the Datum path instead of re-implementing them; the error remap and the non-replaceable float/Try branches stay.
• Replace small hand-rolled element loops with arrow kernels and arity helpers (#5091)
  • Area labels: area:expressions
  • Rationale: A batch of small equivalent-result loop replacements (decimal-to-boolean via neq, validity masks via is_not_null/and, days_to_date, pow arity).
• Use arrow cast for hand-rolled temporal unit conversion loops (#5090)
  • Area labels: area:expressions, area:scan
  • Rationale: The loops were verified bit-identical to arrow's cast against the arrow-cast 58.4.0 source; touches both the spark-expr conversions and the Parquet cast_column path.
• Remove is_valid_decimal_precision duplicated from arrow-rs (#5089)
  • Area labels: area:expressions
  • Rationale: Deletes a character-for-character copy of arrow's implementation whose own comment says to remove it once arrow-rs fix: [branch-1.0] log partial memory grants at DEBUG and drop the memory usage dump (#6269) #6419 shipped; no semantic risk.
• Triage the spark.sql.legacy.* configs dropped from the session-wide fallback in chore: fallback to Spark if legacy sql configurations are set #4799 (#5087)
  • Area labels: none
  • Rationale: An audit task to determine whether each remaining curated legacy config is respected or safe to ignore; no specific defect is asserted yet, so any bugs it uncovers should be filed separately.
• Clean up dead ANSI-related plumbing (CometEvalMode helpers, inert allow_incompat on casts) (#5077)
  • Area labels: area:expressions
  • Rationale: Removal or documentation of unreachable helpers and an inert serialized field; explicitly a cleanup, and it notes which error variants must be kept for other fixes.
• ANSI mode test coverage: re-enable stale ignored tests and add missing cases (#5076)
  • Area labels: area:expressions, test
  • Rationale: Test-coverage work — un-ignoring files whose blocking issue (ANSI mode array access error messages don't match Spark format #3375) is closed and adding missing ANSI cases; no product defect claimed.
• Guard against silent fail_on_error loss in scalar function wiring (#5074)
  • Area labels: area:expressions
  • Rationale: Hardening against a class of mistake that has happened once and is now fixed; the reporter verified no currently wired datafusion-spark function is ANSI-sensitive, so this is preventative.
• Cast from float/double to decimal should return NULL for NaN/Infinity under ANSI mode (#5069)
  • Area labels: area:expressions
  • Rationale: The pair is already marked Incompatible and requires allowIncompatible=true, so this removes one of the divergences blocking compatibility rather than fixing a promised behavior.
• [EPIC] Complete interval type support: scan, operator gates, and remaining interval expressions (#5061)
  • Area labels: area:expressions, area:scan
  • Rationale: Tracks completing interval support — mostly opening Scala-side type gates for functionality the native engine already handles; the individual hard failures it found are filed separately (Null CalendarInterval literal throws native exception instead of evaluating to null #5058, Hashing a CalendarInterval value fails with "Unsupported data type in hasher: Interval(MonthDayNano)" #5059).
• Support reading ANSI interval columns (YearMonthIntervalType / DayTimeIntervalType) in the native Parquet scan (#5060)
  • Area labels: area:scan
  • Rationale: Unsupported scan types fall back cleanly today, as designed; adding them to the supported set is new functionality.

Escalations to consider

• Codegen dispatcher: whole-tree NullIntolerant short-circuit suppresses ANSI errors (#5218)
  • Guide trigger: prioritization principle 1, "Correctness over crashes." This was filed as priority:high and has been raised to priority:critical because the short-circuit turns a Spark ANSI error into a silent NULL. A reviewer who reads the divergence as error-semantics-only (the reporter's framing: "values are unaffected") may want to move it back to priority:high.
• Native Parquet schema-on-read diverges from Spark for ANSI interval targets (#5188)
  • Guide trigger: correctness-over-crashes. Left at priority:medium because the wrong-results direction only occurs for a read Spark itself rejects with PARQUET_COLUMN_DATA_TYPE_MISMATCH, so no Spark-valid query gets bad data; a reviewer who counts silent reinterpretation of INT as microseconds as data corruption regardless of Spark's rejection should escalate to priority:critical.
• fix: string-to-timestamp does not trim ISO control characters (#5165)
  • Guide trigger: correctness-over-crashes. Comet returns NULL where Spark parses successfully, which is silent. Kept at priority:medium because the surface is control-character-padded input, but a reviewer who has seen this in real ingest data (logs, CSV) should escalate — note the same file's date_parser already trims correctly, so CAST(... AS DATE) and CAST(... AS TIMESTAMP) disagree.
• Native make_date rejects Spark-valid years outside chrono range (#5208)
  • Guide trigger: correctness-over-crashes. Silent NULL where Spark returns a date, so the decision tree reads critical, but the input range (years beyond ±262143) makes production exposure effectively nil; escalate if any real workload is found.
• [EPIC] Memory pool and accounting audit sweep (#5212)
  • Guide trigger: "A priority:medium bug ... affects a common workload → consider escalating." Already at priority:high; flagged here because finding 1 affects the default pool type (fair_unified) for every task, so a reviewer may want the individual sub-issues split out and prioritized independently rather than tracked only under the EPIC.

Skipped — needs more info

• Bug triage results: 2026-07-27 (#5052)
  • Prior triage summary issue still carrying the auto-applied requires-triage label; not itself a bug or enhancement — a reviewer should sanity-check the referenced results and close it.
• Bug triage results: 2026-07-20 (#4980)
  • Prior triage summary issue still carrying the auto-applied requires-triage label; not itself a bug or enhancement — a reviewer should sanity-check the referenced results and close it.
• Bug triage results: 2026-07-13 (#4905)
  • Prior triage summary issue still carrying the auto-applied requires-triage label; not itself a bug or enhancement — a reviewer should sanity-check the referenced results and close it.
• Bug triage results: 2026-07-06 (#4838)
  • Prior triage summary issue still carrying the auto-applied requires-triage label; not itself a bug or enhancement — a reviewer should sanity-check the referenced results and close it.
• Bug triage results: 2026-06-29 (#4751)
  • Prior triage summary issue still carrying the auto-applied requires-triage label; not itself a bug or enhancement — a reviewer should sanity-check the referenced results and close it.