[yjhjstz]

When a memo group's cached statistics lack columns needed by a later derivation request (e.g. a ROLLUP expanded into a CTE whose consumers request different column sets), the missing columns are derived separately and merged via CStatistics::AppendStats(). That merge copied histograms and widths only, so the colid -> attno mapping (added for extended statistics) diverged from the histogram map: a column could have a histogram but no attno entry.

ApplyCorrelatedStatsToScaleFactorFilterCalculation() then dereferenced the result of the attno lookup without a null check and crashed the coordinator with SIGSEGV whenever the table had any extended statistics object with stored data and a filter referenced one of the appended columns (e.g. TPC-DS Q67 after CREATE STATISTICS (ndistinct) on item/date_dim). Reproduced on main with a cassert build: colid 11 had a histogram (5 entries) but the attno map only held 4 entries, attnum was NULL at CExtendedStatsProcessor.cpp:280. This is the same class of bug as the CTE remapping fix in 124a048e8c6 (gpdb#16212), which repaired CopyStatsWithRemap() but not AppendStats().

Fix both layers:

• CStatistics::AppendStats() now also merges the colid -> attno mapping (copy-on-write, since the mapping is shared by refcount across stats objects, see ScaleStats()) and adopts the input's extended-stats info when none is set, restoring the invariant that every column with a histogram has an attno entry.

• CExtendedStatsProcessor now null-checks the attno and histogram lookups and skips clauses it cannot translate, falling back to the independence assumption, matching what ApplyCorrelatedStatsToNDistinctCalculation() already does. Also bail out gracefully if no ndistinct item matches the attribute set.

• CExtendedStatsProcessor also skips system columns (attno <= 0): extended statistics never cover them, and feeding a negative attno into CBitSet::ExchangeSet() would convert it to a huge unsigned bit index. The backend rejects these clauses too (dependency_is_compatible_clause()).

Add a regression test with the minimized reproduction: a ROLLUP over a join where the dimension table has ndistinct extended statistics and the filter column is appended to the scan group's stats after the initial derivation. Verified the gporca test passes with both optimizer=on and optimizer=off answer files.

Ported from warehouse-pg/warehouse-pg#251.

Fixes #1949

What does this PR do?

Type of Change

• Bug fix (non-breaking change)
• New feature (non-breaking change)
• Breaking change (fix or feature with breaking changes)
• Documentation update

Breaking Changes

Test Plan

• Unit tests added/updated
• Integration tests added/updated
• Passed make installcheck
• Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:

User-facing changes:

Dependencies:

Checklist

• Followed contribution guide
• Added/updated documentation
• Reviewed code for security implications
• This PR contains AI-assisted code generation
• Requested review from cloudberry committers

Additional Context

CI Skip Instructions

---

[leborchuk]

LGTM

[Alena0704]

It looks like colid = gpos::ulong_max; is missing here.
Without this reset, the next nested branch on the same column overwrites the previous branch’s scale factor instead of accumulating it, making OR estimates depend on branch order. Resetting colid preserves both branches’ contributions.

[yjhjstz]

Good catch, and it turns out the consequence is worse than just the scale factor.

The multi-column path at the bottom of the loop is otherwise only reached with colid == gpos::ulong_max, which makes IsNewStatsColumn() true at the top of the loop, so that path assumes the pending per-column state has already been flushed. By falling into it with a real colid still set, my patch broke that assumption twice:

1. as you say, a later disjunct on the same column sees previous_colid == colid, so previous_scale_factor is never appended to scale_factors and is overwritten instead of accumulated — the OR estimate becomes branch-order dependent;
2. the path's previous_histogram = nullptr; (a no-op in the original code, since the flush had already run) now drops a live CHistogram * — its contribution to the disjunction is lost and the object is leaked.

Fixed in a1af361 by flushing the pending histogram and scale factor in the fallback and then resetting colid:

if (nullptr != previous_histogram)
{
    scale_factors->Append(
        GPOS_NEW(mp) CDouble(previous_scale_factor.Get()));
    CStatisticsUtils::UpdateDisjStatistics(
        mp, non_updatable_cols, input_rows, cumulative_rows,
        previous_histogram, disjunctive_result_histograms,
        previous_colid);
    previous_histogram = nullptr;
}
is_colid_present = false;
colid = gpos::ulong_max;

The nullptr != previous_histogram guard is what distinguishes "state pending for this column" from "already flushed at the top of this iteration" — in the latter case previous_scale_factor still holds the value that was just appended, so an unconditional append would double-count it.

Verified on a cassert build (ORCA on): gporca (which carries this PR's new test), stats_ext, bfv_statistic, table_statistics, gporca_faults, qp_orca_fallback, bfv_joins, bfv_subquery, bfv_partition_plans, subselect_gp, qp_select, select, subselect, orca_static_pruning all pass.

[Alena0704]

would MDId()->Equals() be better?