Skip to content

IGNITE-28899 Calcite. Fix litmus error during join - #13389

Open
zstan wants to merge 22 commits into
apache:masterfrom
zstan:ignite-28899
Open

zstan wants to merge 22 commits into
apache:masterfrom
zstan:ignite-28899

Conversation

@zstan

@zstan zstan commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

continue;

// Keys can relate to affinity not pk, just assumption here.
ImmutableIntList distKeys = table.distribution().getKeys();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

of course we need to have 'real pk' here, but fix it right now - seems too huge.

inputRows += mq.getRowCount(input);

double mem = 0.5 * inputRows * aggregateFieldsCount() * IgniteCost.AVERAGE_FIELD_SIZE;
double mem = 0.7 * inputRows * aggregateFieldsCount() * IgniteCost.AVERAGE_FIELD_SIZE;

@zstan zstan Jul 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empiric constants need to be fixed cause row count estimation is changed for now and some tests (2-3) are fails, all empirics need to be reevaluated during further (i hope) statistics activity changes.

prepareTables();

String sql = "SELECT sum(i.price * i.amount)" +
String sql = "SELECT /*+ MERGE_JOIN */ sum(i.price * i.amount)" +

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes i observe hashJoin here, i really don`t know what algo is more preferably here and i think it need to be discovered more accurate in further activities.

private static final int DATE_DIM_SIZE = 73_049;

/** */
private static final String SELECT = "SELECT /*+ DISABLE_RULE(" +

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course it work perfectly well without this disablings, i just afraid about some king of flaky...

@zstan
zstan marked this pull request as ready for review July 24, 2026 12:30
@zstan
zstan requested a review from Copilot July 24, 2026 12:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Updates Calcite planner metadata/costing and merge-join execution behavior to address a Calcite litmus failure seen during join planning/execution (IGNITE-28899), and adds regression tests to cover the scenario.

Changes:

  • Reworked join row-count estimation and adjusted join cost formulas/constants to influence plan selection.
  • Modified merge-join runtime node control flow around buffering and early termination.
  • Added new planner/integration tests and updated several existing assertions to match the new behavior.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
modules/calcite/src/test/java/org/apache/ignite/testsuites/PlannerTestSuite.java Registers the new join row-count estimation planner test in the suite.
modules/calcite/src/test/java/org/apache/ignite/testsuites/IntegrationTestSuite.java Adds the new litmus regression integration test to the suite.
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/hints/JoinTypeHintPlannerTest.java Relaxes/adjusts hint-related plan assertions (notably around wrong-table hints).
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/UncollectPlannerTest.java Updates plan shape expectations for UNNEST + join planning.
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/PlanSplitterTest.java Refactors colocation assignments setup to reuse a single assignments instance.
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/JoinRowCountEstimationTest.java New planner tests validating join row-count estimation logic.
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/CorrelatedSubqueryPlannerTest.java Updates plan assertion for table scan condition presence.
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/AggregatePlannerTest.java Removes an early return that skipped part of a test scenario.
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/integration/QueryBlockingTaskExecutorIntegrationTest.java Adds hint to force merge join and simplifies plan matching into a single regex check.
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/integration/LitmusCheckIntegrationTest.java New integration test meant to catch Calcite litmus exceptions/regressions.
modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/integration/DistributedJoinIntegrationTest.java Forces MERGE_JOIN via hint for deterministic join plan coverage.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/util/Commons.java Adds a helper to build a target mapping from a projected bitset.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/set/IgniteSetOp.java Adjusts memory estimation coefficient used in set-op costing.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteNestedLoopJoin.java Revises NLJ cost computation to use metadata row-count and applies an additional penalty factor.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteMergeJoin.java Revises merge join cost computation to use metadata row-count.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteHashJoin.java Revises hash join cost computation to use metadata row-count.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteCorrelatedNestedLoopJoin.java Revises CNLJ cost computation to use metadata row-count.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/metadata/cost/IgniteCost.java Changes the hash lookup cost constant affecting planner decisions.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/metadata/IgniteMdSelectivity.java Updates metadata provider wiring and exposes selectivity constants for tests/consumers.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/metadata/IgniteMdRowCount.java Implements custom join row-count estimation with PK/FK heuristics and new metadata provider wiring.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/metadata/IgniteMdColumnOrigins.java Updates metadata provider wiring and minor indentation cleanup.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/MergeJoinNode.java Reworks buffering/join scheduling and replaces “finishing” cleanup with immediate downstream end conditions.
modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/CalciteQueryProcessor.java Removes a previously-added workaround disabling Correlate assertions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@zstan
zstan force-pushed the ignite-28899 branch 3 times, most recently from a6dbda5 to eea65ee Compare August 19, 2026 13:54
assertPlan(sql, schema, hasFragmentsCount(3).and(hasChildThat(isInstanceOf(IgniteTrimExchange.class))));
// First table is replicated and planned with Exchange to colocate data, but set of nodes for partitioned
// table is differ, so exchange is added after fragments split for colocation.
assertPlan(sql, schema, hasFragmentsCount(3)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current plan:

IgniteMergeJoin(condition=[=($0, $3)], joinType=[inner]): rowcount = 5.625E9, cumulative cost = IgniteCost [rowCount=5.627075E9, cpu=2.2525784224829422E10, memory=4900000.0, io=0.0, network=4000000.0]
  IgniteSort(sort0=[$0], dir0=[ASC]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=575000.0, cpu=4600679.76331665, memory=900000.0, io=0.0, network=0.0]
    IgniteTableScan(table=[[PUBLIC, DEVELOPER]], filters=[=(+($t2, 1), ?0)]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=2000000.0, memory=0.0, io=0.0, network=0.0]
  IgniteExchange(distribution=[single]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=1500000.0, cpu=2.118354506610649E7, memory=4000000.0, io=0.0, network=4000000.0]
    IgniteSort(sort0=[$0], dir0=[ASC]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=1000000.0, cpu=2.068354506610649E7, memory=4000000.0, io=0.0, network=0.0]
      IgniteTableScan(table=[[PUBLIC, PROJECT]], requiredColumns=[{0, 2}]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=500000.0, memory=0.0, io=0.0, network=0.0]

master branch plan:

IgniteProject(ID=[$2], NAME=[$3], PROJECTID=[$4], ID0=[$0], VER0=[$1]): rowcount = 11250.0, cumulative cost = IgniteCost [rowCount=1672500.0, cpu=8572500.0, memory=1170000.0, io=0.0, network=225000.0]
  IgniteExchange(distribution=[single]): rowcount = 11250.0, cumulative cost = IgniteCost [rowCount=1661250.0, cpu=8561250.0, memory=1170000.0, io=0.0, network=225000.0]
    IgniteHashJoin(condition=[=($2, $0)], joinType=[inner]): rowcount = 11250.0, cumulative cost = IgniteCost [rowCount=1650000.0, cpu=8550000.0, memory=1170000.0, io=0.0, network=0.0]
      IgniteTableScan(table=[[PUBLIC, PROJECT]], requiredColumns=[{0, 2}]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=500000.0, memory=0.0, io=0.0, network=0.0]
      IgniteTrimExchange(distribution=[affinity[identity=hash, cacheId=1355342585][0]]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=575000.0, cpu=2300000.0, memory=0.0, io=0.0, network=0.0]
        IgniteTableScan(table=[[PUBLIC, DEVELOPER]], filters=[=(+($t2, 1), ?0)]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=2000000.0, memory=0.0, io=0.0, network=0.0]

// fragments split to send data to initiator node.
assertPlan(sql, schema, hasFragmentsCount(3).and(hasChildThat(isInstanceOf(IgniteTrimExchange.class))));
// table is differ, so exchange is added after fragments split for colocation.
assertPlan(sql, schema, hasFragmentsCount(2)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current plan:

IgniteMergeJoin(condition=[=($2, $3)], joinType=[inner], variablesSet=[[]], leftCollation=[[2]], rightCollation=[[0]]): rowcount = 5.625E9, cumulative cost = IgniteCost [rowCount=5.627075E9, cpu=2.2525784224829422E10, memory=4900000.0, io=0.0, network=4000000.0], id = 3041
  IgniteSort(sort0=[$2], dir0=[ASC]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=575000.0, cpu=4600679.76331665, memory=900000.0, io=0.0, network=0.0]
    IgniteTableScan(table=[[PUBLIC, DEVELOPER]], filters=[=(+($t2, 1), ?0)]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=2000000.0, memory=0.0, io=0.0, network=0.0]
  IgniteExchange(distribution=[single]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=1500000.0, cpu=2.118354506610649E7, memory=4000000.0, io=0.0, network=4000000.0]
    IgniteSort(sort0=[$0], dir0=[ASC]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=1000000.0, cpu=2.068354506610649E7, memory=4000000.0, io=0.0, network=0.0]
      IgniteTableScan(table=[[PUBLIC, PROJECT]], requiredColumns=[{0, 2}]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=500000.0, memory=0.0, io=0.0, network=0.0]

master branch plan:

IgniteProject(ID=[$2], NAME=[$3], PROJECTID=[$4], ID0=[$0], VER0=[$1]): rowcount = 11250.0, cumulative cost = IgniteCost [rowCount=1672500.0, cpu=8572500.0, memory=1170000.0, io=0.0, network=225000.0]
  IgniteExchange(distribution=[single]): rowcount = 11250.0, cumulative cost = IgniteCost [rowCount=1661250.0, cpu=8561250.0, memory=1170000.0, io=0.0, network=225000.0]
    IgniteHashJoin(condition=[=($4, $0)], joinType=[inner], variablesSet=[[]]): rowcount = 11250.0, cumulative cost = IgniteCost [rowCount=1650000.0, cpu=8550000.0, memory=1170000.0, io=0.0, network=0.0]
      IgniteTableScan(table=[[PUBLIC, PROJECT]], requiredColumns=[{0, 2}]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=500000.0, memory=0.0, io=0.0, network=0.0]
      IgniteTrimExchange(distribution=[affinity[identity=hash, cacheId=1355342585][2]]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=575000.0, cpu=2300000.0, memory=0.0, io=0.0, network=0.0]
        IgniteTableScan(table=[[PUBLIC, DEVELOPER]], filters=[=(+($t2, 1), ?0)]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=2000000.0, memory=0.0, io=0.0, network=0.0]

assertPlan(sql, schema, hasFragmentsCount(3).and(hasChildThat(isInstanceOf(IgniteTrimExchange.class))));
// First table is replicated and planned with Exchange to colocate data, but set of nodes for partitioned
// table is differ, so exchange is added after fragments split for colocation.
assertPlan(sql, schema, hasFragmentsCount(2)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current plan:

IgniteMergeJoin(condition=[=($0, $3)], joinType=[inner], variablesSet=[[]], leftCollation=[[0]], rightCollation=[[0]]): rowcount = 5.625E9, cumulative cost = IgniteCost [rowCount=5.627075E9, cpu=2.2525784224829422E10, memory=4900000.0, io=0.0, network=4000000.0]
  IgniteSort(sort0=[$0], dir0=[ASC]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=575000.0, cpu=4600679.76331665, memory=900000.0, io=0.0, network=0.0]
    IgniteTableScan(table=[[PUBLIC, DEVELOPER]], filters=[=(+($t2, 1), ?0)]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=2000000.0, memory=0.0, io=0.0, network=0.0]
  IgniteExchange(distribution=[single]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=1500000.0, cpu=2.118354506610649E7, memory=4000000.0, io=0.0, network=4000000.0]
    IgniteSort(sort0=[$0], dir0=[ASC]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=1000000.0, cpu=2.068354506610649E7, memory=4000000.0, io=0.0, network=0.0]
      IgniteTableScan(table=[[PUBLIC, PROJECT]], requiredColumns=[{0, 2}]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=500000.0, memory=0.0, io=0.0, network=0.0]

master branch plan:

IgniteProject(ID=[$2], NAME=[$3], PROJECTID=[$4], ID0=[$0], VER0=[$1]): rowcount = 11250.0, cumulative cost = IgniteCost [rowCount=1672500.0, cpu=8572500.0, memory=1170000.0, io=0.0, network=225000.0]
  IgniteExchange(distribution=[single]): rowcount = 11250.0, cumulative cost = IgniteCost [rowCount=1661250.0, cpu=8561250.0, memory=1170000.0, io=0.0, network=225000.0]
    IgniteHashJoin(condition=[=($2, $0)], joinType=[inner], variablesSet=[[]]): rowcount = 11250.0, cumulative cost = IgniteCost [rowCount=1650000.0, cpu=8550000.0, memory=1170000.0, io=0.0, network=0.0]
      IgniteTableScan(table=[[PUBLIC, PROJECT]], requiredColumns=[{0, 2}]): rowcount = 500000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=500000.0, memory=0.0, io=0.0, network=0.0]
      IgniteTrimExchange(distribution=[affinity[identity=hash, cacheId=1355342585][0]]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=575000.0, cpu=2300000.0, memory=0.0, io=0.0, network=0.0]
        IgniteTableScan(table=[[PUBLIC, DEVELOPER]], filters=[=(+($t2, 1), ?0)]): rowcount = 75000.0, cumulative cost = IgniteCost [rowCount=500000.0, cpu=2000000.0, memory=0.0, io=0.0, network=0.0]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also want to show that previously expected plan still possible but for lower table size, if we try to change
protected static final int DEFAULT_TBL_SIZE = 10;//500_000;
below plan will be raized:

IgniteProject(ID=[$2], NAME=[$3], PROJECTID=[$4], ID0=[$0], VER0=[$1]): rowcount = 2.25, cumulative cost = IgniteCost [rowCount=28.25, cpu=83.0, memory=23.4, io=0.0, network=45.0], id = 4147
  IgniteExchange(distribution=[single]): rowcount = 2.25, cumulative cost = IgniteCost [rowCount=26.0, cpu=80.75, memory=23.4, io=0.0, network=45.0], id = 4146
    IgniteHashJoin(condition=[=($2, $0)], joinType=[inner], variablesSet=[[]]): rowcount = 2.25, cumulative cost = IgniteCost [rowCount=23.75, cpu=78.5, memory=23.4, io=0.0, network=0.0], id = 4145
      IgniteTableScan(table=[[PUBLIC, PROJECT]], requiredColumns=[{0, 2}]): rowcount = 10.0, cumulative cost = IgniteCost [rowCount=10.0, cpu=10.0, memory=0.0, io=0.0, network=0.0], id = 1784
      IgniteTrimExchange(distribution=[affinity[identity=hash, cacheId=1355342585][0]]): rowcount = 1.5, cumulative cost = IgniteCost [rowCount=11.5, cpu=46.0, memory=0.0, io=0.0, network=0.0], id = 4144
        IgniteTableScan(table=[[PUBLIC, DEVELOPER]], filters=[=(+($t2, 1), ?0)]): rowcount = 1.5, cumulative cost = IgniteCost [rowCount=10.0, cpu=40.0, memory=0.0, io=0.0, network=0.0], id = 1793

Thus i think - if we want a fine tuning here - it need to be done under different issue.

@zstan
zstan force-pushed the ignite-28899 branch 3 times, most recently from 809cdcb to 063753c Compare August 20, 2026 11:08
.and(hasDistribution(IgniteDistributions.affinity(0, null, "hash")))),
algo.rulesToDisable);

// TODO: https://issues.apache.org/jira/browse/IGNITE-16334 Eventually planner skips optimal join plan.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you delete the link? Does that mean the problem with the deleted link has been fixed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, i run huge number of runs here - all ok, thus i delete it.

rows * (IgniteCost.ROW_COMPARISON_COST + IgniteCost.ROW_PASS_THROUGH_COST), 0, rightSize, 0);
double rowCnt = mq.getRowCount(this);

RelOptCost cost = costFactory.makeCost(rowCnt,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep the CPU cost proportional to leftCnt * rightCnt? A nested-loop join still compares every input pair, while the output row count may be orders of magnitude smaller. Multiplying it by 10 does not preserve the actual complexity and may make NLJ incorrectly cheaper than a hash join.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All such playing with coeffs is some kind of hack and i agree that multiplying complexity looks strange but all these hacks need to be eliminated when we will implement a real statistics for calcite now - it`s just a kind of assumptions. The main goal of this fix - is to store test plans and expectations as more as possible. If you want to play with it - try to comment multiplication and run PlanSplitterTest tests for example.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the current cost model is heuristic and preserving existing plans is useful. However, output cardinality and the amount of work are different estimates. Even with exact statistics, an NLJ over two 1M-row inputs may perform up to 1e12 comparisons while producing a single row, so 10 * rowCnt still severely underestimates its CPU cost.
Could we keep rowCnt as the output cardinality, but calculate CPU from leftCnt * rightCnt, using rowCnt only for the pass-through part? Also, multiplyBy(10) scales memory cost as well as CPU, which seems unintended.
If this changes PlanSplitterTest, I suggest reviewing the affected plans and updating justified expectations rather than tuning a global coefficient specifically to preserve snapshots.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change it, if I correctly understand your idea.

/** {@inheritDoc} */
@Override public Double getRowCount(Join rel, RelMetadataQuery mq) {
return rel.estimateRowCount(mq);
return joinRowCount(mq, rel);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bypasses IgniteCorrelatedNestedLoopJoin.estimateRowCount(), which removes the join-condition selectivity because it is already applied by an external filter. The generic estimator may therefore count the condition twice. Could we preserve the specialized CNL handling here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missunderstand plz explain this ? Also seems this behind of scope of this issue but if this relevant (i still don`t know why) - seems it need to be investigated.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this change, getRowCount(Join, mq) called rel.estimateRowCount(mq). Due to virtual dispatch, an IgniteCorrelatedNestedLoopJoin invoked its specialized estimateRowCount() override.
CorrelatedNestedLoopJoinRule pushes the join condition into an external filter on the right input, while the resulting CNL still keeps the same join condition. Therefore, the right input row count already includes the condition selectivity. The generic joinRowCount() applies the join-condition selectivity again.
IgniteCorrelatedNestedLoopJoin.estimateRowCount() explicitly compensates for this by dividing the generic estimate by mq.getSelectivity(this, getCondition()).
Calling joinRowCount(mq, rel) directly bypasses that override and reintroduces the double application. This is relevant to this change because the previous dispatch preserved the specialized behavior, while the new dispatch does not.
Could we add a specific metadata overload for IgniteCorrelatedNestedLoopJoin, delegating to rel.estimateRowCount(mq), and cover it with a regression test?

@zstan zstan Aug 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I changed it.

@zstan

zstan commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Current cut shows:

TpchBenchmark, scale = 0.01, q = 5. (H2 just for comparison here, not related to an issue of course)

PR:
Benchmark             (engine)  (queryId)  (scale)  Mode  Cnt     Score     Error  Units
TpchBenchmark.cached   CALCITE          5     0.01  avgt    3    25.519 ±   4.548  ms/op
TpchBenchmark.cached        H2          5     0.01  avgt    3   193.700 ± 168.796  ms/op
TpchBenchmark.cold     CALCITE          5     0.01    ss       1093.271            ms/op
TpchBenchmark.cold          H2          5     0.01    ss        816.027            ms/op

master 
Benchmark             (engine)  (queryId)  (scale)  Mode  Cnt     Score    Error  Units
TpchBenchmark.cached   CALCITE          5     0.01  avgt    3   566.541 ± 49.195  ms/op
TpchBenchmark.cold     CALCITE          5     0.01    ss       1564.145           ms/op

@tkalkirill

Copy link
Copy Markdown
Contributor

Perhaps it would be more correct to compare before and after your changes?

@zstan

zstan commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Perhaps it would be more correct to compare before and after your changes?

I just have a fresh master rebase with my PR, but ok re-check it:

git checkout ff25813
with calcite module rebuild

Benchmark             (engine)  (queryId)  (scale)  Mode  Cnt     Score     Error  Units
TpchBenchmark.cached   CALCITE          5     0.01  avgt    3   603.527 ± 168.897  ms/op
TpchBenchmark.cold     CALCITE          5     0.01    ss       1530.101            ms/op

.and(hasDistribution(broadcast()))
))
)
.and(nodeOrAnyChild(isInstanceOf(IgniteUncollect.class)))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

current plan:

IgniteHashJoin(condition=[=($0, $2)], joinType=[inner], variablesSet=[[]]): rowcount = 1.0, cumulative cost = IgniteCost [rowCount=1000007.0, cpu=1000016.0, memory=30.4, io=0.0, network=4000000.0]
  IgniteExchange(distribution=[single]): rowcount = 500000.0
    IgniteTableScan(table=[[PUBLIC, HASH_TBL]]): rowcount = 500000.0
  IgniteUncollect
    IgniteProject(EXPR$0=[ARRAY(1, 2, 3)])
      IgniteValues(tuples=[[{ 0 }]])

master branch plan:

IgniteProject(ID=[$1], NAME=[$2], EXPR$0=[$0]): rowcount = 4.0, cumulative cost = IgniteCost [rowCount=2300018.0, cpu=9200030.0, memory=0.0, io=0.0, network=48.0], id = 2654
  IgniteExchange(distribution=[single]): 
    IgniteCorrelatedNestedLoopJoin(condition=[=($1, $0)], joinType=[inner], variablesSet=[[$cor0]], variablesSet=[[0]], correlationVariables=[[$cor0]]): rowcount = 4.0
      IgniteTrimExchange(distribution=[hash[0]])
        IgniteUncollect
          IgniteProject(EXPR$0=[ARRAY(1, 2, 3)])
            IgniteValues(tuples=[[{ 0 }]])
      IgniteTableScan(table=[[PUBLIC, HASH_TBL]], filters=[=($t0, $cor0.EXPR$0)]): rowcount = 75000.0

@zstan

zstan commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

@tkalkirill @alex-plekhanov make a review plz ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants