executor, distsql, sessionctx, util: add adaptive limit scan profiles

[solotzg]

What problem does this PR solve?

Issue Number: ref #66658

Problem Summary:

For ORDER BY ... LIMIT queries that can stop early, especially when the plan uses keep-order scans and IndexJoin, TiDB may read far more rows than needed before the root Limit stops the pipeline.

When the system is under pressure, this over-read can amplify RPC work, queueing, memory usage, and downstream task fan-out. In bad cases, slower cancellation causes more prefetched work, which further increases system pressure.

What changed and how does it work?

This PR introduces adaptive limit scan profiles behind a new feature gate:

tidb_enable_adaptive_limit_scan

The variable is disabled by default.

Main changes:

• Add an in-memory early-stop profile cache keyed by:
  • schema
  • SQL digest
  • plan digest
  • reader type
  • keep-order flag
  • limit bucket
• Register statement-scoped early-stop candidates during executor construction.
• Observe runtime rows/keys/actRows after statement execution and update the profile with EWMA signals.
• Use the learned profile cap on later executions to reduce unnecessary scan concurrency for repeated early-stop workloads.
• Apply the cap only when the session still uses the default scan concurrency, so explicit user/session tuning is respected.
• For IndexJoin, apply learned caps to:
  • IndexJoin inner concurrency
  • build-side IndexLookUp batch size/concurrency
  • build-side scan concurrency cap
  • build-side result channel size
• Do not throttle on profile miss. The first executions only collect samples.
• Do not throttle healthy profiles. A learned cap is applied only when it is lower than the default scan concurrency.
• Keep profile entries for 48 hours.
• Add metrics for adaptive limit scan lookup/cap/observe events.

This PR also keeps the earlier static keep-order LIMIT cap path for reader executors and adds adaptive profile support around it.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Unit tests:

./tools/check/failpoint-go-test.sh pkg/executor -run 'TestAdaptiveIndexJoinLimitSettings|TestApplyAdaptiveIndexJoinLimitSettingsToLookupThroughProjection|TestObserveEarlyStopProfile|TestSelectEarlyStopProfileCandidate' -count=1

Other local checks:

make bazel_prepare
make lint
git diff --cached --check

Manual E2E test steps:

1. Build TiDB from this branch.

make

2. Start a local playground with the newly built TiDB binary.

tiup playground nightly --tiflash=0 --db.binpath=/Users/solotzg/Work/tidb/bin/tidb-server

3. Prepare an issue planner: decrease concurrency / batch_size automatically for streaming plans with limit-clause #66658 style workload:

• outer_t: 400000 rows
• inner_t: 400000 rows
• outer index: idx_order_key(order_key)
• inner index: idx_join_key(join_key)
• target range: order_key BETWEEN 1 AND 392857
• target limit: LIMIT 1000

4. Run the target query with adaptive limit scan disabled.

SET tidb_index_join_batch_size = 25000;
SET tidb_index_lookup_join_concurrency = 5;
SET tidb_enable_adaptive_limit_scan = 0;

EXPLAIN ANALYZE
SELECT /*+ INL_JOIN(i) */ o.payload, i.v
FROM outer_t o USE INDEX(idx_order_key)
JOIN inner_t i USE INDEX(idx_join_key)
  ON o.join_key = i.join_key
WHERE o.order_key BETWEEN 1 AND 392857
  AND o.filter_col < 7
ORDER BY o.order_key
LIMIT 1000;

5. Enable adaptive limit scan, run the same query several times to warm up the profile, then run EXPLAIN ANALYZE.

SET tidb_enable_adaptive_limit_scan = 1;

E2E result:

Metric	OFF	ON
result rows	1000	1000
IndexJoin actRows	1000	1000
IndexJoin inner concurrency	5	1
build-side Projection actRows	32048	3988
build-side IndexLookUp actRows	32906	3988
build-side IndexRangeScan actRows	110464	26560
build-side Selection actRows	32906	5712
build-side table_task concurrency	5	2
RU	166.26	27.40
Limit/root execution time	49.3ms	25.2ms

The ON case keeps the SQL result unchanged while reducing build-side over-read from 32906 rows to 3988 rows, which is within LIMIT * 4.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 57ec0a1c-0bff-4d0c-8c16-9432d997f5ac

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign terry1purcell, xuhuaiyu for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• pkg/distsql/OWNERS
• pkg/executor/join/OWNERS
• pkg/metrics/OWNERS
• pkg/sessionctx/vardef/OWNERS
• pkg/sessionctx/variable/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment