Skip to content

Stable 25.8 Backport of #93663 - Fix trash in TokenExtractor - #2672

Open
zvonand wants to merge 2 commits into
stable-25.8from
backports/stable-25.8/93663
Open

zvonand wants to merge 2 commits into
stable-25.8from
backports/stable-25.8/93663

Conversation

@zvonand

@zvonand zvonand commented Oct 25, 2026

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Critical Bug Fix (crash, data loss, RBAC) or LOGICAL_ERROR

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

The usage of ngram_bf index on a non-UTF-8 data led to an uninitialized memory read, with values that could reside in the resulting index structure. Closes ClickHouse#92576. (ClickHouse#93663 by @alexey-milovidov)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

alexey-milovidov and others added 2 commits October 25, 2026 15:24
…rator

Fix trash in `TokenExtractor`

Conflicts committed as-is; the next commit resolves them:
- src/Interpreters/ITokenExtractor.cpp (both modified)
Applied the buffer-length checks in stable-25.8's pointer style (`*pos`, `*token_length`) and dropped the `SparseGramsTokenExtractor` code, which doesn't exist on this branch.
@zvonand zvonand added 25.8 25.8 Altinity Stable backport Backport stable stable-25.8 labels Oct 25, 2026
@zvonand

zvonand commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

Backporter: cherry-pick conflicts resolved by AI

Conflicted files:

  • src/Interpreters/ITokenExtractor.cpp (both modified)

Commits:

  • 8d261e6: cherry-pick with the conflicts committed as-is

  • 8148987: resolution

  • NgramTokenExtractor::nextInString: kept the upstream std::min cap that stops the token running past the end of the buffer, rewritten as *token_length = std::min(*token_length, length - *token_start);.

  • NgramTokenExtractor::nextInStringLike: kept upstream's split of pos = ++i into ++i; *pos = i;. The i + j < length loop bound applied cleanly.

  • DefaultTokenExtractor::nextInStringLike (upstream's SplitByNonAlphaTokenExtractor): added the *pos < length bound to the UTF-8 copy loop.

  • NoOpTokenExtractor::nextInStringLike: the conflict block pulled in upstream's whole SparseGramsTokenExtractor, which stable-25.8 doesn't have. I kept the branch's existing code and left that class out, so its one-line fix doesn't apply here.

@github-actions

Copy link
Copy Markdown

Workflow [PR], commit [8148987]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

25.8 25.8 Altinity Stable backport Backport stable stable-25.8

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants