[ML] Fail gracefully when restoring a categorizer with an out-of-range token ID

[edsavage]

Summary

Fixes a residual native crash (SIGSEGV) when restoring a categorizer from an inconsistent or truncated state document, seen on a 9.6.0-SNAPSHOT build (see #2875).

CTokenListDataCategorizerBase::acceptRestoreTraverser restores a list of tokens (m_TokenIdLookup) and a list of categories. Each restored category references token IDs (its base and common-unique token IDs). Those IDs are later used to index m_TokenIdLookup without a bounds check — e.g. in cacheReverseSearch:

const CTokenInfoItem& info{m_TokenIdLookup[tokenId]};      // out of range -> garbage
... m_ReverseSearchCreator->addInOrderCommonToken(m_TokenIdLookup[tokenId].str(), ...);

m_TokenIdLookup is a Boost multi-index container whose random-access operator[] performs no bounds checking. If a corrupt/truncated state leaves a category referencing a token ID at or beyond the end of the restored lookup, this is an out-of-bounds access that returns a garbage CTokenInfoItem; calling .str() on it then dereferences a bogus pointer, crashing the autodetect process with a SIGSEGV inside libc (strlen/memcpy) rather than failing the restore.

This matches the crash signature reported in #2875 on 9.6: si_signo 11, si_code 1 (SEGV_MAPERR), crash PC in libc.so.6. The graceful invalid-state handling added in #2895 / #2898 does not cover this particular path.

Fix

Validate, at the end of acceptRestoreTraverser, that every token ID referenced by a restored category exists in the restored token ID lookup. If any is out of range, log an error and fail the restore gracefully (returning false), consistent with the intent of #2895 / #2898, instead of proceeding with an inconsistent state that crashes later.

Testing

Added two unit tests in CTokenListDataCategorizerTest:

• testRestoreWithInconsistentTokenIdFailsGracefully — a state with a single token but a category referencing token ID 5; restore now returns false (previously undefined behaviour / crash).
• testRestoreWithConsistentTokenIdSucceeds — a valid state referencing the only valid token ID (0) still restores successfully, confirming the new check does not reject good state.

Both pass, along with the existing testPersist round-trip test (so valid states are unaffected).

Relates to #2875

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @edsavage, I've created a changelog YAML for you.