GH-1479: Configure the tokenizer alphanumeric optimization with TokenizerCharacterPolicy instead of a regex by krickert · Pull Request #1481 · apache/opennlp

krickert · 2026-10-11T16:54:09Z

The tokenizer's alphanumeric optimization now tests candidate tokens with a TokenizerCharacterPolicy instead of a java.util.regex Pattern. TokenizerME applies the policy through its model's factory; TokSpanEventStream, TokenizerFactory, and the language Factory take it directly.

The Pattern API stays as a deprecated path for removal in 4.0: the Pattern variants of the TokenizerFactory constructor, TokenizerFactory.create, and the TokSpanEventStream constructor, plus TokenizerFactory.getAlphaNumericPattern(), Factory.getAlphanumeric(String), and Factory.DEFAULT_ALPHANUMERIC. Regex use is now confined to that path and the legacy manifest handling. The protected TokenizerFactory.init now takes a policy, so subclasses that override it must update. New TokenizerFactory overloads without the last argument use the policy for the language. Passing a null literal no longer compiles because it matches both the Pattern and the policy variants; use the new overloads without that argument.

The factory writes the policy to the model manifest as characterPolicy, using the text form of TokenizerCharacterPolicy.toString(), which TokenizerCharacterPolicy.parse(String) reads. For a language policy it also writes the alphaNumericPattern that earlier releases wrote for that language, so OpenNLP 2.x tokenizes such a model the same way. A model with one of those alphaNumericPattern values loads with the matching language policy; a model with a custom pattern still loads and tests candidates with that regular expression. A manifest with neither property uses the policy for its language code.

TokenizerCharacterPolicyBenchmark (JMH, JDK 25, nanoseconds per call over 50 candidate tokens) against the regular expressions of earlier releases:

Language	Policy (ns)	Regex (ns)
en	257	1003
es	291	2542
de	266	2460
pl	267	9646

TokenizerMEBenchmark.instancePerThread on English: 110,685 ops/s against 97,647 ops/s on main.

Tests pin each language policy against its old regular expression, pin TokenizerME output with and without the optimization, and cover a custom policy, the policy text form, the manifest with a policy, every legacy pattern, a custom and an invalid pattern, a malformed policy, a manifest with neither property, and the deprecated constructors, create, and getters. The tokenizer chapter and the README describe the policy, the deprecation, and the migration.

An eval build is needed before merge, since the tokenizer training and inference path changes.

Fixes #1479

rzo1

Hi @krickert, thanks for the PR. Before going through the details, a devil's advocate question on the approach:

The removed API (TokenizerFactory(…, Pattern), TokenizerFactory.create(…, Pattern), getAlphaNumericPattern(), the TokSpanEventStream constructor and Factory.getAlphanumeric) has a user-supplied regular expression as its contract, and the pattern is stored in the model manifest. This is exactly the case that #1348 and CONTRIBUTING.md cover with @SuppressForbidden instead of a rewrite.
A model trained with a custom alphaNumericPattern no longer loads in 3.0, and a custom policy written by 3.0 is silently ignored by 2.x. Removing public API that exists since 1.x and shipped in M6 without a deprecation step is a hard break for these users.
The main benefit I see is performance, since the pattern is matched for every candidate token. Could you add a JMH benchmark to quantify it?

Suggestion: map the known patterns to TokenizerCharacterPolicy, which covers the released models and gives the faster path for them, but keep the Pattern API as @Deprecated(forRemoval = true) and as fallback for custom patterns under @SuppressForbidden. Existing models keep loading, and the removal can follow in 4.0. WDYT?

krickert · 2026-10-12T05:59:55Z

I'm typically against fallbacks but the transition is a good compromise. The speed is a clear winner too :) Thanks for the suggestion.

rzo1

Hi @krickert - thanks, that addresses my concern from the first round. Some remarks inline.

WDYT?

krickert · 2026-10-18T14:14:02Z

Hi @krickert - thanks, that addresses my concern from the first round. Some remarks inline.

WDYT?

Thank you! All were good suggestions and updated in 150196dce

AI supplied evidence:

opennlp-runtime tokenize package: 608 tests pass, forbidden-API check passes.

rzo1

Hi @krickert - thanks, LGTM. One small thing inline.

mawiesne

No actual changes requested for now, still I'm throwing a blockage in here.

I need (time) to check some of the proposed changes in detail locally, hopefully confirming nothing breaks. Pls allow for 1-2 weeks time for deeper validation. Will come back to this PR.

rzo1

Code is fine from my side. Still missing the eval run the description asks for.

rzo1 · 2026-11-02T09:37:01Z

    TokenizerFactory factory = model.getFactory();
    this.cg = factory.getContextGenerator();
-    this.alphanumeric = factory.getAlphaNumericPattern();
+    this.alphanumeric = factory.alphanumericTest();


Subclasses overriding getAlphaNumericPattern() still compile, but the override is now ignored here and in createManifestEntries. Needs a README breaking-changes entry.

rzo1 · 2026-11-02T09:37:01Z

+    if (pattern == null) {
+      return;
+    }
+    TokenizerCharacterPolicy known = Factory.policyOfLegacyExpression(patternText(pattern));


Ignores pattern.flags(). A preset compiled with CASE_INSENSITIVE gets swapped for a stricter policy. Only map when flags() == 0?

rzo1 · 2026-11-02T09:37:01Z

+        readManifest();
+      }
+    } catch (IllegalArgumentException e) {
+      throw new InvalidFormatException(e.getMessage(), e);


A bad custom pattern now fails at model load with InvalidFormatException instead of PatternSyntaxException in TokenizerME. Fine, but put it in the README.

rzo1 · 2026-11-02T09:37:01Z

+  @Deprecated(since = "3.0.0", forRemoval = true)
+  @SuppressForbidden("alphanumeric pattern of earlier releases, kept for the deprecated API")
+  public Pattern getAlphanumeric(String languageCode) {
+    return compile(LEGACY_EXPRESSIONS.get(getCharacterPolicy(languageCode)));


Compiles a new Pattern on every call. These used to be constants, so getAlphanumeric(null) == DEFAULT_ALPHANUMERIC no longer holds. Cache them?

…izerCharacterPolicy instead of a regex TokenizerME, TokSpanEventStream, TokenizerFactory and the language Factory take a TokenizerCharacterPolicy for the alphanumeric optimization. The Pattern variants of the TokenizerFactory constructor, TokenizerFactory.create and the TokSpanEventStream constructor, getAlphaNumericPattern(), Factory.getAlphanumeric(String) and Factory.DEFAULT_ALPHANUMERIC stay as deprecated for removal in 4.0, and only they keep @SuppressForbidden. New overloads without the last argument use the policy of the language. The lang.Factory lookups that serve only the deprecated Pattern path are @internal and deprecated for removal with it. The factory writes the policy to the model manifest as characterPolicy, in the text form of TokenizerCharacterPolicy.toString(), which TokenizerCharacterPolicy.parse(String) reads; for a language policy it also writes the alphaNumericPattern of earlier releases. A model with one of the alphaNumericPattern values OpenNLP wrote loads with the matching language policy, and a model with a custom pattern loads and tests candidate tokens with that regular expression. A manifest without either property uses the policy for its language code. A pattern with the text of a language pattern given to the deprecated constructor or create is replaced by that policy only when it has no flags. The deprecated getters return the patterns of earlier releases compiled once, so Factory.getAlphanumeric(null) is Factory.DEFAULT_ALPHANUMERIC. The language policies are tested against their old regular expressions, TokenizerME output is pinned with and without the optimization, and the manifest tests cover the policy, every legacy pattern, a custom and an invalid pattern, a malformed policy and a manifest without either, plus the deprecated constructors, create and getters. TokenizerCharacterPolicyBenchmark compares the policy with the regular expressions. The tokenizer chapter and the README describe the policy, the deprecation, the migration, and the breaking edges: an ignored getAlphaNumericPattern() override and an invalid custom pattern failing at model load. Fixes #1479 Signed-off-by: Kristian Rickert <kristian@apache.org> Co-authored-by: Claude Code 2.1.296 (Opus 5.5) <noreply@anthropic.com>

rzo1 reviewed Oct 11, 2026

View reviewed changes

krickert force-pushed the feat/GH-1479-tokenizer-character-policy branch from ff7c076 to b928348 Compare October 12, 2026 05:31

krickert force-pushed the feat/GH-1479-tokenizer-character-policy branch from b928348 to 1cb19b3 Compare October 15, 2026 13:33

krickert marked this pull request as ready for review October 15, 2026 14:09

rzo1 requested changes Oct 17, 2026

View reviewed changes

rzo1 requested a review from mawiesne October 18, 2026 12:33

rzo1 assigned krickert Oct 18, 2026

rzo1 added this to the 3.0.0 milestone Oct 18, 2026

krickert requested a review from rzo1 October 18, 2026 14:14

rzo1 approved these changes Oct 18, 2026

View reviewed changes

Comment thread opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/tokenize/lang/Factory.java Resolved

mawiesne requested changes Oct 18, 2026

View reviewed changes

krickert force-pushed the feat/GH-1479-tokenizer-character-policy branch 2 times, most recently from 1da21cd to 83958f6 Compare October 24, 2026 03:25

rzo1 reviewed Nov 2, 2026

View reviewed changes

rzo1 force-pushed the feat/GH-1479-tokenizer-character-policy branch from 83958f6 to 559dbac Compare November 2, 2026 10:17

krickert force-pushed the feat/GH-1479-tokenizer-character-policy branch from 559dbac to 2da6c92 Compare November 3, 2026 21:18

Conversation

krickert commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

krickert commented Oct 12, 2026

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

krickert commented Oct 18, 2026

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mawiesne left a comment

Choose a reason for hiding this comment

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

rzo1 Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Nov 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

krickert commented Oct 11, 2026 •

edited

Loading