[krickert]

So when embedding on a GPU there are 3 padding strategies we are testing. I've tried all 3 of these strategies before, and a embeddings fine-tuner this is one aspect to get a max vertical scaling. As you can see from below, you can save a lot of compute time and energy consumption when digging this deep.

SentenceVectorsDL.embedAll can batch texts of different lengths together on a GPU, reducing the number of inference calls. SentenceVectorsOptions selects pooling, normalization, maximum token length and padding; InferenceOptions supplies the ONNX session settings separately.

PaddingStrategy supports:

• AUTO, the default: LONGEST on a GPU for token-output models whose vocabulary has a padding token, EXACT_LENGTH otherwise. Models with pooled output retain EXACT_LENGTH because they may ignore the attention mask.
• EXACT_LENGTH: group texts by tokenized length without padding.
• LONGEST: order texts by length and pad each batch to its longest text.
• MAX_LENGTH: pad every input to the configured maximum token length, including single embed calls. If construction must probe an unknown output dimension, that inference also uses the maximum length, which must fit the model's position limit.

The onnx provider accepts auto, exact_length, longest and max_length in any letter case. Its CPU session resolves AUTO to EXACT_LENGTH. Null padding is rejected. The unreleased seven-argument constructor is removed; callers supplying session settings use the constructor accepting SentenceVectorsOptions and InferenceOptions. Older convenience constructors remain.

Padding uses the vocabulary's [PAD] token, or for RoBERTa. Attention-mask entries for padding are zero, and pooling excludes padded positions. Vectors match up to floating-point rounding when the model honors the mask. A pooled output that ignores the mask can change with explicit padding. Batches are capped at 16,384 token positions, except that a single wider row runs alone.

Release note: GPU sentence-vector sessions now select LONGEST automatically for eligible models. Mixed-length batches can introduce small floating-point differences. Set PaddingStrategy.EXACT_LENGTH in SentenceVectorsOptions to retain unpadded GPU batches. The README migration notes, InferenceOptions.setGpu Javadoc and manual document this behavior.

The existing benchmark measurements on all-MiniLM-L6-v2, RTX 4080 SUPER, with texts of 4 to 40 words are retained below. They were not rerun for this API and documentation follow-up.

Texts	EXACT_LENGTH	LONGEST
8	3603 us	862 us
32	11431 us	2152 us
128	23815 us	7443 us

Tests cover batch boundaries, input order, vocabulary padding IDs, mask behavior, pooled output, single-input MAX_LENGTH, automatic defaults, null validation, case-insensitive provider values, and CPU/GPU parity. The follow-up tests first failed with 2 failures and 9 errors before implementation.

Validation: JDK 21 reactor verify for opennlp-dl-gpu and opennlp-docs with their dependencies reports 8,495 unit cases and 19 integration cases, zero failures/errors, and 12 unit skips. All 1,313 GPU-module tests ran without skips on RTX 4080 SUPER with CUDA 13 libraries. Packaging, Javadoc, Checkstyle, forbidden APIs, RAT and manual generation passed. The existing NameFinderDL Javadoc reference warning remains. A corpus eval build is still required before merge.

Quick merge note -
This pull request and #1472 both change SentenceVectorsDL.run. The one merged later needs to be reconciled with the other so padding uses the pooled direct buffers.

Fixes #1392

[rzo1]

Hi @krickert - thanks, nice speedup on the GPU. Some remarks inline. In addition, .gitignore needs a negation for tiny-masked.onnx like for the other tiny models.

WDYT?

[rzo1]

My earlier points are addressed. Whichever of this and #1472 lands second needs a rebase, both touch run().