[krickert]

Fixes #1471

SentenceVectorsDLBenchmark measures how fast SentenceVectorsDL embeds text, and how much heap, direct, and resident memory it uses, on the CPU or a CUDA GPU. It runs all-MiniLM-L6-v2 (384 dimensions) and e5-large-v2 (1024 dimensions) on calls of 1 to 128 texts, and reports latency percentiles, time per text and per token, and direct buffers and RSS through NativeMemoryProfiler. BENCHMARKS.md explains how to run it and lists results on an x86 desktop with an RTX 4080 SUPER and on a Jetson Orin Nano.

SentenceVectorsDL keeps the tensors of an inference in one direct buffer that ONNX Runtime reads and writes in place, instead of copying the inputs twice and building one float array per token for the output. Each running inference holds one buffer, which the instance keeps for reuse until it is closed. InferenceOptions.setMaxConcurrentInferences limits how many inferences one instance runs at once, one per available processor by default. A caller beyond the limit waits, so direct memory stays bounded, also with -XX:+DisableExplicitGC. Each buffer grows to the largest inference it has held, about 67 MB for a token output at 16384 positions with a hidden dimension of 1024. InferenceOptions.setIntraOpNumThreads fixes the ONNX Runtime intra-op thread count, and the benchmark uses 4 so CPU times compare across machines.

How an inference gets its tensor memory, 8 callers sharing one instance, e5-large-v2, 128 texts per call:

Device	Tensor memory	Mean	p99	Direct MB	Heap per call	RSS MB
CPU	ONNX Runtime output	10.07 s	12.23 s	4.5	15,960 KB	3,658
CPU	pool of 2, allocate on overflow	9.96 s	10.85 s	194.8	3,047 KB	3,089
CPU	pool of 2, wait	19.03 s	182.0 s	2.4	3,124 KB	3,033
CPU	one buffer per caller	9.73 s	10.75 s	9.5	3,164 KB	3,014
GPU	ONNX Runtime output	761 ms	800 ms	5.3	16,036 KB	3,191
GPU	pool of 2, allocate on overflow	777 ms	813 ms	301.0	3,040 KB	3,017
GPU	pool of 2, wait	985 ms	4,396 ms	2.4	3,124 KB	2,869
GPU	one buffer per caller	769 ms	798 ms	9.5	3,123 KB	2,800

One buffer per caller, which the default limit gives, is as fast as the ONNX Runtime output, keeps direct memory flat, and allocates a fifth of the heap. A pool smaller than the number of callers either lets direct memory grow or makes callers queue. BENCHMARKS.md has the full CPU and GPU comparison and the Jetson numbers.

This pull request and #1474 both change SentenceVectorsDL.run. The one merged later is rebased on the other, so padding on a GPU (#1474) runs on the pooled direct buffers.

[krickert]

@rzo1 I needed to make this to test the embeddings. I know before you were talking about expanding something like this/unifying it.. so I'm just keeping as a draft now. I'll add more details to the ticket a lil later

[rzo1]

Hi @krickert - thanks for the benchmark, nice to have numbers for this. Some thoughts:

• I am not sure the allocation numbers tell the whole story. run now allocates new direct buffers on every inference, and the output buffer can get large (16384 positions × dimension × 4 bytes, so about 25 MB at 384). gc.alloc.rate.norm only looks at the heap, so to me the table mainly shows that the allocation moved off-heap. I am also a bit worried about the release side: direct memory is only freed after a GC, and with less heap garbage that happens less often. If someone runs with -XX:+DisableExplicitGC, I would expect OutOfMemoryError: Cannot reserve direct buffer memory sooner or later. Could we reuse the buffers (per thread, for example) or stay with the ONNX Runtime output? It would also be good to see direct memory in the benchmark.
• The dimension > 0 branch reads a bit odd to me, since run now has a second path that is only used by the constructor. Can't we take dimension from the output metadata?
• The jmh profile in opennlp-dl/pom.xml is more or less a copy of the one in opennlp-runtime. Maybe we can move it to the parent.
• In BENCHMARKS.md I would either show the numbers or drop the line pointing to the PR.
• For the CPU times, fixing the thread settings might make them comparable. I think that is where allocateDirect would actually show up.

WDYT?

[krickert]

I agree with most - this is the first of an array of tests we should write. This is just a simple first round.

I'll add a few more. The memory tuning suggestions make sense.

[krickert]

I did everything requested -

On the numbers not telling the whole story: agreed, by design it's the first of a set.

It now runs a 384 and a 1024 dimension model and reports latency percentiles and the time per text and per token next to the average. Next I would like to add cold start (opening the session and the first call), concurrent callers, longer passages, and padded vs exact-length batching once #1474 is in.

Any others? I think that covers most speed concerns.

My summary followed by ai-assisted

1. Got the OOM error you talked about. Fixed it per your suggestion of a shared buffer. Details below
2. We can still add the thread count. I suggest we wait.
3. Other minor points mentioned below:

1. Direct buffers: you were right. With -XX:+DisableExplicitGC -XX:MaxDirectMemorySize=8m the old commit failed with OutOfMemoryError: Cannot reserve ... bytes of direct buffer memory. The tensors of an inference now share one direct buffer, and each instance keeps its largest one as a spare until close(), so repeated calls don't allocate direct memory. A concurrent call takes its own buffer. I went with reuse instead of going back to the ONNX Runtime output, because before this PR ONNX Runtime already copied our heap input arrays into a new direct buffer on every call. One spare per instance also avoids a ThreadLocal. SentenceVectorsDLDirectMemoryTest checks that the direct buffer count does not grow and that concurrent callers get their own vectors.
2. Dimension: it does come from the output metadata when the model declares it. The other case is an export with a symbolic hidden size. That is now measured once in the constructor (measureDimension), so run has no branch. A tiny test model with 1024 undeclared dimensions covers it.
3. jmh profile: moved to the parent POM and removed from opennlp-runtime and opennlp-dl.
4. BENCHMARKS.md: dropped the line pointing to the PR. It now lists what the benchmark reports. I will add fresh CPU and GPU numbers to the PR when I rerun.
5. CPU times: InferenceOptions does not expose the ONNX Runtime thread count yet, so for now BENCHMARKS.md says to pin the JVM to fixed cores with taskset. Making the thread count configurable would be a small separate PR if you think it is worth it.
6. Direct memory in the benchmark: -prof opennlp.dl.vectors.NativeMemoryProfiler reports the direct buffer bytes and count and the resident set size after each iteration.

[rzo1]

Hi @krickert - thanks, most of my points are addressed. Some remarks inline. In addition, .gitignore needs a negation for tiny-wide.onnx like for the other tiny models.

WDYT?

[rzo1]

With one spare buffer per instance, every call that finds it taken still allocates direct memory, so concurrent callers on a shared instance hit the same DisableExplicitGC issue as before. Could we let ONNX Runtime allocate the output in that case, or keep a small bounded pool? The Javadoc of run ("repeated calls allocate no new direct memory") and BENCHMARKS.md should say this holds for a single caller.

[rzo1]

The spare buffer stays at its largest size until close() (about 67 MB at dimension 1024). Please document this, or don't keep very large buffers.

[rzo1]

The buffer size is computed in int and can overflow for a large maxLength × dimension.