Defer inline feature serialization and batch complete graph records

[tlwillke]

Purpose

Serialize complete graph records in bounded batches after construction, avoiding synchronized per-insertion NVQ/FP feature writes and repeated small final writes. The core writer uses the existing Feature interface and preserves V6; it does not depend on a particular vector type.

#744 is merged, and this PR targets main. Grid uses its VectorAccess.openRange API; this PR adds the deferred writer, its integration, documentation, and tests.

Changes

• With the parallel writer, Grid supplies NVQ/FP features during final serialization, after graph cleanup. With the sequential writer, it preserves encoding during insertion and incremental inline writes. Construction scoring and queries are unchanged.
• Parallel L0 serialization writes up to 64 complete records per worker buffer; partial tails are handled without changing record layout.
• withParallelBatchSize(int) optionally overrides 64 on both existing parallel-capable builders. It does not enable parallel writing or affect the sequential selection.
• Missing feature suppliers preserve the incremental/prewritten-feature path. Existing vector sources remain supported; buffered sources use Add optional bounded file-backed vector loading and batched access #744’s range API.
• Join workers before releasing suppliers, cursors, views, or buffers, including failure/interruption paths. Caller-owned executors remain caller-owned.
• Correct the pre-existing consumed-buffer offset bug in hole records and make their placeholder ordinals match the sequential V6 writer.

The core writer accepts ordinary application-supplied feature-state callbacks. It has no dependency on Grid, BenchYAML, dataset preloading, or #744's buffered hooks. The following change is needed only for callers that currently write features during insertion; callers already supplying all features to the parallel final writer automatically obtain complete-record batching.

Caller usage and production integration

Both examples assume the same trained nvq, graph builder, node count,
writerWorkers, and output path. The builder's construction scoring is unchanged;
its peer vectors are supplied separately. vectorForOrdinal is the application's
lookup of an original input vector, not a harness loader.

Old: write each inline feature during insertion

try (var writer = new OnDiskParallelGraphIndexWriter.Builder(builder.getGraph(), outputPath)
        .withParallelWorkerThreads(writerWorkers)
        .withMapper(new OrdinalMapper.IdentityMapper(count - 1))
        .with(new NVQ(nvq))
        .build()) {
    for (int ordinal = 0; ordinal < count; ordinal++) {
        var vector = vectorForOrdinal.apply(ordinal);
        builder.addGraphNode(ordinal, vector);
        writer.writeFeaturesInline(ordinal, Feature.singleState(
                FeatureId.NVQ_VECTORS, new NVQ.State(nvq.encode(vector))));
    }
    builder.cleanup();
    writer.write(Map.of()); // Preserve the earlier feature writes; add adjacency.
}

New: supply the features when writing the completed graph

for (int ordinal = 0; ordinal < count; ordinal++) {
    builder.addGraphNode(ordinal, vectorForOrdinal.apply(ordinal));
}
builder.cleanup();

try (var writer = new OnDiskParallelGraphIndexWriter.Builder(builder.getGraph(), outputPath)
        .withParallelWorkerThreads(writerWorkers)
        .withMapper(new OrdinalMapper.IdentityMapper(count - 1))
        .with(new NVQ(nvq))
        .build()) {
    writer.write(Feature.singleStateFactory(FeatureId.NVQ_VECTORS,
            ordinal -> new NVQ.State(nvq.encode(vectorForOrdinal.apply(ordinal)))));
}

The second call supplies NVQ payloads during final serialization, together with
adjacency, in batches of up to 64 complete records. No writeFeaturesInline pass
is needed. The same pattern works for FP using InlineVectors and
InlineVectors.State, or another supported inline feature. Supply every configured
inline feature to obtain complete-record batching, including fused PQ if configured.

Final suppliers run concurrently. vectorForOrdinal must remain available through
write and provide independent borrowed vectors per worker, or stable vectors.
For an existing RandomAccessVectorValues, threadLocalSupplier() supplies those
independent views; it does not require buffered hooks. A caller can instead return
already encoded feature states from its own storage, without retaining original FP
vectors for this write.

Integrations that already call write with every inline feature supplier receive
batching when using the parallel writer. Integrations that write features early
must move those calls to final suppliers to use the deferred path. The incremental
API remains compatible: a one-pass source whose values cannot be obtained again
must retain encoded states for final writing or continue using incremental writes.

Performance

Fresh construction-only BenchYAML/Grid runs on RAID, with existing main native Highway support, automatic avx3_spr, preferred width 512, JDK 23, and 48 insertion/cleanup/writer workers.

Setup for the experiments: GCE VM, 48-core Intel Xeon 6985P-C (96 threads), local SSD RAID10; JDK 23, native Highway. Heap and worker limits are specified per study.

All datasets use DOT_PRODUCT; PQ mFactor=8, K=256, centering=false, anisotropic threshold=-1; NVQ with two subvectors; degree 32, beam 100, overflow 1.2, hierarchy/refinement enabled, pruning disabled, fused PQ, V6. Every arm trains fresh codebooks and builds a fresh graph. Input file cache was explicitly evicted and verified before each run.

Input footprints: Cohere 10M 38.184 GiB, MPNet 10M 28.641 GiB, CAP 6M 34.758 GiB. Ample: 128 GiB heap, no cgroup total-memory cap. Limited: 32 GiB total cgroup memory, 20 GiB heap, zero swap. These are experiment settings, not committed defaults.

The before/after measurements below compare #744’s original inline-writing path (1bab1881) with the recorded 64-record integration (9194992f). They are separate fresh runs; training and parallel graph variation qualify attribution outside serialization.

Dataset	Inline final write, ample	Batch final write, ample	Inline total process, ample	Batch total process, ample	Batch total process, limited
Cohere 10M	148.50 s	64.02 s	633.04 s	549.01 s	531.77 s
MPNet 10M	136.10 s	50.69 s	541.15 s	407.89 s	391.17 s
CAP 6M	162.64 s	51.76 s	478.80 s	314.48 s	335.15 s

Limited-memory inline runs were excluded because the existing final-writing path slowed dramatically under the memory limit.

Phase breakdown (seconds)

Dataset	Arm / memory	PQ train	PQ encode	NVQ setup	Insertion	Cleanup	Final write	Pipeline
Cohere 10M	Inline / ample	8.13	37.67	4.66	403.19	29.52	148.50	631.67
Cohere 10M	Batch / ample	8.22	37.47	4.72	403.28	29.96	64.02	547.68
Cohere 10M	Batch / limited	8.73	39.15	14.73	378.65	30.29	58.37	529.91
MPNet 10M	Inline / ample	7.52	35.27	3.89	336.67	20.54	136.10	539.98
MPNet 10M	Batch / ample	6.13	29.59	3.79	295.40	20.97	50.69	406.56
MPNet 10M	Batch / limited	5.68	29.25	5.59	281.59	22.29	44.70	389.12
CAP 6M	Inline / ample	9.90	35.31	4.14	244.68	20.75	162.64	477.42
CAP 6M	Batch / ample	9.79	35.12	4.14	192.15	20.19	51.76	313.15
CAP 6M	Batch / limited	11.53	35.11	11.65	202.61	19.37	53.34	333.62

Pipeline is the sum of these phases. Total process additionally includes dataset preparation, setup, reporting, and shutdown. Final write includes deferred NVQ/FP feature encoding; previously that work occurred during insertion.

Memory (GiB)

Dataset	Arm / memory	Peak Java heap	Buffer pool capacity	Peak process RSS	Cumulative GC reclaimed
Cohere 10M	Inline / ample	15.38	49.65	17.07	600.38
Cohere 10M	Batch / ample	14.99	49.67	16.66	610.21
Cohere 10M	Batch / limited	13.69	49.67	15.19	633.27
MPNet 10M	Inline / ample	13.74	37.71	15.33	656.84
MPNet 10M	Batch / ample	18.38	37.73	21.82	564.28
MPNet 10M	Batch / limited	17.77	37.73	20.70	544.25
CAP 6M	Inline / ample	14.28	44.65	16.55	409.74
CAP 6M	Batch / ample	10.95	44.68	12.43	403.89
CAP 6M	Batch / limited	9.59	44.68	10.77	340.65

RSS is the separately measured Linux child-process peak (getrusage), not heap plus buffer pool. Mapped buffer capacity is not resident memory, and buffer pool excludes other native allocations. GC reclaimed is cumulative. MPNet’s ample-memory RSS increased; this is a throughput improvement, not a claim of universally lower resident memory.

Correctness and compatibility

• 29 focused tests passed with the native provider, including existing storage tests and legacy V0 fixtures.
• 64 combinations compare parallel output byte for byte with the sequential V6 writer: FP/NVQ plus fused PQ, complete/prewritten features, batch sizes 1/7/64/128, heap/direct buffers, reversed ordinals, holes, and an embedded starting offset.
• Separate tests cover multiple full batches and a partial tail, bounded range reads, ordinary borrowed-vector fallback, worker failure, cursor-close failure, and executor ownership.
• A negative control with the original offset handling fails the new prewritten-hole test; the corrected code passes.
• No recall acceptance criteria or query kernels changed. Byte equivalence for a fixed graph/codebook verifies serialization without conflating independently trained graphs.

Usage

See Deferred batch serialization. Applications opt in by supplying their final features to the parallel writer. The source must remain available through serialization and suppliers must support concurrent calls. Existing writeFeaturesInline callers remain supported.

The recorded measurements explicitly selected the parallel writer. BenchYAML and AutoBenchYAML now select it by default; -Djvector.parallel_writer=false preserves the sequential path. The library default remains sequential, and programmatic Grid callers retain their configured selection. useDatasetPrefetching controls input loading separately. Worker counts follow PhysicalCoreExecutor; no server-specific worker count or memory limit is committed.

Final-source confirmation

A fresh MPNet 10M run on da094d0d (production unchanged by the documentation commits) passed with the same 32 GiB total / 20 GiB heap / zero-swap configuration. Writing performance held near the recorded batch result; the lower total is mostly insertion variation in an independently trained graph.

PQ train	PQ encode	NVQ setup	Insertion	Cleanup	Final write	Pipeline	Total process
6.86 s	29.87 s	6.19 s	213.95 s	21.34 s	45.39 s	323.59 s	325.58 s

Peak Java heap 15.32 GiB; buffer pool capacity 37.73 GiB; separately measured peak RSS 17.50 GiB; cumulative GC reclaimed 550.05 GiB. Input file cache eviction, native dispatch, fresh training, cgroup membership, and zero swap were verified.

Preloaded V6 validation

A focused MPNet 1M comparison uses ordinary fully resident ListRandomAccessVectorValues in both arms, with useDatasetPrefetching=false. The control is #744's inline path (87bb4530); the candidate is #747 (cb2fc873). Both use native Highway, V6, the same PQ/NVQ and graph settings as above, 48 build/cleanup/writer workers, a 128 GiB heap, and no cgroup memory cap. Input files are on the boot drive; output indexes are on RAID for both arms. Each arm starts cold, preloads the input before construction, and independently trains/builds a fresh graph.

Metric	Preloaded inline	Preloaded deferred batch
PQ training (s)	6.06	6.46
PQ encoding (s)	3.19	3.19
NVQ setup (s)	0.31	0.30
Insertion (s)	37.75	24.85
Cleanup (s)	2.11	2.18
Final writing (s)	14.87	5.72
Upfront dataset preparation (s)	13.16	12.36
Pipeline (s)	64.29	42.71
Total process (s)	78.45	56.05
Peak Java heap (GiB)	10.37	10.03
Buffer pool capacity (GiB)	3.77	3.78
Peak process RSS (GiB)	12.99	12.40
Cumulative GC reclaimed (GiB)	67.69	61.66

Total process time decreased 28.6%; final writing decreased 61.6%. This demonstrates a benefit with resident input and no prefetcher. It is one fresh pair, so graph/training variation qualifies the insertion change; it is not a universal speedup guarantee.

Construction example refresh

DiskIntro, LargerThanMemory, NvqExample, SiftSmall and IPC serialization now supply final FP/NVQ features to the parallel writer. The parallel-write example uses the supported API and verifies all serialized features byte for byte. Tutorials document source lifetime and independent worker views. Java 11 compilation and six native-backed SIFT Small smoke checks passed; sequential and parallel V6 payloads matched. Query algorithms and quantization settings are unchanged.

Review fixes and validation

Thanks to Jonathan Shook for the repros and fixes in #752. All five adopted commits retain his authorship. Follow-up changes:

• Preserve the sequential Grid path, and default only BenchYAML/AutoBenchYAML to parallel writing with an explicit sequential override.
• Signal failure from the worker immediately, even while the coordinator joins an earlier range. Interruption/failure stops remaining records; every submitted task is still joined before resources are released.
• Remove unused record padding; production V6 records have exactly their serialized payload size. The earlier 4 KiB stride experiment is not part of this PR.
• Retain the deterministic resident/buffered adjacency comparison and add separate concurrent buffered-access coverage across exact/PQ/BQ scorers and distance metrics. Recall acceptance criteria are unchanged.

Validation of the updated source: 432 tests passed, 2 skipped, 0 failures across the core, examples, and native suites with -ea and the native provider. Production compilation on JDK 23 and base/examples compilation with --release 11 passed. Four construction-only BenchYAML SIFT Small smoke runs passed: preload/buffered, each with the default parallel writer and explicit sequential override. These are functional checks, not additional 1M performance measurements. All recorded benchmark values above are unchanged.

[github-actions]

Before you submit for review:

• Does your PR follow guidelines from CONTRIBUTIONS.md?
• Did you summarize what this PR does clearly and concisely?
• Did you include performance data for changes which may be performance impacting?
• Did you include useful docs for any user-facing changes or features?
• Did you include useful javadocs for developer oriented changes, explaining new concepts or key changes?
• Did you rebase your branch onto the latest main for regression testing and PR submission?
• Did you trigger regression testing via Run Bench Main and review results?
• Did you adhere to the code formatting guidelines (TBD)
• Did you group your changes for easy review, providing meaningful descriptions for each commit?
• Did you ensure that all files contain the correct copyright header?
• Did you add documentation for this feature to the release notes directory?

If you did not complete any of these, then please explain below.

[MarkWolters]

Just the one comment on ParallelGraphWriter that I don't feel should be a blocker but I would like explained.

[MarkWolters]

this does fix an issue with the code already in place but now every future will run to completion even if one of them fails, then afterwards the exception will be thrown. Not sure if this is intended or not but the entire graph is invalid because of a failure, and there is a lot of work done that then gets thrown away

[tlwillke]

Agreed—this was a problem. Fixed: a failing worker immediately signals the other tasks to stop. Queued tasks skip their work, and running tasks stop before the next record. We still join every task before releasing resources and propagating the exception. Regression tests cover both failure and interruption, including a later task failing while an earlier task is blocked.