Defer inline feature serialization and batch complete graph records

README.md

@@ -46,6 +46,8 @@ Introductory tutorials for JVector are available in [docs/tutorials](./docs/tuto
 For file-backed input datasets, see [Batched input vector access](./docs/batched-vector-access.md),
 including runnable 100k-vector examples and optional integration with PQ and NVQ.
 
+For writing completed graphs with FP or NVQ features, see [Deferred batch serialization](./docs/deferred-batch-writer.md). It works with application vector sources and preloaded data; dataset prefetching is optional.
+
 The older step-by-step guide for JV can be found [here](./docs/legacy/jvector-step-by-step.md). New users should start with the tutorials mentioned earler, but the step-by-step guide contains useful commentary for advanced users.
 
 

benchmarks-jmh/README.md

@@ -133,7 +133,7 @@ The formatter will output a clean table showing:
 - **Visited_Count**: Average number of nodes visited during search
 - **Expanded_Count_BaseLayer**: Average number of nodes expanded in base layer
 
-Example output:
+Example output:[^machine]
 ```
  k  PQ_Subspaces   Time_ms  Recall  ReRanked_Count  Visited_Count  Expanded_Count_BaseLayer
 50             0  19.283   1.000             0.0        3290.8                     253.7
@@ -143,3 +143,5 @@ Example output:
 
 
 
+
+[^machine]: Machine type and core count were not recorded for this formatting example; its timings are illustrative, not a hardware baseline.

docs/benchmarking.md

@@ -115,3 +115,11 @@ To add a custom fvecs/ivecs dataset:
     ```
 
 For remote datasets, use `base_url` to specify where files should be downloaded from. The `${VAR}` and `${VAR:-default}` syntax is supported for environment variable expansion. See the example config for details.
+
+## Final index serialization
+
+BenchYAML and AutoBenchYAML default to parallel complete-record serialization
+(default batch size 64) after graph cleanup. `-Djvector.parallel_writer=false`
+selects the sequential path, which encodes FP/NVQ features during insertion.
+Dataset prefetching is a separate option and is not required. See
+[Deferred batch serialization](deferred-batch-writer.md) for activation and caller examples.

docs/deferred-batch-writer.md

@@ -0,0 +1,134 @@
+# Deferred batch serialization
+
+The parallel graph writer can serialize complete layer-zero records into bounded
+worker buffers. Each record contains its ordinal, inline features, neighbor count,
+and adjacency list. By default, a worker writes up to 64 complete records together.
+The final partial batch is written without adding records or padding to the format.
+This preserves the V6 layout and existing readers.
+
+## Production sources
+
+The library writer does not depend on Grid, BenchYAML, dataset preloading, or
+#744's buffered-access API. It accepts the existing
+`Map<FeatureId, IntFunction<Feature.State>>`: callers supply feature states for the
+requested original ordinals from their own source. That source can be resident
+vectors, a database's flush/compaction source, or another source; it must support
+concurrent supplier calls and remain available through final serialization.
+The writer does not require the original dataset to be fully resident.
+
+Complete-record batching also applies when all input vectors are preloaded. Its
+benefits come from deferring per-insertion feature writes and grouping final writes,
+independently of how input vectors are loaded. Buffered range access is an optional
+Grid integration, not a prerequisite for the core writer.
+
+## Caller migration: early writes versus deferred batches
+
+Use the parallel writer when deferring NVQ/FP encoding to final serialization.
+A sequential final writer performs that encoding on one thread and can regress
+construction time. Sequential callers should retain encoding during insertion
+and `writeFeaturesInline`.
+
+Both examples assume the same trained `nvq`, graph `builder`, node `count`,
+`writerWorkers`, and output path. The builder's construction scoring is unchanged;
+its peer vectors are supplied separately. `vectorForOrdinal` is the application's
+lookup of an original input vector, not a harness loader.
+
+### Old: write each inline feature during insertion
+
+```java
+try (var writer = new OnDiskParallelGraphIndexWriter.Builder(builder.getGraph(), outputPath)
+        .withParallelWorkerThreads(writerWorkers)
+        .withMapper(new OrdinalMapper.IdentityMapper(count - 1))
+        .with(new NVQ(nvq))
+        .build()) {
+    for (int ordinal = 0; ordinal < count; ordinal++) {
+        var vector = vectorForOrdinal.apply(ordinal);
+        builder.addGraphNode(ordinal, vector);
+        writer.writeFeaturesInline(ordinal, Feature.singleState(
+                FeatureId.NVQ_VECTORS, new NVQ.State(nvq.encode(vector))));
+    }
+    builder.cleanup();
+    writer.write(Map.of()); // Preserve the earlier feature writes; add adjacency.
+}
+```
+
+### New: supply the features when writing the completed graph
+
+```java
+for (int ordinal = 0; ordinal < count; ordinal++) {
+    builder.addGraphNode(ordinal, vectorForOrdinal.apply(ordinal));
+}
+builder.cleanup();
+
+try (var writer = new OnDiskParallelGraphIndexWriter.Builder(builder.getGraph(), outputPath)
+        .withParallelWorkerThreads(writerWorkers)
+        .withMapper(new OrdinalMapper.IdentityMapper(count - 1))
+        .with(new NVQ(nvq))
+        .build()) {
+    writer.write(Feature.singleStateFactory(FeatureId.NVQ_VECTORS,
+            ordinal -> new NVQ.State(nvq.encode(vectorForOrdinal.apply(ordinal)))));
+}
+```
+
+The second call supplies NVQ payloads during final serialization, together with
+adjacency, in batches of up to 64 complete records. No `writeFeaturesInline` pass
+is needed. The same pattern works for FP using `InlineVectors` and
+`InlineVectors.State`, or another supported inline feature. Supply every configured
+inline feature to obtain complete-record batching, including fused PQ if configured.
+
+Final suppliers run concurrently. `vectorForOrdinal` must remain available through
+`write` and provide independent borrowed vectors per worker, or stable vectors.
+For an existing `RandomAccessVectorValues`, `threadLocalSupplier()` supplies those
+independent views; it does not require buffered hooks. A caller can instead return
+already encoded feature states from its own storage, without retaining original FP
+vectors for this write.
+
+Integrations that already call `write` with every inline feature supplier receive
+batching when using the parallel writer. Integrations that write features early
+must move those calls to final suppliers to use the deferred path. The incremental
+API remains compatible: a one-pass source whose values cannot be obtained again
+must retain encoded states for final writing or continue using incremental writes.
+
+## Batch size and memory
+
+The default is 64 records. To override it, call `withParallelBatchSize(records)`
+before `with(...)` on either `OnDiskParallelGraphIndexWriter.Builder` or
+`RandomAccessOnDiskGraphIndexWriter.Builder`. The value must be positive. Each active
+worker holds one buffer of `recordSize * records` bytes; feature encoders and vector
+sources have their own memory costs. No dataset-sized output buffer is allocated.
+
+The unified `RandomAccessOnDiskGraphIndexWriter.Builder` retains its existing
+parallel/sequential selection. Setting the batch size does not enable parallel
+writing, and is ignored by the sequential writer. A caller-supplied executor
+remains caller-owned and must outlive the write.
+
+## Dataset prefetching in Grid
+
+When the parallel writer is selected, Grid defers its inline NVQ or FP feature serialization
+until after cleanup. Its final serialization suppliers consume worker-local ranges through
+`VectorAccess.openRange`. With the sequential writer, which calls suppliers on one
+thread, Grid keeps encoding these features on the insertion threads and writing them inline.
+Sources implementing the buffered-access capability use that implementation;
+ordinary sources retain the ordinary-access fallback. Borrowed vectors are consumed
+before advancing a cursor, and ranges are released after the workers finish.
+
+This integrates with #744's optional `useDatasetPrefetching` setting. It does not
+change construction scoring, quantization parameters, or the query path. Production
+integrations can opt into deferred serialization by supplying their features at the
+final write; their existing vector-source and incremental-write APIs remain valid.
+
+## Running the benchmark writer paths
+
+BenchYAML and AutoBenchYAML select the parallel writer by default. To use the
+sequential writer instead, add `-Djvector.parallel_writer=false` to the Java VM
+options. In that mode, Grid encodes FP/NVQ features during insertion and preserves
+the earlier inline-write path. No YAML setting is needed.
+
+Grid sizes its writer pool from `PhysicalCoreExecutor`; no machine-specific worker
+count is embedded in this change. Dataset prefetching is a separate optional setting
+in the index YAML. An ordinary preloaded dataset can use the same deferred writer.
+
+This default applies only to the benchmark entry points. The library's unified
+writer builder retains its existing sequential default, and application callers
+still explicitly choose parallel writing. Code calling Grid directly can select
+its writer through `GraphIndexBuilderConfig.getInstance().setParallelBuild(...)`.

docs/legacy/jvector-step-by-step.md

@@ -1,5 +1,9 @@
 ## JVector step-by-step
 
+> Historical guide: API names and early-write examples below reflect an older
+> version. Use the [current tutorials](../tutorials/2-disk-tutorial.md) and
+> [deferred serialization guide](../deferred-batch-writer.md) for current construction APIs.
+
 All code samples are from [SiftSmall](https://github.com/jbellis/jvector/blob/main/jvector-examples/src/main/java/io/github/jbellis/jvector/example/SiftSmall.java) in the JVector source repo, which includes the siftsmall dataset as well.  Just import the project in your IDE and click Run to try it out!
 
 #### Step 1: Build and query an index in memory

docs/release notes/4.0.2/747.performance.md

@@ -0,0 +1,36 @@
+### Deferred batch graph serialization
+
+Parallel graph serialization can now write complete layer-zero records in bounded
+batches of 64, with an optional positive batch-size override. When the parallel writer
+is selected, Grid supplies inline NVQ or full-precision features during final
+serialization, after graph cleanup, and uses buffered vector ranges when the input
+source supports them. With the sequential writer selected, Grid keeps encoding those
+features during insertion.
+
+The core writer accepts application-supplied feature states independently of the
+benchmark harness or dataset prefetching. Complete-record batching applies equally
+to fully preloaded inputs; the source does not have to preload the whole dataset.
+
+Integrators should use the parallel writer when deferring NVQ/FP encoding to
+final serialization. A sequential final writer performs that encoding on one
+thread and can regress construction time. Sequential callers should retain
+encoding during insertion and `writeFeaturesInline`.
+
+The existing V6 layout, readers, ordinary vector sources, and incremental
+`writeFeaturesInline` API remain supported. Prewritten features are preserved.
+Concurrent feature suppliers must keep their input source available until writing
+finishes. BenchYAML and AutoBenchYAML default to parallel writing, with
+`-Djvector.parallel_writer=false` selecting the sequential path. The library
+writer default remains unchanged.
+
+Recorded construction studies with dataset prefetching reduced ample-memory final
+write time from 136.10 to 50.69 seconds on MPNet 10M, 148.50 to 64.02 seconds on
+Cohere 10M, and 162.64 to 51.76 seconds on CAP 6M.[^machine] These are separate fresh runs,
+not a universal throughput or memory guarantee. See
+[Deferred batch serialization](../../deferred-batch-writer.md) for usage.
+
+Construction tutorials and examples now demonstrate final feature suppliers for
+deferred FP/NVQ writing, with per-worker source views and an explicit source-lifetime
+requirement. The query algorithms and quantization settings are unchanged.
+
+[^machine]: GCE VM, 48-core Intel Xeon 6985P-C (96 threads), local SSD RAID10, JDK 23, 128 GiB heap; native Highway, 48 workers.

docs/tutorials/2-disk-tutorial.md

@@ -8,7 +8,7 @@ Unlike the previous example, this time we'll use a proper ANN dataset based on O
 // This is a preconfigured dataset that will be downloaded automatically.
 DataSet dataset = DataSets.loadDataSet("ada002-100k").orElseThrow(() ->
     new RuntimeException("Dataset doesn't exist or wasn't configured correctly")
-);
+).getDataSet();
 ```
 
 > [!TIP]
@@ -45,7 +45,8 @@ Path graphPath = Files.createTempFile("jvector-example-graph", null);  // or whe
 
 // Create a writer for the on-heap graph we just built.
 // Remember to close when done.
-GraphIndexWriter writer = GraphIndexWriter.getBuilderFor(GraphIndexWriterTypes.RANDOM_ACCESS_PARALLEL, heapGraph, graphPath)
+GraphIndexWriter writer = new OnDiskParallelGraphIndexWriter.Builder(heapGraph, graphPath)
+    .withParallelWorkerThreads(PhysicalCoreExecutor.getPhysicalCoreCount())
     // Let the writer know that we'll also be passing in the actual vector data
     // to be saved "inline" with the data for each corresponding graph node.
     .with(new InlineVectors(dim))
@@ -60,13 +61,20 @@ Since we want to include a feature, we need to tell the writer how to obtain the
 // Supply one map entry for each feature.
 // The key is a FeatureId enum corresponding to the feature
 // and the value is a function which generates the feature state for each graph node.
+var writerVectors = ravv.threadLocalSupplier();
 writer.write(Map.of(
     FeatureId.INLINE_VECTORS,
     // we already have a RAVV, so we'll just use that to supply the writer.
-    nodeId -> new InlineVectors.State(ravv.getVector(nodeId))));
+    nodeId -> new InlineVectors.State(writerVectors.get().getVector(nodeId))));
 // writer.close() if not using try-with-resources
 ```
 
+Supplying every inline feature lets the parallel writer serialize complete records
+in batches of up to 64. The source must remain readable until `write` finishes.
+`threadLocalSupplier()` provides independent views when a source borrows a scratch
+vector; it also works with ordinary preloaded sources. See
+[Deferred batch serialization](../deferred-batch-writer.md) for NVQ and migration examples.
+
 At this point the graph index has been written to disk and can be used by creating an instance of `OnDiskGraphIndex`.
 
 To do so, we'll need to specify a `RandomAccessReader` implementation which JVector will use to read parts of the file as needed. This interface isn't thread safe, so when creating an `OnDiskGraphIndex` we pass in a `ReaderSupplier` object which can be used by the `OnDiskGraphIndex` to create `RandomAccessReader`s as needed.