Add optional bounded file-backed vector loading and batched access

[tlwillke]

This PR supersedes the dataset-loading approaches proposed in #737 and #721.

Purpose of this PR

Enable quantizer-independent, bounded access to input vector files without requiring the original base dataset to be fully resident in memory.
Buffered prefetching supports:

• Training: retrieve samples or scan vectors for statistics; release temporary training input afterward.
• Encoding: quantize vectors in batches for resident graph construction or on-disk reranking storage.
• Serialization: stream input vectors through encoding, where needed, into the existing on-disk format for reranking, including FP and NVQ.

It is not intended for repeated raw-vector comparisons during full-precision graph construction. Encoded vectors, trained models and graph structures retain their existing memory requirements.

Bounded file-backed input vector loading

This PR allows quantization and construction to consume file-backed vectors without preloading the complete base dataset into Java heap.

• BatchedVectorValues optionally supports ranges and arbitrary selections.
• VectorAccess uses batching when available and preserves ordinary RandomAccessVectorValues fallback.
• PQ sampling and learning policy remain unchanged.
• BenchYAML and AutoBenchYAML preserve preload by default. Set useDatasetPrefetching: true to opt into buffered input where supported. Queries and ground truth remain resident.
• Existing application-supplied vector sources remain compatible.

I/O workers follow JVector’s compute sizing. The shared input payload budget defaults to at most 1% of file size or 64 MiB, with one record as the minimum. Reads use ordinary positional file I/O and the OS page cache.

The loader supports immutable, seekable files on local or network-attached filesystems. Storage performance affects throughput. Buffering is bounded, but training samples, encoded vectors, graph state and JVM overhead still require memory.

No machine-specific heap limits, cgroup configuration or IntelliJ launch settings are included in this PR.

The flag is optional and defaults to false. Full-precision construction and
raw-input comparison workloads retain resident input. Grid uses resident quantized
vectors for construction peer scoring; each original insertion vector is supplied
from a bounded input batch. Query algorithms and scorer semantics remain unchanged.
AutoBenchYAML's existing exact compaction validation retains its resident input.

Benchmark results

The following tables contain only the three large datasets requested for final
validation. Times are seconds; memory is GiB. Measurements are copied from the
completed raw evidence without changing phase definitions or substituting estimates.

Ample memory: latest main versus this PR, both using main's existing
V6 writer. Limited memory: This PR plus the deferred complete-record batch writer, a separate follow-up PR (#747). That writer is not included in #744.
It retains V6 and batches complete records containing NVQ, FP or other fixed-size
feature payloads. The limited-memory timings shown require that additional writer
change; standalone #744 also completed under these limits with higher writing cost.

The follow-up writer implementation is on the deferred-batch-writer branch.
Its PR number will be linked here when opened.

Memory configurations: ample = 128 GiB maximum Java heap, uncapped total
memory on the approximately 354 GiB host, zero swap; limited = 32 GiB total cgroup memory,
20 GiB maximum heap, zero swap. These are experimental settings, not library defaults.
All input datasets exceed the limited heap; Cohere and CAP also exceed the total cap.

Both configurations use JDK23, native Highway automatic avx3_spr dispatch,
preferred512, 48 insertion/cleanup workers, DOT_PRODUCT, PQ width8/K256/no centering/
anisotropic−1, NVQ2, M32/efConstruction100/overflow1.2, hierarchy and refinement
enabled, fused PQ, pruning disabled. Input files started cold, with verified
file-specific eviction. Every successful run trained fresh independent codebooks
and built a fresh graph. No query benchmark is included in this construction study.

Main source: 056d93c2; #744 source: 1bab1881 (the subsequent 0b7c24e5 change
only fixes a documentation link); combined writer source: 9194992f.

The table values remain the measurements of those frozen sources. Subsequent audit
commits 7b5d8c7e and b5761f33 fix JDK test compatibility, bound scan ordinal metadata and
demand-read progress, correct mapped-input diagnostic validation, and make test subprocess logs UTF-8
across platforms; no table
values were changed or extrapolated to the later commits.

OOM† is the recorded preload failure from earlier tests with the same
32 GiB/20 GiB/zero-swap limits, before graph construction or writer execution.
Those failures were not rerun in the final study. A dash means no completed-run
metric is available. The reported ample MPNet pair is the final counterbalanced
repeat; prior runs remain archived. Independent training and parallel graph
topology introduce variability, so these tables do not establish universal speedups.

Total process time summary

Dataset	Main (ample)	This PR (ample)	This PR + new writer (limited)
Cohere 10M	764.85 s	633.04 s	531.77 s
MPNet 10M	566.90 s	541.15 s	391.17 s
CAP 6M	955.61 s	478.80 s	335.15 s

Cohere 10M

Base input: 10,000,000 vectors × 1,024 dimensions; 38.184 GiB fvecs, including record headers. Memory limits are specified above.

Metric	Ample: main preload	Ample: #744	Limited: preload†	Limited: #744 + deferred writer
Outcome	Completed	Completed	OOM†	Completed
Preparation / upfront preload s	34.467	0.187	—	0.178
PQ training s	9.147	8.131	—	8.731
PQ encoding s	41.414	37.672	—	39.147
NVQ setup s	5.088	4.665	—	14.726
Graph insertion s	458.655	403.187	—	378.654
Graph cleanup s	34.273	29.524	—	30.290
Final writing s	179.072	148.495	—	58.366
Pipeline s	727.649	631.674	—	529.914
Total process s	764.852	633.042	—	531.767
Peak heap, sampled GiB	76.396	15.377	—	13.687
Peak buffer pool, sampled GiB	49.591	49.653	—	49.668
Peak process RSS GiB	85.116	17.068	—	15.193
Cumulative GC reclaimed GiB	644.595	600.377	—	633.267

MPNet 10M

Base input: 9,997,697 vectors × 768 dimensions; 28.641 GiB fvecs, including record headers. Memory limits are specified above.

Metric	Ample: main preload	Ample: #744	Limited: preload†	Limited: #744 + deferred writer
Outcome	Completed	Completed	OOM†	Completed
Preparation / upfront preload s	26.962	0.169	—	0.178
PQ training s	5.886	7.516	—	5.683
PQ encoding s	33.505	35.267	—	29.254
NVQ setup s	3.605	3.885	—	5.595
Graph insertion s	335.704	336.670	—	281.590
Graph cleanup s	21.755	20.541	—	22.293
Final writing s	137.574	136.098	—	44.702
Pipeline s	538.029	539.978	—	389.116
Total process s	566.896	541.146	—	391.170
Peak heap, sampled GiB	62.429	13.740	—	17.771
Peak buffer pool, sampled GiB	37.651	37.714	—	37.725
Peak process RSS GiB	67.766	15.331	—	20.702
Cumulative GC reclaimed GiB	589.510	656.837	—	544.245

CAP 6M

Base input: 6,070,488 vectors × 1,536 dimensions; 34.758 GiB fvecs, including record headers. Memory limits are specified above.

Metric	Ample: main preload	Ample: #744	Limited: preload†	Limited: #744 + deferred writer
Outcome	Completed	Completed	OOM†	Completed
Preparation / upfront preload s	32.218	0.200	—	0.191
PQ training s	10.810	9.901	—	11.531
PQ encoding s	39.712	35.314	—	35.111
NVQ setup s	4.792	4.139	—	11.653
Graph insertion s	717.113	244.676	—	202.612
Graph cleanup s	23.106	20.751	—	19.371
Final writing s	124.744	162.644	—	53.338
Pipeline s	920.277	477.424	—	333.616
Total process s	955.611	478.798	—	335.151
Peak heap, sampled GiB	64.152	14.276	—	9.592
Peak buffer pool, sampled GiB	44.590	44.653	—	44.675
Peak process RSS GiB	68.975	16.553	—	10.772
Cumulative GC reclaimed GiB	441.479	409.741	—	340.649

Preparation includes upfront base-vector loading for main; #744 opens the file
and reads vectors during the pipeline. Avoiding preload explains essentially all
of the ample MPNet total-time saving (its pipeline is 1.949 s slower). Cohere and
CAP also show faster pipelines, principally insertion; these measurements do not
isolate the cause of those additional gains, especially CAP's large difference.

Pipeline is the sum of training, encoding, NVQ setup, insertion, cleanup and
writing. Process time also includes preparation, JVM startup, logging and cache
bookkeeping. Deferred NVQ encoding occurs during final writing rather than insertion.

RSS is a separate process resident peak, not heap plus buffer pool. Buffer pool
includes mapped address ranges and is not resident memory; it excludes other native
memory. GC reclaimed is cumulative. The constrained cgroup limit includes charged
file cache and kernel memory; all three combined runs completed without OOM or swap.

Reproduction

1. Download the selected public dataset completely before timing. Use immutable
   fvecs files, NO_SCRUB metadata and the construction settings listed above.
2. For the ample comparison, build main and Add optional bounded file-backed vector loading and batched access #744 separately with the same native
   backend, V6 writer, storage placement, concurrency and resource limits. Enable
   useDatasetPrefetching: true only in Add optional bounded file-backed vector loading and batched access #744's index-parameter YAML.
3. For the limited buffered measurement, build the separate deferred-writer
   integration branch, retain V6, enable its existing parallel writer with 48
   workers and the default 64-record batches, and apply the recorded 32 GiB cgroup /
   20 GiB heap / zero-swap limits. The ordinary Add optional bounded file-backed vector loading and batched access #744 branch does not contain this writer.
4. Use isolated working directories and empty index/PQ caches for every arm. Verify
   file-specific cold input cache before each run; allow natural warming during it.
   Preserve failures and outliers. Do not overlap benchmark workloads.
5. Run BenchYAML/Grid construction, reporting preparation and each pipeline phase
   separately. Record total child-process time and peak RSS; sample heap/buffer pool
   at 100 ms; collect GC logs and in-run cgroup high-water under the limited protocol.
   The recorded construction-only study used external phase markers and a return
   before queries; those measurement adapters are not committed library features.

Choose heap and worker counts for the machine; no machine-specific limits are
committed. For a standard BenchYAML invocation, use caller-supplied runtime values:

java --add-modules=jdk.incubator.vector --enable-native-access=ALL-UNNAMED \
  -Xmx"$BENCH_HEAP" \
  -Djvector.experimental.enable_native_vectorization=true \
  -Djvector.vectorization_provider=native \
  -Djvector.physical_core_count="$BENCH_WORKERS" \
  -Djava.util.concurrent.ForkJoinPool.common.parallelism="$BENCH_WORKERS" \
  -cp "$BENCH_CLASSPATH" \
  io.github.jbellis.jvector.example.BenchYAML benchmark.yml

The existing -Djvector.dataset_loader=preload|buffered remains an explicit override.
Full-precision construction and raw-input exact comparisons keep their resident
source. Production applications can continue supplying ordinary RAVVs without
implementing the optional bulk hooks.

Documentation and validation

docs/batched-vector-access.md
covers public-dataset hello-world examples, arbitrary selections, sequential scans,
borrowed-vector ownership, resource limits, fallback behavior and the benchmark opt-in.
The required 4.0.2 release note
is included.

The post-audit focused suite completed 127 tests on Java 11 and 127 tests
on Java 23 with native Highway enabled, with zero failures. The FFM cold-cache test
is skipped on Java 11. Coverage includes the AutoBenchYAML configuration-error
regression, a 1,000,003-ordinal bounded mapped scan, array-free PQ identity scans,
prefix/tail encoding parity, mapped ordering/duplicates with debug validation,
and demand-read zero-progress/EOF handling. Added coverage exercises concurrent reads,
buffer bounds, cancellation/closure, zero-progress reads, ordinary-source fallback,
sampling order/duplicates, quantizer encoding parity, and borrowed-vector ownership.
The quantization algorithms, distance metrics, stored V6 layout and query semantics
are preserved. The deferred-writer follow-up has its own serialization validation.

[github-actions]

Before you submit for review:

• Does your PR follow guidelines from CONTRIBUTIONS.md?
• Did you summarize what this PR does clearly and concisely?
• Did you include performance data for changes which may be performance impacting?
• Did you include useful docs for any user-facing changes or features?
• Did you include useful javadocs for developer oriented changes, explaining new concepts or key changes?
• Did you rebase your branch onto the latest main for regression testing and PR submission?
• Did you trigger regression testing via Run Bench Main and review results?
• Did you adhere to the code formatting guidelines (TBD)
• Did you group your changes for easy review, providing meaningful descriptions for each commit?
• Did you ensure that all files contain the correct copyright header?
• Did you add documentation for this feature to the release notes directory?

If you did not complete any of these, then please explain below.

[MarkWolters]

Left some comments requesting changes.

Full regression run in progress, will add results when it finishes.

[MarkWolters]

pr744.tar.gz
Full regression test - prior to commit 0544989

[MarkWolters]

Looks good, thanks for addressing my comments.

[jshook]

Concerns addressed on sub-branch. LGTM!

[jshook]

Reapproving after merge fixes