feat(vdb): add optional streaming ingest with LanceDB

[jioffe502]

Why

RayDataExecutor historically sends VDB ingestion through one global Ray batch. That preserves backend compatibility, but it creates a corpus-sized handoff before LanceDB can write.

What changed

• VDB has one optional, non-abstract stream_ingest() capability over a lazy iterable of canonical NRL record dictionaries.
• IngestVdbOperator owns canonical conversion. RayDataExecutor.ingest() owns Ray iteration, prefetch, cleanup, retained results, and downstream ordering; it has no LanceDB dependency.
• LanceDB owns bounded Arrow packing, one native table mutation, schema/index/optimization, validation, and process-shared serialization for embedded local stores.
• Default streaming stores no durable retry history. A caller-persisted stream_operation_id explicitly enables durable retry markers and reconstructed-backend recovery; ambiguous append outcomes fail closed.
• Backends without the capability retain the global-batch path. PutVdbOperator, service/in-process execution, and build_dataset() retain their historical behavior. Stream-only settings are rejected on legacy paths rather than ignored.
• Removed the experimental policy/report surface, redundant failure-test module, and research document.

Validation

• Qualified production source: baseline 6cdbc1daca379fccbd94fb46ad6ef963ca4cacb1; candidate 964db8e83ff5c19fdaafb4d7629690bb6941656b (tree 5168a398cccaf93ee9a7e1d714cf1b21fdbd632b). Final head a9a0c62057dbbc4945ac199d2be55969eed7e385 adds only a test-fixture compatibility fix; production code is unchanged.
• 327 focused graph, service, custom-VDB, and real-Lance regression tests passed.
• Real three-block Ray-to-LanceDB integration passed.
• Repository Black, Flake8, compile, pre-commit, and strict MkDocs checks passed.
• Source-frozen 12-dataset replay passed 24/24 baseline/candidate arms: 197,796 stored rows with exact content, schema, index, version, tag, and readiness audits.
• Counterbalanced full-ingest JP20 control passed all four baseline/candidate runs, including result and semantic audits: 1,940 pages, 3,201 stored rows, and 115 queries per run.

Full-ingest JP20 measurement	Baseline mean	Streaming mean	Change
Ingest time	103.34 s	87.66 s	-15.17%
Launcher wall time	134.98 s	120.09 s	-11.03%
Ray object-store peak	1.84 GB	1.15 GB	-37.19%
/dev/shm peak	2.24 GB	1.34 GB	-40.08%
Cgroup peak memory	65.00 GB	65.02 GB	+0.04%

Baseline ingest runs were 99.976-106.695 seconds; streaming runs were 85.514-89.796 seconds. The 12-dataset replay is a one-repeat sink correctness/resource check, not an end-to-end performance claim. Harnesses and artifacts remain outside NRL.

Relates to #2476 and #2570.

[jioffe502]

Refreshed this branch onto current upstream/main (bc9f2ce5, including #2566) and resolved the LanceDB index-wait overlap without dropping either side's behavior.

Post-merge validation:

• 53 real-Lance sink, recovery, concurrent-write, and IVF tests passed;
• 163 operator and Ray executor tests passed;
• the CPU-only real Ray 2.56.1 multi-block integration passed;
• merge commit hooks passed (Black and Flake8).

The 12-dataset measurements in the PR body remain the source-frozen 41ab2753 control/candidate run; I did not relabel or silently rerun that evidence after the merge.

Architectural scope is now explicit: this PR is LanceDB-first. The Arrow schema, RecordBatchReader write, retry markers, version/index validation, and optimization lifecycle remain Lance-owned. Decoupling terminal streaming capability detection from concrete LanceDB knowledge is tracked separately in #2570 and is not being folded into this PR.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no outstanding blocking or actionable non-blocking issue was identified.

Summary

• Adds a lazy canonical-record streaming capability to the VDB interface.
• Moves Ray batch iteration, cleanup, result retention, and ordering into RayDataExecutor.
• Implements bounded Arrow packing, process-shared locking, durable operation markers, retry validation, and finalization checks for LanceDB.
• Adds documentation and focused unit and integration coverage for streaming, recovery, compatibility, and failure behavior.
• The only change since the previous review updates a retrieval test fixture to represent readiness metadata expected by the new table checks.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Ray dataset blocks] --> B[RayDataExecutor.ingest]
    B --> C[IngestVdbOperator canonical conversion]
    C --> D{Backend supports streaming?}
    D -->|Yes: eligible local LanceDB| E[VDB.stream_ingest]
    E --> F[Bounded Arrow batches]
    F --> G[Serialized table mutation]
    G --> H[Validate indexes and finalization]
    D -->|No| I[Global batch VDB.run]
    H --> J[Retained ordered result]
    I --> J

Loading

_{Reviews (11) · Last reviewed commit: "test(vdb): model table readiness in hybr..."}

[jioffe502]

Follow-up on the fill compatibility regression: the repository lock (LanceDB 0.34.0) replaces the entire vector when any element is invalid, while the newer LanceDB exercised by current container CI preserves otherwise-valid elements. That made the public behavior dependency-version-sensitive.

The sink now owns and normalizes the historical/backward-compatible full-vector replacement policy before both create and append/put operations; the default drop policy is unchanged. The focused real-Lance 0.34 tests pass locally, and the complete current CI matrix—including Docker against its resolved dependency set—is green at fc8166ef. This keeps the application contract deterministic rather than branching on the installed LanceDB version.