Support concurrent native indexes with bounded shared resources

[kylebernhardy]

Outcome

Allow multiple native Tantivy indexes to ingest and search concurrently while protecting the hosting process.

Acceptance criteria

• One writer per canonical physical directory/generation; independent index writers may progress in parallel.
• Bound process-wide resident writers, indexing/merge threads, writer memory, queued bytes, searches, mappings and rebuild generations.
• Measure engine-owned threads and memory; per-index limits or bounded command queues alone do not establish aggregate limits.
• Preserve fair query, recovery and shutdown progress under sustained ingestion and merge load.
• Qualify worker attachments and path aliases, owner loss, close/reopen, and generation retirement with no live-handle reuse.
• Measure actual physical/virtual memory and shared file pages rather than assuming each Node worker duplicates index contents.
• Measure achieved ingestion and lag/backlog drain together; account for merge/rebuild disk headroom.
• Harper controls source admission and disk policy. No host-storage wait pool, Rocks transaction lane or wrapper lease service is introduced.

Design

Native Tantivy storage and Harper derived indexes. Native Tantivy files are the only delivery target for standalone use and Harper. Proposed APIs and integration gates are not claims of completed implementation.

Comment generated by kAIle (GPT-6)