Skip to content

Add priority-aware host reclamation transport - #29

Merged
kylebernhardy merged 5 commits into
mainfrom
codex/issue-11-reclamation-lifecycle
Sep 10, 2026
Merged

Add priority-aware host reclamation transport#29
kylebernhardy merged 5 commits into
mainfrom
codex/issue-11-reclamation-lifecycle

Conversation

@kylebernhardy

Copy link
Copy Markdown
Member

Adds the first slice-6 lifecycle unit for the experimental Harper-backed Tantivy directory in
Implement Harper-backed Tantivy Directory mapping and atomic publication.
This remains test-only and does not expose the Harper storage backend in the release artifact.

Architecture

  • Adds one low-priority cleanup admission class to the existing per-index host transport; it shares
    the foreground callback channel while allowing at most one cleanup operation in flight.
  • Preserves foreground operation and byte headroom independently. Cleanup fails fast when current
    capacity is unavailable, while permanent configuration/request-size errors are classified before
    transient occupancy.
  • Constructs the cleanup KvDirectory from the foreground store, namespace, and ReclaimBudget in
    one operation. This preserves the exact storage identity and reader-pin state and prevents request
    bounds from being paired with a different namespace.
  • Keeps a timed-out response charged until its dispatched callback completes or the transport closes;
    admission and response timeouts remain distinct.
  • Treats N-API Closing as normal shutdown and fails the transport closed for other callback-queue
    inconsistencies.

For the human reviewer

Two deliberate contracts remain for the later production lifecycle unit:

  1. One shared FIFO host callback channel is used initially. The design requires a measured foreground
    p99 gate and moves cleanup to a separate callback channel if bounded work cannot remain within it.
  2. max_operations is sized as configured foreground concurrency plus one cleanup slot. The transport
    enforces one cleanup operation; demand beyond the configured foreground concurrency waits normally.

The dedicated cleanup task will bind and own its ReclaimBudget, expose health, and prevent use of
non-reclamation directory methods. Those concerns are intentionally outside this test-only transport
slice and must be complete before Harper enables the backend.

Verification

  • npm run check: formatting and TypeScript checks passed; Clippy passed with warnings denied; 108
    Rust tests, 42 active Node/native tests, and the packed consumer artifact test passed. Three existing
    opt-in rocksdb-js lease tests remained skipped because no feature-enabled checkout was configured.
  • npm run benchmark:smoke: passed against the unchanged public mmap backend (1,000 documents; warm
    approximate p99 0.229 ms in this run).
  • The host-transport integration test proves same-identity reader-pin visibility, exact request-bound
    sizing, physical deletion of both retired payloads, close wake-up, worker teardown, and conservative
    timeout accounting.
  • Independent Claude and direct Gemini reviews found and drove corrections to permanent/transient
    error classification, request-bound enforcement, namespace binding, shutdown taxonomy, and test
    overclaims. Both final convergence reviews found no material remaining defect; one Gemini diff-only
    clamp finding was rejected because the shared helper already applies the same clamp.

Complexity: complicated

Comment generated by kAIle (GPT-5)

Review-Coverage: authored=unknown; ran=none; rounds=1 @ f161f5e

Human-Review-Need: 4 @ f161f5e

@kylebernhardy kylebernhardy added this to the v5.3 milestone Sep 10, 2026
@kylebernhardy
kylebernhardy merged commit 449cb73 into main Sep 10, 2026
6 checks passed
@kylebernhardy
kylebernhardy deleted the codex/issue-11-reclamation-lifecycle branch September 10, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant