Skip to content

Implement native Tantivy backend and performance baseline - #21

Merged
kylebernhardy merged 13 commits into
mainfrom
codex/issue-16-native-backend
Sep 8, 2026
Merged

Implement native Tantivy backend and performance baseline#21
kylebernhardy merged 13 commits into
mainfrom
codex/issue-16-native-backend

Conversation

@kylebernhardy

@kylebernhardy kylebernhardy commented Sep 8, 2026

Copy link
Copy Markdown
Member

Closes #16.

Summary

  • adds the standalone @harperfast/fulltext/native entry point backed by Tantivy MmapDirectory
  • keeps the indexing and query engine generic over Tantivy Directory so the RocksDB adapter uses the same code path
  • adds versioned packed mutation/search protocols, one bounded writer actor per index, parallel search workers, stable errors, deterministic close, crash recovery, and backend identity validation
  • coordinates all indices owned by one Node environment through one bounded napi-rs cleanup hook
  • adds BM25 search with approximate totals by default and explicit exact counts
  • adds release-mode product-catalog benchmarks with commit cadence, durable throughput, queue/engine timing, same-concurrency exact-total comparison, search percentiles, index size, and sampled RSS
  • documents that Harper releases select only the RocksDB backend

Verification

  • npm run check
  • 36 Rust tests passed
  • 21 active Node/package tests passed; 3 rocksdb-js integration tests remain environment-gated
  • search-while-write, poison-close/reopen, abrupt-exit, single- and multi-index worker teardown, overload, identity, and package-artifact regressions covered
  • all GitHub checks pass on macOS, Windows, Ubuntu/Node 22.18, Ubuntu/Node 24, supply-chain/native linkage, and benchmark smoke
  • Claude full and delta reviews: all major findings resolved; final verdict COMMENTS
  • Gemini CLI was attempted directly but its configured API key was rejected, so no Gemini verdict was available

Local release benchmark

Apple arm64, Node 24.16.0, Tantivy 0.26.1:

  • 100k documents, commits every 25k: 93.2k durable docs/s; approximate p99 0.80 ms at concurrency 4; sampled peak RSS 268 MB
  • 1M documents, commits every 100k: 162.6k durable docs/s; approximate p99 6.20 ms at concurrency 8; exact-count p99 8.82 ms at concurrency 1; sampled peak RSS 684 MB; 30.4 MB index

These are development baselines, not the 100M-document or Harper p99 release qualification. Fixed-host release history and native-versus-Rocks comparisons remain in #15.

Comment generated by kAIle (Codex GPT-5)

@kylebernhardy
kylebernhardy merged commit 88f422b into main Sep 8, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement MmapDirectory mode as the standalone backend and reference oracle

1 participant