Use fixed chunks for Harper-backed Tantivy files

docs/phase-0-rocks-bridge-plan.md

@@ -63,9 +63,10 @@ Pinned `MmapDirectory` source also exposes a reference limitation: its path cach
 same older mmap to a new `open_read()` while a handle to that mmap remains alive, even after the
 writer flushes more bytes. The shared reference test therefore proves writer continuation, old-handle
 immutability, and later visibility after the old handle is released. The Rocks prototype separately
-proves simultaneous old and new binding revisions because its write-once fragments support that
-stronger behavior. The wrapper does not claim Mmap and Rocks have identical raw Directory cache
-semantics; it requires identical observable index, commit, reopen, and search behavior.
+proves simultaneous old and new binding revisions because its write-once chunks and tail revisions
+support that stronger behavior. The wrapper does not claim Mmap and Rocks have identical raw
+Directory cache semantics; it requires identical observable index, commit, reopen, and search
+behavior.
 
 ### rocksdb-js 2.8.0
 
@@ -153,32 +154,48 @@ and package-isolation cases still run.
 
 ## Logical object prototype
 
-Phase 0 uses the smallest mapping capable of exercising the Directory semantics. It is not yet the
-production format.
+The first Harper mapping uses fixed-size data chunks and a versioned tail. The chunk size is an
+internal format choice, not a schema or factory option. It remains subject to the benchmark sweep
+before the persisted format is declared stable.
 
 ```text
 namespace / index generation
-  working/<object-id>/fragment/<sequence>     -> immutable flushed bytes
-  object/<object-id>                          -> sealed ordered fragments and total length
-  binding/<logical-path>                      -> object-id and visible length
-  atomic/<logical-path>                       -> complete small-file bytes and revision
-  pending/<object-id>                         -> recovery marker
+  chunk/<object-id>/<ordinal>          -> immutable 256 KiB data chunk
+  tail/<object-id>/<revision>          -> immutable final partial chunk
+  binding/<logical-path>               -> v2 object-id, chunk count, tail revision and visible length
+  atomic/<logical-path>                -> complete small-file bytes
 ```
 
-`open_write()` creates a new object identity and a logical binding with visible length zero. Bytes
-may be buffered only until the next writer flush. A successful flush writes one or more new fragment
-keys and atomically replaces the binding with the complete ordered fragment list and visible length.
-Fragment keys are write-once: an append after flush starts a new fragment even when the preceding
-fragment is smaller than the target chunk size. This is the isolation source for an open handle; a
-handle captures one binding revision, object identity, fragment list, and visible length, so later
-flushes never change any key it may read. Termination flushes and seals the object. Deleting removes
-the binding; object bytes remain available while native handles retain them and become reclaimable
-afterward.
-
-`atomic_write()` stores the complete small value and replaces its logical revision in one RocksDB
-write batch. It is used for metadata such as `meta.json` and `.managed.json`, not large segment
-output. Phase 0 verifies the semantic boundary; issue #11 defines the production encoding,
-versioning, garbage collection, and bounded chunk policy.
+`open_write()` creates a new object identity and a zero-length binding. The writer stages each full
+chunk under its final ordinal with a WAL write while retaining at most one partial chunk. `flush()`
+atomically publishes a new binding and an immutable, revisioned tail. Filling a previously published
+tail creates a full chunk and a later binding revision; it never overwrites bytes visible to an
+existing file handle. Every flush that replaces a partial tail leaves its previous immutable tail
+revision unreachable. A file appended across K such flushes can therefore leave K-1 tails of up to
+`CHUNK_SIZE - 1` bytes, while a failed publication may additionally leave an unpublished tail or
+full chunks. These values remain until the derived-index reclaimer is implemented.
+
+A file handle captures one binding value. It computes a full-chunk key directly from the requested
+offset and reads the versioned tail only when the range intersects it. A range contained in one
+chunk therefore performs one payload lookup regardless of file size; ranges spanning boundaries
+perform one lookup per intersecting chunk. No read walks or fetches preceding payloads.
+
+A full-chunk read response requires at least `CHUNK_SIZE + 7` bytes for the protocol envelope, so
+the host store rejects a smaller response budget at construction. Transport admission separately
+accounts for the exact encoded request and reserved response before dispatch. The prototype does
+not claim a universal construction-time transport minimum: namespace and path lengths vary, and
+`atomic_write()` does not yet enforce a metadata size bound. The production factory must bound
+those inputs and validate its aggregate byte budget before this backend is exposed. The Phase 0
+sweep records small-read amplification and retained `OwnedBytes` because a slice keeps its complete
+chunk allocation alive; a bounded chunk cache is considered only if those measurements justify it.
+
+Termination flushes the remaining tail. Deleting removes the binding but leaves every immutable
+chunk and tail behind. They remain readable through already-open handles and become eligible for
+reclamation after those handles drain.
+
+`atomic_write()` stores the complete small value in one RocksDB write batch. It is used for metadata
+such as `meta.json` and `.managed.json`, not large segment output. Issue #11 still owns format
+qualification, bounded orphan reclamation and the chunk-size performance sweep before release.
 
 ## Experiment matrix
 

src/directory_harness.rs

@@ -212,6 +212,50 @@ where
 	verify_query_count(&reopened, body, "shoes", 1)
 }
 
+pub fn verify_large_file<D>(directory: D, chunk_size: usize) -> Result<(), String>
+where
+	D: Directory,
+{
+	let length = chunk_size
+		.checked_mul(2)
+		.and_then(|length| length.checked_add(17))
+		.ok_or_else(|| "large-file test length overflowed".to_owned())?;
+	let bytes = (0..length).map(|offset| (offset % 251) as u8).collect::<Vec<_>>();
+	let path = Path::new("large-file");
+	let mut writer = directory.open_write(path).map_err(|error| error.to_string())?;
+	writer.write_all(&bytes).map_err(|error| error.to_string())?;
+	writer.terminate().map_err(|error| error.to_string())?;
+
+	let file = directory.open_read(path).map_err(|error| error.to_string())?;
+	let full_chunk = file
+		.slice(0..chunk_size)
+		.read_bytes()
+		.map_err(|error| error.to_string())?;
+	if full_chunk.as_slice() != &bytes[..chunk_size] {
+		return Err("large-file full chunk returned unexpected bytes".to_owned());
+	}
+	let range_start = chunk_size
+		.checked_sub(13)
+		.ok_or_else(|| "large-file test chunk size is too small".to_owned())?;
+	let range = range_start..chunk_size + 19;
+	let crossing = file
+		.slice(range.clone())
+		.read_bytes()
+		.map_err(|error| error.to_string())?;
+	if crossing.as_slice() != &bytes[range] {
+		return Err("large-file boundary range returned unexpected bytes".to_owned());
+	}
+	let range = length - 30..length;
+	let tail_crossing = file
+		.slice(range.clone())
+		.read_bytes()
+		.map_err(|error| error.to_string())?;
+	if tail_crossing.as_slice() != &bytes[range] {
+		return Err("large-file tail boundary range returned unexpected bytes".to_owned());
+	}
+	Ok(())
+}
+
 fn verify_query_count(
 	index: &Index,
 	field: tantivy::schema::Field,
@@ -742,6 +786,11 @@ mod tests {
 		verify_tantivy_lifecycle(MmapDirectory::create_from_tempdir().unwrap()).unwrap();
 	}
 
+	#[test]
+	fn mmap_directory_supports_large_file_ranges() {
+		verify_large_file(MmapDirectory::create_from_tempdir().unwrap(), 256 * 1024).unwrap();
+	}
+
 	#[test]
 	fn harness_rejects_non_exclusive_locks() {
 		let directory = BrokenDirectory::new(true);

src/host_storage.rs

@@ -13,7 +13,7 @@ use napi_derive::napi;
 use tantivy::directory::OwnedBytes;
 
 use crate::boundary;
-use crate::phase0::{KvDirectory, KvStore, KvStoreIdentity, Mutation, WritePolicy};
+use crate::phase0::{KvDirectory, KvStore, KvStoreIdentity, Mutation, WritePolicy, CHUNK_SIZE};
 
 type HostCallback = ThreadsafeFunction<Vec<u8>, ErrorStrategy::Fatal>;
 type CompletionCallback = ThreadsafeFunction<Vec<u8>, ErrorStrategy::Fatal>;
@@ -365,6 +365,7 @@ const VALUE_MISSING: u8 = 0;
 const VALUE_PRESENT: u8 = 1;
 const MUTATION_PUT: u8 = 1;
 const MUTATION_DELETE: u8 = 2;
+const READ_RESPONSE_OVERHEAD: usize = 7;
 
 #[derive(Clone)]
 struct HostKvStore {
@@ -380,13 +381,20 @@ impl HostKvStore {
 		identity: KvStoreIdentity,
 		max_read_response_bytes: usize,
 		max_control_response_bytes: usize,
-	) -> Self {
-		Self {
+	) -> io::Result<Self> {
+		let minimum_read_response = CHUNK_SIZE + READ_RESPONSE_OVERHEAD;
+		if max_read_response_bytes < minimum_read_response {
+			return Err(io::Error::new(
+				io::ErrorKind::InvalidInput,
+				format!("host read response budget must be at least {minimum_read_response} bytes"),
+			));
+		}
+		Ok(Self {
 			transport,
 			identity,
 			max_read_response_bytes,
 			max_control_response_bytes,
-		}
+		})
 	}
 
 	fn request(&self, request: Vec<u8>, response_budget: usize) -> io::Result<ResponseDecoder> {
@@ -742,13 +750,20 @@ pub fn test_verify_tantivy_on_host_transport(
 						KvStoreIdentity(1, handle as u64, 1),
 						max_read_response_bytes as usize,
 						max_control_response_bytes as usize,
-					);
+					)?;
 					let run = NEXT_TRANSPORT_HANDLE.fetch_add(1, Ordering::Relaxed);
 					let case = AtomicU32::new(0);
 					crate::directory_harness::verify_directory_contract(|| {
 						let namespace = format!("host-contract/{run}/{}", case.fetch_add(1, Ordering::Relaxed));
 						KvDirectory::with_namespace(store.clone(), namespace.as_bytes())
 					})
+					.and_then(|_| {
+						let namespace = format!("host-large-file/{run}");
+						crate::directory_harness::verify_large_file(
+							KvDirectory::with_namespace(store.clone(), namespace.as_bytes()),
+							CHUNK_SIZE,
+						)
+					})
 					.and_then(|_| {
 						let namespace = format!("host-lifecycle/{run}");
 						crate::directory_harness::verify_tantivy_lifecycle(KvDirectory::with_namespace(

src/lib.rs

@@ -257,6 +257,12 @@ pub fn phase0_verify_tantivy_on_storage_lease(id: u32) -> boundary::Result<()> {
 			phase0::KvDirectory::with_namespace(lease.clone(), namespace.as_bytes())
 		})
 		.map_err(|error| napi::Error::new("E_STORAGE", error))?;
+		let namespace = format!("phase0-large-file/{run}");
+		directory_harness::verify_large_file(
+			phase0::KvDirectory::with_namespace(lease.clone(), namespace.as_bytes()),
+			phase0::CHUNK_SIZE,
+		)
+		.map_err(|error| napi::Error::new("E_STORAGE", error))?;
 		let namespace = format!("phase0-lifecycle/{run}");
 		directory_harness::verify_tantivy_lifecycle(phase0::KvDirectory::with_namespace(lease, namespace.as_bytes()))
 			.map_err(|error| napi::Error::new("E_STORAGE", error))