Measure memory costs when opening large xmd repl histories #1000

Story

As a person reopening a long xmd repl session, I want the next memory improvement to target the work that actually causes opening pressure, so changes reduce the cost I experience without weakening history validation.

This evaluation produces a reproducible attribution of allocation and elapsed time during opening, plus a decision about whether a reader optimization is justified.

Current gap

#997 removed repeated retained-event copies and whole-file reads. Its compiled opening journeys and its forced-GC import probe measured different quantities in different processes. Subtracting retained heap from peak RSS does not identify an allocation stage or establish a leak.

The reader still constructs each complete record as text before parseDurableEvent parses it: record-file.ts joins decoded pieces in forEachLine, and journal.ts consumes the resulting line. Whether that transient dominates a real cold opening remains unmeasured. The #997 implementor reports that an attempted alternative which buffered bytes before decoding performed worse and was reverted; preserve and verify those original receipts before using them as evidence. If the receipts cannot be found or verified, report the account as unverified and do not use it to establish a performance result.

A new incremental JSON parser is therefore a candidate to evaluate, not an accepted implementation requirement.

Evaluation

Use deterministic synthetic histories and no real conversation contents. Include the #997 corpus shape and controls varying record size, record count, repeated payloads and distinct payload volume. Record those differences explicitly, including the largest single record's encoded byte length and decoded UTF-16 length; equal file size alone does not make two corpora equivalent.

Measure the production cold-open journey through a compiled binary at a pinned revision, with no new entry submissions. Separately attribute line assembly, parsing and validation, immutable retention/sealing, projection and entry preparation through diagnostic probes where needed. Explain overlapping stages and measurement limits rather than adding overlapping totals.

When refreshing measurements against #999's URL-referencing storage, report journal size and total referenced payload size separately. Measure cold opening, first access to a referenced value and repeated access as distinct journeys; record bytes actually fetched or streamed and any store caching state. Include a history whose referenced values are not consumed by the initial view, so a smaller opening footprint cannot conceal a deferred cost. An unused payload does not need a full read or verification just to open the journal.

Report process peak RSS, retained heap after any explicit GC intervention, transient allocation evidence, GC activity and elapsed time as distinct observations. Record source revision, binary identity, generator and corpus identity, runtime versions, platform, run count, terminal route and runtime options. Diagnostic probes do not substitute for the real compiled journey under default memory limits.

Repeat comparable runs with a small-history control and with different record-size distributions. Equality or an inconclusive difference is a valid result; it does not by itself invalidate the instrument. Distinguish elapsed-time improvements from memory improvements and describe observed variation without inventing a statistical threshold.

Decisions the evidence enables

• If complete-record text construction is a material cost, evaluate the smallest reader prototype that avoids that allocation. Compare it against the current reader using the same inputs and conditions. An additional parser or dependency needs a demonstrated benefit and an explicit review of its necessity.
• If another stage dominates, identify that production consumer and propose its smallest intervention instead.
• If no material cost is isolated or a prototype is slower or larger, record that result and recommend no change.
• If Store unchanged large binding values once in xmd repl histories #999 changes the storage representation, refresh the relevant measurement before pursuing an optimization whose premise was large inline records.

The outcome includes a recommended next action, the evidence supporting it, and any product or architecture decision it requires. This issue does not require an optimization to ship.

Correctness boundaries

For the #997 baseline and reader-only prototypes, preserve the existing journal validation and diagnostic behavior, exact byte fingerprints, decoding rules, supported journal-reading limits, refusal before effects or append, and cancellation that closes and joins owned readers. For measurements after #999, use its explicitly amended payload-fetch and validation contract; this evaluation adds no requirement to eagerly verify every referenced payload at opening and no new cap on total referenced store size. Instrumentation neither changes journal bytes nor becomes a shipped profiler, global cache or new document API.

Completion evidence

Publish the generator, corpus metadata, exact commands and raw measurements; identify which journey each proves. Include controls that exercise the instrument and a record-size comparison that tests the proposed attribution. Explain whether observed differences support an intervention, contradict it or remain inconclusive. Do not claim bounded memory, a leak or a speedup from the gap between unrelated heap and RSS measurements.

Relationships

Follows merged #997. Collect the relevant baseline at the #997 revision before #999 changes the storage representation. Reuse identical baseline receipts for #999's cold-open comparison rather than measuring the same journey twice. #999 owns removing repeated payload storage; this issue studies remaining opening costs and avoids freezing a competing reader design before that evidence exists. Source trace starts at packages/cli/src/repl/{record-file,journal,retained-event,model,session}.ts. No implementation plan is frozen by this evaluation proposal.