Store unchanged large binding values once in xmd repl histories #999

Story

As a person running xmd repl for hours, I want a large value captured once to occupy one stored payload while later entries carry it forward, so ordinary work does not repeatedly grow my history by that value's size.

For example, one entry reads a large build log:

<File path="build.log" as="log" />

Later entries bind small values or inspect other work:

<Let as="note" value="checking the failing shard" />

Those entries still inherit log as ordinary text. Recording their outcomes refers internally to its existing stored payload rather than writing the unchanged log again. The author uses the same components, bindings and expressions.

Current gap

#997 reduced retained copies and whole-file reads without changing journal encoding. Each entry's root outcome still records its complete eligible binding environment. A large unchanged value therefore appears again in later outcomes, increasing disk use and the size of records that must be serialized and parsed. With an accumulating set of large values, this repeated storage can grow quadratically with entry count.

The historical metadata reported in #992 measured a 536,365,110-byte journal; root close records accounted for 525,110,323 bytes, about 98%, with under 15 MB of distinct captured command output. That output figure does not bound all eligible root data, which also includes Let values, exports and structured captures. The separately generated synthetic corpus used for #997 measured 508,523,892 bytes. These are different histories. #997 establishes the remaining repeated-storage shape, not a measured benefit from the proposed change. The historical exit cause remains unknown.

Required outcome

• Retained references to the same unchanged large payload within one execution use one stored copy. Total storage includes the journal and every retained payload; moving duplicated bytes to another file does not count as deduplication.
• Storage supports internal URL references to payloads in another store; payloads are not required to live inside the journal file. The URL scheme, store implementation and local or remote deployment remain design choices.
• Opening a history and carrying unchanged bindings forward do not require loading every referenced payload. Fetch referenced data when a consumer needs its contents and stream it where the consumer supports streaming. Materialize an ordinary value when the consumer requires one. Expressions, props, output and captures preserve their value semantics; authors do not manage storage URLs or call a new fetch API.
• Every outcome still describes the root as it stood at settlement. A later entry receives its own mutable data; nested mutation publishes the changed value while earlier history remains unchanged. Authored failure, cancellation, eligible-value omission and the exclusion of props keep their existing semantics.
• An ordinary string resembling a storage URL, or ordinary JSON resembling a reference, remains ordinary user data.
• A history retains every payload needed to reconstruct its acknowledged records. A store holding the only copy is durable state and cannot evict it as disposable cache data.
• Resource safeguards distinguish total stored history from bytes fetched, buffered or materialized at once. This story introduces no new cap on total referenced payload volume. Specify any in-memory or per-operation bound against the allocation it protects, with an understandable refusal when reached. State how the existing journal-reading limit applies to the new representation rather than treating total store size as memory required to reopen.

Durability, integrity and disclosure

A recoverable journal record never refers to a payload that is not durable. The payload becomes durable before a referencing record can become recoverable, or both become recoverable through one atomic publication. Delaying the acknowledgement alone is insufficient: a crash must not leave a complete readable record pointing at absent data. A failed reference append or an unreferenced payload left by interruption does not make earlier acknowledged history unreadable. Settle orphan retention and cleanup policy before implementation without requiring automatic pruning.

Interruption, payload-write failure and journal-append failure expose no false completion; the existing first durability failure still fences later durable work.

The execution-owned secret policy examines the complete logical event, including actual payload values, before any new payload is written to durable storage or a referencing record is written. Externalizing a value does not reduce scanning to a digest, skip other event fields, or leave a newly written rejected credential-bearing payload in durable storage. Previously acknowledged history remains unchanged after refusal, including any payload it already owns. Preserve the trusted host's existing detection configuration. This story supplies no exemption from complete-event scanning and claims no scanning speedup.

Opening does not require eagerly reading and verifying every referenced payload. Journal validation and payload validation have separate timing. Before implementation, specify when referenced data is fetched and validated and which operations fail when a required payload is unavailable, malformed or mismatching. A payload failure never becomes an empty value or triggers the original effect again. Preserve the existing validation of journal records; amend the payload-reading contract explicitly rather than implying that all payload contents were checked during opening.

The documented reopening and movement procedure accounts for the journal, every referenced store and access to required payloads. A URL may continue to reference another store after the journal moves; moving the journal alone does not imply that it becomes self-contained. Required store dependencies are explicit rather than an undisclosed machine-local cache. Specify ownership and continued access before implementation; this does not itself introduce an entry export command or require a new archive format.

Product verification proposal

1. Capture large text and structured values once, then submit repeated entries that introduce no new large data. Compare total retained bytes and appended bytes with the same logical journey at the Reduce memory use when opening and continuing REPL histories #997 baseline. Generate the baseline history using the baseline build and the candidate history using the candidate's format. This measurement arrangement makes no claim about compatibility between those formats. Increasing entry count does not add another full copy of each unchanged payload. The baseline demonstrates the repetition this change removes.
2. Add genuinely distinct large values and show each remains recoverable. The result retains their actual content rather than discarding, truncating or replacing it with previews.
3. Change a nested member of an inherited value, settle, reopen cold and inspect positions before and after the change. Both show the correct values; executing a later entry cannot mutate retained history.
4. Reopen completed and unfinished histories, including authored failures and cancellation. Restore completed effects without repeating them and continue only the ordinary unrecorded work.
5. Exercise interruption before payload durability, after payload durability but before reference publication, and after publication but before caller acknowledgement. Include failed payload and reference writes. No recoverable reference lacks durable data, an orphan does not poison earlier acknowledged history, and later work respects the append fence.
6. Reopen a history with large referenced values that the initial view does not consume, then carry those bindings forward through an entry that does not use their contents. Instrument store reads and allocations to show neither journey fetches and materializes all payloads. Access one referenced value and witness its fetch or stream, validation and correct result without fetching unrelated values. Remove or alter a required payload in a controlled copy and request data that depends on it. Verify the specified validation timing and failure behavior, with no empty substitute or repetition of the original effect. Exercise literal URL-like strings and reference-like JSON as ordinary data.
7. Exercise enabled and explicitly disabled secret detection under the existing host contract. With detection enabled, a rejected complete event publishes neither its secret-bearing payload nor its reference.
8. Move the journal according to the documented procedure, preserving or reestablishing access to every required store, and reopen with the same values and historical results. Include a URL that continues to reference another store after the move. Measure cold-open, first payload access and append latency, retained heap and peak RSS separately against the same corpus and runtime. Report journal bytes and stored payload bytes separately.

These journeys are proposed evidence for refinement with the Product Owner. No storage savings, speedup or constant-memory result is claimed before measurement.

Architecture and scope

This changes retained storage, so its implementation requires an explicit amendment to the REPL storage and journal encoding contracts. The logical language-result contract stays unchanged when the retention layer resolves references before events reach Core; amend that boundary only if the chosen design changes Core's responsibilities.

Internal URLs may reference another store. Amend the existing one-physical-journal and no-sidecar sentences to permit this storage boundary. The design review selects the URL representation, store arrangement, ownership and movement procedure; it does not revisit whether another store is permitted. URL references do not inherently require a compatibility break. Identify the actual effect of the chosen encoding on existing histories and follow the repository's current persistence-format policy. This story assumes neither that existing histories become unreadable nor that a migration system is needed.

Before an implementation handoff, settle the retained unit and movement procedure, internal encoding and literal-data distinction, which payloads qualify, storage ownership and lifetime, payload fetch and validation timing and failure behavior, resource safeguards and the existing journal-reading limit, failure/publication ordering, and orphan treatment. Account explicitly for payload storage during Sidekick's existing writer redirect and storage handoff so the surviving destination retains the acknowledged journal and continued access to its required payloads. Keep production host behavior behind contextual adapters and owned resources. Fetching or streaming stored payloads when needed belongs to this story; changing expression evaluation semantics does not. Cross-execution deduplication, automatic pruning, delta or chunk deduplication and new streaming language APIs are separate decisions.

Relationships and evidence

Follows #992 and merged PR #997. The historical metadata and synthetic evidence there establish different baselines and must remain separately identified.

Preserve Sidekick's existing exact-byte prefix correlation and publication guarantees when amending the storage boundary. Coordinate changes with architecture.md (REPL composition and root-result handoff), specs/repl-spec.md, specs/executable-mdx-spec.md (computed root data and secret detection), and the durable-stream protocol specification. This story records the proposed outcome; it does not freeze a storage API or start implementation.