[P2] Define safe BSON-size handling across commits, snapshots and recycling #100

Summary

Define and test BSON-size handling across synchronous/asynchronous commits, snapshot upserts and stream recycling. This follows the size concern found while reviewing #97 for #87; the broader lifecycle redesign remains in #96.

MongoDB has a 16 MiB (16,777,216-byte) BSON document limit. An ordinary over-limit insert is already rejected by the driver/server. The adapter work is to make rejection understandable and state-safe, not to bypass that limit or promise arbitrary-size documents without a storage redesign.

Priority: P2 (correctness and compatibility).

Evidence and scope

At #97 head 8c479293:

• Commit and CommitAsync allocate a checkpoint before serialization/insertion and schedule head updates after insertion.
• AddSnapshot and its async counterpart upsert the payload before changing snapshot/head metadata, catch exceptions and return false. Snapshot/head writes are unacknowledged by default, so return values alone cannot establish server-side acceptance.
• DeleteStream and DeleteStreamAsync in Fix recycle-bin identity collisions and metadata cleanup ordering (#87) #97 replace the bucket with :rb, replace the stored stream ID with a 32-character hexadecimal GUID, and add OriginalBucketId/OriginalStreamId. For ordinary documents with the expected string fields and neither provenance field already present, the resulting BSON grows by exactly 81 bytes, independent of the original ID lengths.

The last point is a distinct regression risk: a valid, successfully stored commit in the final 81 bytes below the document limit can become too large when recycled. The source and byte accounting have been checked; a live MongoDB boundary reproduction has not yet been run. This issue does not claim that every write path has a demonstrated defect.

Acceptance criteria

• Add byte-accurate boundary reproductions using the fully serialized resulting BSON, including commit envelope, headers, all events, snapshot identity/payload, UTF-8 identifiers, custom document serializers and BSON mappings. Cover below/at/above the limit, plus commits that fit when inserted but exceed it after recycling. Record driver/server versions and test sync/async paths on supported configurations.
• Define consistent, actionable size-failure semantics for each sync/async API pair. Preserve underlying diagnostics without dumping event/snapshot payloads. Explicitly decide how this fits the existing snapshot false contract and acknowledged versus unacknowledged writes; document any compatibility change.
• Verify rejected operations do not cause unintended changes to commits, stream heads, snapshot revision/unsnapshotted counts, existing snapshots or checkpoint state. Where allocation side effects or write-concern behavior cannot be avoided, document the exact contract and recovery implications. Cover built-in/custom checkpoint generators and background metadata work.
• Test mixed-size, multi-commit streams where recycling updates some documents before encountering an oversized result. Assert live/recycled visibility, retained metadata, retry/recovery behavior and isolation from other streams/buckets. Do not claim UpdateMany is atomic or that retry alone fixes a permanently oversized document.
• Decide separately how to protect future writes and how operators must handle already persisted near-limit commits. Reserving headroom at insertion cannot repair existing immutable event data. Provide a deployment audit and explicit, non-destructive recovery policy, coordinated with [P2] Technical debt: define safe stream lifecycle and recycle-bin redesign #96.
• Evaluate validation/preflight, driver/server error handling and representation choices without prescribing one implementation. Measure additional serialization/allocation, database reads and write-path latency if preflight or headroom checks are proposed.
• Add regression tests and user/operator documentation, including changelog and breaking-change/migration guidance for any reduced accepted size, changed error contract or persisted-format change.

No silent truncation, dropped events or fallback hard deletion. Splitting/offloading payloads or changing the recycle storage model requires an explicit design, not an implicit workaround in this issue.

Co-authored by Alessandro Giorgetti and Codex (OpenAI).