diff --git a/CHANGELOG.md b/CHANGELOG.md index 6176629..1d9c01a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,9 +10,13 @@ Every entry says what changed as a caller sees it, why, and what to watch for ### Added +- **`performance/projects/large-document/result-read-once.json`**: the same run on Memo 0.80.0 / Core 0.369.0 charges + 95,260 bytes, the document once (476,300 on 0.79.0). `after-fix.mjs` now pins those versions. + - **`performance/projects/large-document/after-fix.mjs`** reruns the stalled large-document reproduction on Memo 0.79.0 and Core 0.367.0 and asserts progress: the 95,200-byte document is found in one invocation and the run completes. The result also shows each matching lane reading the whole document again (476,300 charged bytes). + - A synthetic extraction-contract study records released SDK quote/schema limits and accepted semantic contradictions using fake completions, without treating structural validation as model quality. diff --git a/performance/projects/README.md b/performance/projects/README.md index 454508f..d8ea10e 100644 --- a/performance/projects/README.md +++ b/performance/projects/README.md @@ -9,7 +9,7 @@ - [Large documents](large-document/README.md): published Memo reproduction of stalled document lanes and repeated budget charges across two resumes, with a working small-document control; after the fix (Memo 0.79.0) - the same document is found in one invocation, at five times its size in charged bytes. + the same document is found in one invocation, and since Memo 0.80.0 read once (95,260 charged bytes, was 476,300). - [Process and metric definitions](PROCESS.md), [provenance ledger](provenance.json) and [offline verification](verify-provenance.mjs): immutable artifact references, hashes, release provenance diff --git a/performance/projects/large-document/README.md b/performance/projects/large-document/README.md index 5437321..d9bbd54 100644 --- a/performance/projects/large-document/README.md +++ b/performance/projects/large-document/README.md @@ -66,30 +66,35 @@ It does not claim complete project coverage, full history, semantic extraction o fit together, a record larger than the page is read alone up to 256 KiB, and one larger still is listed under `oversized` and skipped. The question: does the released Memo now read this same document and finish? -**Setup:** Node 24.19.0, exact registry Memo 0.79.0, Core 0.367.0 and Base 0.47.1, the same seeding, guard and +**Setup:** Node 24.19.0, exact registry Memo 0.79.0 / Core 0.367.0, then Memo 0.80.0 / Core 0.369.0, Base 0.47.1, the same seeding, guard and invented document as `run.mjs`. [`after-fix.mjs`](after-fix.mjs) asserts progress instead of the stall: ```sh -S=$(mktemp -d -p /var/tmp large-doc-sdk-XXXXXX) && HOME=$S XDG_CONFIG_HOME=$S/config XDG_DATA_HOME=$S/data XDG_CACHE_HOME=$S/cache MESSAGING_STORE=$S/store WIRECAT_STATE_DIR=$S/state TMPDIR=$S NPM_CONFIG_USERCONFIG=/dev/null npm install --prefix "$S/prefix" --ignore-scripts --no-audit --no-fund --save-exact @wirecat/cli-memo@0.79.0 @wirecat/cli-core@0.367.0 @wirecat/cli-base@0.47.1 +S=$(mktemp -d -p /var/tmp large-doc-sdk-XXXXXX) && HOME=$S XDG_CONFIG_HOME=$S/config XDG_DATA_HOME=$S/data XDG_CACHE_HOME=$S/cache MESSAGING_STORE=$S/store WIRECAT_STATE_DIR=$S/state TMPDIR=$S NPM_CONFIG_USERCONFIG=/dev/null npm install --prefix "$S/prefix" --ignore-scripts --no-audit --no-fund --save-exact @wirecat/cli-memo@0.80.0 @wirecat/cli-core@0.369.0 @wirecat/cli-base@0.47.1 T=$(mktemp -d -p /var/tmp large-doc-run-XXXXXX) && HOME=$T XDG_CONFIG_HOME=$T/config XDG_DATA_HOME=$T/data XDG_CACHE_HOME=$T/cache MESSAGING_STORE=$T/store WIRECAT_STATE_DIR=$T/state TMPDIR=$T node performance/projects/large-document/after-fix.mjs "$S/prefix" "$T/result.json" ``` **Results** ([`result-after-fix.json`](result-after-fix.json)): -| | Memo 0.76.0 (three invocations) | Memo 0.79.0 (one invocation) | -| --- | --- | --- | -| References found | 0 | 1 (the large document, matched by all five lanes) | -| Units | five `read-limited`, 0 pages | five `done`, 1 page each | -| Checkpoint / `complete` | present / false | none / true | -| Charged bytes | 983,040 | 476,300 | -| Reads | — | 10 | +| | Memo 0.76.0 (three invocations) | Memo 0.79.0 (one invocation) | Memo 0.80.0 (one invocation) | +| --- | --- | --- | --- | +| References found | 0 | 1 (the large document, matched by all five lanes) | 1 | +| Units | five `read-limited`, 0 pages | five `done`, 1 page each | five `done` | +| Checkpoint / `complete` | present / false | none / true | none / true | +| Charged bytes | 983,040 | 476,300 | 95,260 | +| Reads | — | 10 | 6 | + +The Memo 0.79.0 column is [`result-after-fix.json`](result-after-fix.json), from `after-fix.mjs` as of commit `aa40142` +(pinned to Memo 0.79.0 / Core 0.367.0). The 0.80.0 column is [`result-read-once.json`](result-read-once.json); the +script now pins Memo 0.80.0 / Core 0.369.0, so install those versions to rerun it. The small control still returns one reference for 184 bytes. **What it showed:** the stall is gone; the run ends in one invocation without a resume. It also showed waste: every lane reads the whole document again (two reads per lane: the refused page, then the record alone), and only -then drops it as already found. 476,300 bytes is five times the document. Skipping the read of a record the run -already holds would charge it once. +then drops it as already found. 476,300 bytes is five times the document. Core 0.369.0 +([cli-core#1676](https://github.com/WireCatLabs/cli-core/pull/1676)) skips that read when the run already holds the +record; on Memo 0.80.0 the same run charges 95,260 bytes, the document once. **Not measured:** timing, memory, a document above 256 KiB through the CLI (covered by Core tests), many medium documents on one page, or owner data. diff --git a/performance/projects/large-document/after-fix.mjs b/performance/projects/large-document/after-fix.mjs index 28ba23b..8176569 100644 --- a/performance/projects/large-document/after-fix.mjs +++ b/performance/projects/large-document/after-fix.mjs @@ -31,8 +31,8 @@ const artifact = (name, expected) => { return { version } } const artifacts = { - "cli-memo": artifact("cli-memo", "0.79.0"), - "cli-core": artifact("cli-core", "0.367.0"), + "cli-memo": artifact("cli-memo", "0.80.0"), + "cli-core": artifact("cli-core", "0.369.0"), "cli-base": artifact("cli-base", "0.47.1"), } const sourceText = "Copper Chronicle CCL Orchard review Planning digest Budget context. ".repeat(1400) diff --git a/performance/projects/large-document/result-read-once.json b/performance/projects/large-document/result-read-once.json new file mode 100644 index 0000000..323bb62 --- /dev/null +++ b/performance/projects/large-document/result-read-once.json @@ -0,0 +1,100 @@ +{ + "classification": "large-document-read-after-fix", + "node": "v24.19.0", + "artifacts": { + "cli-memo": { + "version": "0.80.0", + "integrity": "sha512-nMH+qjvdNOLN5PWjdekKlqAN5P9wLjLPDpgqCoslKiNvvc45UC2RhrbHGr39eFSrQUzlJg3Gk0gX301rFdvx6g==" + }, + "cli-core": { + "version": "0.369.0", + "integrity": "sha512-E0zC86Bp4XEiXXC2FMiy6cyTU2QkiZyhnlU+oEtt3UK2cjr0h/uvparDu2DqmWlDqZQmkJT10sJVz8DAzqKbig==" + }, + "cli-base": { + "version": "0.47.1", + "integrity": "sha512-v6hQV0tKr9FkJenqALpMG9b06H8ZZdDCxZPhpBLAZgr/kKyy1beJBQSCQ4V3jRhRAeM0RQFv1jUlytY/BM0qeA==" + } + }, + "source_bytes": 95200, + "page_budget_bytes": 65536, + "control": { + "items": 1, + "usage": { + "reads": 2, + "read_bytes": 184, + "around_bytes": 0, + "items": 1, + "elapsed_ms": 23 + } + }, + "attempts": [ + { + "invocation": 1, + "items": 1, + "lanes_matching_the_document": 5, + "matched_again": 0, + "returned_excerpt_bytes": 303, + "around_entries": 0, + "output_bytes": 3267, + "usage": { + "reads": 6, + "read_bytes": 95260, + "around_bytes": 0, + "items": 1, + "elapsed_ms": 25 + }, + "units": [ + { + "lane": "l1", + "source": 0, + "after": 1787308376825, + "before": 1792492376825, + "state": "done", + "pages": 1, + "issue": null + }, + { + "lane": "l2", + "source": 0, + "after": 1787308376825, + "before": 1792492376825, + "state": "done", + "pages": 1, + "issue": null + }, + { + "lane": "l3", + "source": 0, + "after": 1787308376825, + "before": 1792492376825, + "state": "done", + "pages": 1, + "issue": null + }, + { + "lane": "l4", + "source": 0, + "after": 1787308376825, + "before": 1792492376825, + "state": "done", + "pages": 1, + "issue": null + }, + { + "lane": "l5", + "source": 0, + "after": 1787308376825, + "before": 1792492376825, + "state": "done", + "pages": 1, + "issue": null + } + ], + "checkpoint_present": false, + "complete": true, + "stopped": null, + "oversized": 0 + } + ], + "limits": "Same synthetic fixture as run.mjs, one CLI invocation on the fixed release; no owner data, remote calls, timing benchmark, memory measurement or claim about documents above 256 KiB (listed under oversized, covered by Core tests)." +}