Repository navigation
Add opt-in recorded outputs and validate output manifests - #356
Merged
Merged
Conversation
luisleo526
force-pushed
the
sm/e1-post354
branch
from
October 8, 2026 06:13
1581db4 to
cd3384f
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| assert lib.names().count("report_free") == lib.names().count("strategy_free") == 1 | ||
| off, on = documents | ||
| on_provenance = json.loads(json.dumps(on["fingerprint"]["provenance"])) | ||
| assert on_provenance["runtime"].pop("outputs") is True |
| assert provenance["strategy_resolution"]["initial_capital"]["reason"] == "foreign_unverified_source" | ||
| assert lib.names().count("report_free") == lib.names().count("strategy_free") == 1 | ||
| off, on = documents | ||
| assert on["runtime"].pop("outputs") is True |
…-outputs A module can record what it computes on each bar besides its trades: doubles in named slots per bar, doubles kept once per run, and events with an optional message, one row per bar the module publishes. - BacktestEngine gains the recorder (engine.hpp tail block, engine_report.cpp): declare_outputs, output_run_begin, output_bar, output_value, output_event, output_constant and the outputs_enabled_ flag for the host, readers for the C side. The host states the shape, each run's boundary and each row's identity (its open time); a row restarts at an equal open and a lower one fails the run. State sits outside both hashed regions; nothing a run computes reads it. - pineforge.h gains group pf_outputs: nine runtime exports (strategy_outputs_set_enabled and the readers, the size-prefixed pf_output_event_v1_t) and three per-module declarations (strategy_outputs_api_version, strategy_outputs_manifest, strategy_signal_safety_receipt). PF_ABI_VERSION stays 4; pf_report_t and every existing symbol are unchanged. - native_c_api.h lists the six writers as C++-only base-class seams at zero line shift; check_c_abi_runtime.py counts 73 runtime implementations and 93 declarations. - docker/run_json.py --outputs writes the record as the report's "outputs" block. - Tests: test_outputs_recorder (hand values, every refusal), test_outputs_stream_equivalence (stream == batch, the stale carried callback, negative controls), test_outputs_truncation (a cut batch records the prefix, negative control), test_outputs_run_reuse, test_outputs_off_identity (trades, equity and hashes equal with recording undeclared, off and on), test_outputs_c_api_c99, scripts/test_run_json_outputs.py and scripts/test_report_outputs_keys.py; the ci_verify row floors rise by those rows. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs/outputs.md describes the record (rows, values, events, run constants, colours as an encoding), the C++ host API and its row rules, the C readers, batches and streams, run_json --outputs, and the costs. ADR-0001 gains the recorder's codegen-ABI row and the C-surface row; the report-schema, abi-stability and docker pages describe the outputs block, the new exports and the epoch ruling; the stated PF_API counts (73 runtime, 93 in pineforge.h, 136 across both headers) follow the tree; CHANGELOG gains the Unreleased entry. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ounts check_native_c_api_surface.py requires every C enumeration to be twinned to its kernel enumeration or ruled C-only: pf_output_phase_t mirrors NativeRunPhase value for value (src/c_abi.cpp asserts the same), so it joins pf_native_run_phase_e as a twin. test_ci_verify.py pins the check_c_abi_runtime.py counts (93 declarations, 73 implementations), and the two remaining pages that state them follow the tree. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s run constant Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the stable run-failure codes (#351): - strategy_outputs_set_enabled records its refusals through the failure record, code outputs_rejected (reason not_declared or run_in_progress), and clears it when it succeeds, so strategy_get_last_error_code reads "" after a switch that worked. - The recorder's refusals carry their codes: a broken precondition is engine_invariant (still a std::logic_error), the event queue or the rows reaching INT_MAX are pine_runtime_limit (limit output_events or output_rows; the row cap is new and keeps bar_index an int32_t), a bar recalculated after a caller cleared its events is outputs_rejected (reason recalculated_after_clear). - docker/run_failure_codes.json gains outputs_rejected, the two limits, strategy_library_incompatible's reasons outputs_api_mismatch and outputs_manifest_invalid, and the recorder's texts; the registry, the catalog diff and the code tables of run-failure-codes.md are regenerated. - run_json: OutputsError is a RunFailure, written as the coded failure line. --outputs requires the manifest and every reader, checks the outputs API version, refuses a manifest that is not a JSON object, names a slot the library does not record or lacks an output an event names, and copies each event's message when it reads the event. Tests: test_outputs_recorder checks each refusal's code and the getters, and a stream ended on a forming bar records confirmed 0; run_json_codes_test.py and test_report_outputs_keys.py cover the new refusals; test_run_json_outputs.py asserts the coded lines and that a colour slot is written as JSON integers. Test comments say what they test instead of naming internal design records. Docs: the switch's codes and that a call changing nothing answers 0 (pineforge.h, docs/outputs.md); a message holding a NUL reads to its first NUL while message_hash64 covers every recorded byte; call output_bar first in every calculation that writes; the ADR row names OutputEvent and the readers. Also restores the endif() of the per-entry-exit fixture block in tests/CMakeLists.txt, which the rebase onto main left out. Revises four sentences of the lane SM-E1 docs commit: the ADR row's first cell (`declare_outputs`, `output_run_begin`, `output_bar`, `output_value`, `output_event`, `output_constant`, `outputs_enabled_`) now also names the readers and OutputEvent; docs/outputs.md's "Fails (throws `std::logic_error`) when" list and its "`open_ms` is below the last row's" rule now give each refusal's code; report-schema.md's "`--outputs` on a module that records nothing" line now shows the coded failure line. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…failure record
The recorder is a generic kernel unit, so its two INT_MAX caps no longer
raise pine_runtime_limit. The catalog gains outputs_limit (class
strategy_limit, "A recorded-outputs capacity limit was reached during the
run.") with reason too_many_events or too_many_rows and max; both caps raise
it. pine_runtime_limit is main's entry again (limits map_pairs,
max_bars_back, matrix_elements, udt_objects), so its row of
run-failure-codes.md reads 4 values again. The enum, the registry, the
catalog diff and the page's tables are regenerated. test_outputs_recorder
checks the caps' exception as their sites throw it, as
test_run_failure_codes does for recalc_cap, and that a reason outside the
list reads engine_invariant.
strategy_outputs_set_enabled now follows strategy_set_trace_enabled and the
checked settings setters, which never clear a failure record: a success
leaves the record as it was, and a refusal keeps a failed run's own text
and code, as the C boundary keeps them for any refused call. Tests cover a
success after a refusal, and a success and a refusal after a failed run.
docs/outputs.md and engine.hpp say which exception type each failure throws
(std::runtime_error for the caps and a recalculation after a clear), and
pineforge.h and docs/outputs.md state the switch's record rule. Revised
sentences: docs/outputs.md's "The event queue and the rows reaching
`INT_MAX` are `pine_runtime_limit`", "A recalculation of a bar after a
caller cleared that bar's events is `outputs_rejected`", the
`strategy_outputs_set_enabled(s, on)` row ("0 with no failure left behind")
and the costs line "The event queue holds at most `INT_MAX` events between
clears"; the pine_runtime_limit row of run-failure-codes.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
luisleo526
force-pushed
the
sm/e1-post354
branch
from
October 8, 2026 11:47
cd3384f to
c1e12ba
Compare
luisleo526
marked this pull request as ready for review
October 9, 2026 04:20
luisleo526
added a commit
that referenced
this pull request
Oct 9, 2026
* Deliver a native run's report only for the call that began it (onto #356) One integration commit on main 70ddaa3 (recorded outputs, #356): the net change 31a40a1..d83ab7d4a of px/native-attempt-fix-20261009, 15 files, applied as a squash of that range. The author branch and its history are untouched. Supersedes 3d2f3f4 (the per-entry Completed gate): the reader gate 86a1856, its revert 124a434 and the restore ec9047d are folded into this net diff, so the documentation 3d2f3f4 wrote is carried here in its final form and this commit removes no documentation sentence of main. What changes (unchanged from the author head): - BacktestEngine::fill_report publishes the retained rows only while the latest run or stream_begin call began (reached reset_run_state). A call refused without beginning (a begin outside Ready, a refused bar array or run option, a calendar or timezone refusal, a begin over a live stream) gets the empty report, and strategy_native_run_v1 answers PF_NATIVE_E_RUN_FAILED for it instead of handing out the earlier run's report. - NativeExecutionConsumer keeps two bools, rows_current_ and batch_current_, behind a private RAII AttemptScope opened first by run_simple, run_tf, run_rich and stream_begin. strategy_stream_fill_report goes through native_stream_snapshot_report and fill_snapshot_report, so a live stream's rows survive a batch call refused over it, which reads empty. - No engine.hpp, ReportC or public API layout change, no codegen change, no version bump. The two bools are report-presentation bookkeeping and are not hashed; identity of the state hashes is owed on the parity population. - Tests kept: the 14 repro assertions (RED on 31a40a1) and the old, twin, current-partial and live-stream rows in tests/test_native_c_api.c, test_checked_settings.cpp and test_native_example_batch.cpp; one obsolete row in test_run_failure_codes.cpp (a second call on a Failed handle is refused, 0 trades). Integration against #356: - One textual conflict, docs/design/native-feature-parity.md rows OT6 and OT7 (adjacent lines changed by different sides). Kept #356's OT6 ("73 runtime exports") and the author's OT7 anchor (native_execution_consumer.cpp:7636). Nothing from either side was dropped. - The other five files #356 also changed (ADR 0001, native-engine.md, native_c_api.h, c_abi.cpp, engine_report.cpp) merged cleanly and keep every recorded-output implementation and doc line. - Six documentation anchors that the author's insertions moved, on lines the author's diff left alone, are carried by an exact line map: parity page lines 126, 159 and 1679, native-engine.md line 647, pine-to-native.md lines 129 and 466. - Not built, not run and not anchor-checked here (no compiler or project helper on this lane); the verification runbook is in the lane report. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Close the report-ownership review findings: pin, nested and snapshot controls, guide wording Follow-up to 44f7d0f (Deliver a native run's report only for the call that began it, onto #356), which integrated the 3d2f3f4 .. d83ab7d4a range onto main 70ddaa3. This commit answers the final review of that work (one P1, two P2) and changes no source under src/ or include/, no PF_API export and no admission behavior. P1, documentation content pin. The stream-report call in strategy_stream_fill_report (src/c_abi.cpp:770) sits inside the window c_abi.cpp:308-963 that docs/design/native-feature-parity.md row 8 pins by content, and a line-neutral edit does not preserve a content hash. The row's claim was re-read against the integrated source first: 73 PF_API definitions in c_abi.cpp, the 12-symbol stream family (strategy_stream_begin .. strategy_stream_fill_report), no export added or removed. The pin is refreshed from sha256:d19d72e6... to sha256:70be189a... by hashing the window's lines joined with newlines. All 15 content pins of the published pages were recomputed by text; 15 of 15 now match. P2, controls. tests/test_native_c_api.c gains check_nested_run_report_ownership: a genuine C callback, after a closed trade, makes a second strategy_native_run_v1 on its own handle over a poisoned output. The nested call must answer PF_NATIVE_E_RUN_FAILED with the empty report, while the outer run ends Failed (Contract, read from the state, not assumed Running) and publishes its own partial trade. tests/test_checked_settings.cpp gains test_snapshot_restores_batch_gate: after a batch call refused over a live stream, a normal and a throwing (injected allocation failure) strategy_stream_fill_report are followed by BacktestEngine::fill_report with no run in between, which must still answer empty. Both files only gain lines: the 14 reproduction assertions, the old, twin, live and started-then-failed controls are untouched. P2, guide. docs/pages/native-engine.md "A run's report." no longer says every listed refusal leaves the lifecycle, the run identity and the retained rows as it found them. It states the cause-by-cause behavior that exists unchanged on the base: a begin on a Completed, Failed or unconfigured handle and a refused bar array or run option preserve them; a calendar or timezone refusal comes after the consumed run-number high-water moved and fails the lifecycle; a begin while Running latches Failed (Contract, Begin; Contract, Configure on a generated source host). It also states that the batch reader stays empty after a snapshot and that a nested run call reads empty while the outer run keeps its rows. This rewords sentences introduced by 44f7d0f (and before it by 3d2f3f4, 86a1856, ec9047d): the dropped sentence "leaves the lifecycle, the run identity and the retained rows as it found them" is replaced, not restored. Not built, not run, not guard-checked here (no compiler or project helper on this lane). Focused GREEN and the integrated-tree proof are owed on a spot box. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Move the stream snapshot bridge into the engine epoch the version guard checks The measured final run (green1, guards/05-native-versions.log) failed the declared source guard scripts/check_native_cpp_versions.py on 1c89e87 while the same guard passed on the newmain base 70ddaa3: ValueError: pineforge out-of-line method must belong to engine_script_run_v19 Cause: native_stream_snapshot_report was defined after the closing brace of `inline namespace engine_script_run_v19` in src/native_execution_consumer.cpp. The guard compares every qualified `Name::member(` match in `namespace pineforge` with the matches inside the epoch, so the bridge's `NativeExecutionConsumer::fill_snapshot_report(` call counted as outside the epoch (guards-mid/diag.mid.txt: {'NativeExecutionConsumer::fill_snapshot_report': 1}); the next comparison, namespace_functions, would have named the bridge itself. Fix, two line-neutral hunks and nothing else: - src/native_execution_consumer.cpp: the six-line bridge (two comment lines, the three-line definition, one blank line) moves above the epoch's closing brace. The definition and the fill_snapshot_report, rows_are_current and AttemptScope bodies are byte-identical; one comment line gains ", in the engine epoch". The pinned native_run_spec_v3 digest block stays where it was. - src/c_abi.cpp: the forward declaration beside the two native_c_host hooks is reopened in `inline namespace engine_script_run_v19`, on its own single line (the shape market_driver.hpp uses for its native_run_spec_v3 forward declaration), so it names the same entity as the definition. The call pineforge::native_stream_snapshot_report( in strategy_stream_fill_report is unchanged: qualified lookup finds the member of the inline namespace. The mangled name gains the epoch component, and the two files above are the only mentions of the bridge in the tree. No PF_API export, header, layout, hash, checker, allowlist or version changes. Both files keep their line counts (c_abi.cpp 1173, native_execution_consumer.cpp 11557) and every line outside the touched hunks keeps its number, so no file:line anchor and none of the content pins of the published pages move. All 21 pin occurrences were re-hashed by text on this tree and match (the six c_abi.cpp windows start at line 308); 137 anchors into these two files were scanned and none touches a changed line. No documentation changes. Not compiled or run here (local-only lane). A read-only text replay of the guard's ownership comparison (stdlib only, the guard itself not executed) reproduces the measured difference exactly on 1c89e87; on this tree TYPE_DEF, ALIAS_DEF, METHOD_DEF and namespace_functions agree for the consumer .cpp and .hpp. The build, ctest, the declared source guards and the doc gates still have to be re-run on a spot box. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Correct attempt-report citations and qualify refusal documentation Apply the retained docdelta1 patch after the private bridge epoch repair. This corrects 1c89e87 and the earlier 44f7d0f citation mapping. Replace the broad claim "A calendar or timezone refusal comes after the consumed run-number high-water moved" with its begin_ready scope. Replace "A begin while the handle is Running" with the actual batch/reentrant path and the repeated-stream exception. Four citations now point at the source they describe. Runtime behavior is unchanged. The author timed out; the patch and sidecar predate the bound, while its detailed report was written after it. Final independent review and exact-head runtime checks remain required. --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds opt-in recorded outputs to the engine and C ABI, exposed through the JSON runner's
--outputspath. Recording stays off by default. The recorder supports row restarts, event clearing, run constants and bounded capacity while preserving existing failure information.Malformed output manifests receive a structured
strategy_library_incompatiblerefusal before lookup or sorting. Consumed indices must be nonnegative integers with booleans rejected; output identifiers remain nonempty opaque strings. Raw manifest hashing, off-mode behavior and exactly-once cleanup are preserved.Recorded outputs are unreleased and targeted for 1.5.0, outside the 1.4.0 release. The catalog retains all 38 released
sincevalues;outputs_limitandoutputs_rejectedare added at 1.5.0. The catalog diff is based on the actual v1.4.0 tag. Documentation marks this boundary, including the two new reasons onstrategy_library_incompatible.Source and review
c1e12bafe016cc47773df4119dd68b49dc6de71b; tree:2551edacc700114174adf72b84a7b4a7c4983f71; base:31a40a14b16c4881184d8b6015df0e4ed19cfe26.cd3384fatree, only the README disclosure clause differs; all 11,657 other tracked entries, including runtime, headers, tests and recipes, are identical. Documentation-reversion and whitespace checks passed. ABI version stays 4.Validation
All eight canonical steps passed on this exact head and were independently re-judged from 240 hash-verified exported files: preflight 49 stages; release 880, debug 861, sanitizers 852 and kernel 313 executed rows; no skipped, disabled or not-run rows; docs zero warnings; full corpus 312/312 and subset 54/54 match their baselines with zero moves or gaps.
The complete cold native/live matrix also passed on this head: native 948/948, live ASan/UBSan 85/85 and live TSan 85/85. Registration, JUnit and console inventories agree with no excluded or skipped rows. All seven recorded-output rows and four receipt-gated ABI rows executed; seven exported provider receipt identities and hashes reconcile. All three official artifact digests and 176 members were verified. Compiler and curl cache restoration were bypassed.
A fresh candidate image was built once at this head, with producer
8663272ba530bacffe5dc0ef054697b87cf48173. Its immutable identity issha256:17e116c5431333fcad691021ca66af7188001c41845c5a43db5a730ad5acf1c4, linux/amd64. Installed audit retained and independently rehashed all 144 file identities. All eight unchanged CLI modes and their external inventories passed: 60 cases / 883 checks / 60 plugins, including the original mode's two expected-refusal cases. The real allowlist packet passed the exact reviewed guard at 16 cases / 124 claims / 16 plugins. All 40 identity controls and 18 original guard controls matched their expected outcomes.The original six compiled-fixture tests passed. Separate actual installed-runner ON/OFF samples passed with the same recording plugin and nine-row synthetic CSV: ON records nine bars, four series and nine events; OFF omits the output block and flag. Expected values, fingerprint bindings and ordinary behavior were independently reconciled. These samples retain actual engine version 1.4.0 with commit c1e12ba, codegen distribution 1.4.0 and
transpiled_from_pine=false; they prove compiled fixtures, not a Pine recording producer. All 18 stage/final archives were verified before acknowledgement, and resources were collected. The complete final archive is SHA25651b5f747c6e257f63b61ca895b910d2d3068f5ebe296d981a05997405773444f.The old image proof remains identified with
cd3384fa. It is not carried over: the recipe copies the whole checkout, including README, into the image, so the current head received the fresh build and complete proof above. Images retain their actual source commit and embedded identity.Historical failure is preserved: the old external guard refused the retained real packet because it applied semantic type checks to canonical-token data instead of raw provenance, and contained an unsupported native-scalar assumption. The corrected guard received both independent static approvals and passed evaluation on that immutable packet: 16 cases, 124 claims, 16 plugins, four genuine positive facts, 20 exact-refusal mutants, 18 historical allowlist controls and 40 identity controls. All 140 runtime gates matched; the old guard reproduced its original refusal. The whole export, raw control outcomes, unchanged guard bytes and early-cutoff binding were independently reconciled. Intermediate per-stage copies were not retained during that uninterrupted 24-second evaluation; all final raw artifacts are present, and the recoverability-only deviation is recorded. The subsequent fresh-image proof used the same guard bytes with complete stage-by-stage preservation.
Evidence limits
Historical canonical runs retained preparation commands, actual ABI runtime passes, textual discovery/result/LastTest sets, CI summaries and clean source/tree bindings. They did not retain raw configuration-bound provider receipt JSON and archive/header products for independent rehash in release/debug/full sanitizers, original JSON inventories and JUnit files, or supplemental per-stage source/tag receipts. Their catalog checks used the documented no-release-tag fallback: the catalog was reconstructed from the pinned diff and regenerated identically, without verifying the release tag or its catalog digest against Git on those hosts.
The maintainer's ruling accepts the repository's canonical verification contract for those runs. The stronger historical manual retention plan was not fulfilled; that retention finding was disposed of by the maintainer's ruling, not repaired or waived here. Missing originals are not reconstructed. The cold native/live workflow fetched full history and release tags and retained normal diagnostics, including seven provider receipt JSON files; provider binaries and complete header/source archives beyond its exporter were not retained or independently rehashed. Current-head canonical and cold evidence was independently reconciled within those normal export contracts.
Campaign gate and follow-ups
The one final-tree sweep
exp-sm-e1-20261009-033800, attempt 1, completed 134/134 cases and all 7,989 probes, with no engine errors or unmeasured probes. It paired this engine head with current codegen main8663272ba530bacffe5dc0ef054697b87cf48173. The official gate returned PASS_NO_IMPROVE_NO_REGRESSION: zero coverage loss, hard regressions, hard moves, target leavers or tier/outcome changes. Both required statuses,pineforge/verifyandpineforge/parity, are recorded success on this exact head.The snapshot publication job
pineforge-publish-snapshot-4222kcompleted successfully. Its 1,521,034-byte snapshot was fetched from the evidence store and independently verified against the parity verdict's candidate hash:1404bbf8599c6029c44cf78d1fe88d1c8149a20f710010dcdc55549f9b1dc650. Parity verdict:25e8c59a56144acc314a4760819a7368a6195edafbf843285910153a12f69b40; recorded gate:13074fc0958834e44ccd73ad5c7275cca7e8dc364fb39e79c2ac98807efc2739. Canonical verification, cold CI, both independent reviews, retained-packet guard evaluation and fresh installed-image proof remain accepted within the evidence limits above.Three retained P3 findings stay outside this request: catalog wording for manifest refusal, hline constant-index boolean handling, and permissive NaN/Infinity parsing in nonconforming manifests. Historical failures keep their original identities.