This repository was archived by the owner on Oct 24, 2026. It is now read-only.
Repository navigation
refactor(scripts): one constant-test loop, not two #326
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -188,18 +188,22 @@ Both write real rows to the target. Point them at staging. | |
| | Command | Proves | Requires | | ||
| |---|---|---| | ||
| | `make trust-conformance-live` | Golden path; sequential replay; 15 identical concurrent requests exposing one receipt identity or explicit `idempotency_in_progress`, followed by a completed replay and one charge; a changed payload under a reused key conflicting rather than replaying; budget denial; expired and forged permit rejection; receipt and audit-chain verification; tenant isolation against a directly-supplied foreign wallet and permit id | `AGENT_MIDDLEWARE_API_KEY`; set `AGENT_MIDDLEWARE_API_URL` or it defaults to production | | ||
| | `make scoped-smoke-loop` | Authority readable for the caller's own wallet; scoped permit issued and carrying a nonce; governed invoke; the debit matching the receipt's own stated cost; replay returning the same receipt and charging nothing further; a changed payload under a reused key refused; an out-of-scope but genuinely registered tool denied and not charged; the portable receipt carrying a signature, key id, and named canonicalization | `AGENT_MIDDLEWARE_API_KEY` holding a **wallet-scoped** key (the loop refuses a bootstrap admin key); `--tool` required off loopback | | ||
| | `python scripts/constant_test_loop.py` | Scoped permit sized from the tool's advertised `creditsPerCall`; governed invoke; signed success receipt with a ledger entry; replay returning the same `receipt_id` with no second debit; an out-of-scope but genuinely registered tool refused with `permit_tool_not_allowed` and charged nothing | `CI_SMOKE_AGENT_KEY` holding a **wallet-scoped** key (self-provisions on loopback when absent); `--tool` and `--tool-args` required off loopback | | ||
| | `make adversarial-battery-live` | Wallet isolation, invalid-key rejection, forged-receipt rejection, permit key binding, expired permits, revoked keys, replay idempotency; always revokes keys it minted | `API_URL` (no default, by design) and `BOOTSTRAP_KEY` | | ||
|
|
||
| `scoped-smoke-loop` is the one built to run continuously: it needs no admin | ||
| credential, so the key it runs on cannot mint keys or read another tenant's | ||
| audit trail if the runner holding it is compromised. That is also its limit — | ||
| tenant isolation, audit-chain verification, and refund reconciliation are | ||
| absent from it precisely because a scoped key must not be able to reach them, | ||
| and they stay in `trust-conformance-live` for an operator to run deliberately. | ||
| It reports SKIP rather than a false PASS for the out-of-scope denial when the | ||
| deployment registers only one tool, since invoking an unregistered name proves | ||
| "tool not found" rather than "the permit refused it". | ||
| `constant_test_loop.py` is the one built to run continuously: it needs no | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The surviving script executes Useful? React with 👍 / 👎. |
||
| admin credential, so the key it runs on cannot mint keys or read another | ||
| tenant's audit trail if the runner holding it is compromised. That is also its | ||
| limit — tenant isolation, audit-chain verification, and refund reconciliation | ||
| are absent from it precisely because a scoped key must not reach them, and | ||
| they stay in `trust-conformance-live` for an operator to run deliberately. | ||
|
|
||
| It skips rather than false-passes the out-of-scope denial when the deployment | ||
| registers only one tool, since invoking an unregistered name proves "tool not | ||
| found" rather than "the permit refused it". Off loopback it refuses to start | ||
| without `--tool` and `--tool-args`: arguments derived from a schema satisfy a | ||
| tool's declared shape but not its semantics, and the derivation picks the | ||
| first enum member, which for a consequential tool could be `delete`. | ||
|
|
||
| The battery reports SKIP — never a false PASS — for MCP-invocation checks when | ||
| the deployment exposes no invokable `golden-path-echo` tool. It does not | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The deleted smoke loop fetched
/v1/receipts/{receipt_id}/portableand checked its signature, key ID, and canonicalization, butrun_constant_testnow stops after checking receipt outcome, ledger linkage, and charge. Consequently, a deployment with a broken portable receipt endpoint or missing verification metadata still printsALL INVARIANTS HELD; the matrix's “signed success receipt” wording overstates what this replacement actually proves. Restore those checks before treating this as the sole continuous receipt proof.AGENTS.md reference: AGENTS.md:L68-L80
Useful? React with 👍 / 👎.