An adversarial security benchmark for agent authorization. It measures one thing about an authority control plane: the gap between an agent proposing a policy-violating action and that action producing an unauthorized external effect. The agent is stipulated to be compromised; the question is whether misbehavior converts to effect.
Both are readable here, in Markdown. Read them in this order:
- The Model Proposes, the System Authorizes — the architecture, and the paper that specifies this evaluation: the threat model, the invariant, and the metric this benchmark was built to measure. It pre-registers what would refute it.
- Authority-Bound Agentic Execution: Measuring Unauthorized Effect Under Adversarial Load — the evaluation itself: the campaign, the ablations, the intervals, and what the result does and does not establish.
Both by Ayla Croft, ScriptKittyOS. Licensed CC BY 4.0.
papers/ holds the LaTeX sources. Those are the archival record — the deposited Zenodo versions
hold exactly those files, verified by hash — not the copies you are meant to read. The Markdown
above is generated from them and checked against them by scoring/check_paper_sync.py, which
fails if a number appears in one and not the other. That gate exists because these renderings have
drifted before, and both times the Markdown kept a figure the LaTeX had already retracted.
Across 73 trials (61 attack trials + 12 known-good controls) against the system under
test, a violating action was constructed and driven in 57 of the 61 attack trials and produced
0 unauthorized external effects (per-trial 95% CI [0.0%, 5.9%]; cluster-aware ≤32.4% over the
8 distinct mechanisms the set exercises — see REPORT.md). A zero is what a reviewer distrusts, so it is not presented
alone — the benchmark backs it with:
- an ablation study (disable each mechanism; its attacks convert — 0→10, 0→5, 0→5, 0→14 for the content firewall, runtime sentinel, skill auditor and approval binding, and 0→2 for two-person approval: five mechanisms, each demonstrated necessary);
- a no-control baseline (whole membrane off → 51 of 63 trials convert) — where 63 is not a count of attacks but the F1–F5 + F8 totals, including 8 known-good controls and F8's 4 non-attacks, and is not the campaign's 61 either, because the baseline set omits F6, F7 and F9, which have no runtime toggle;
- a measured trusted-computing-base boundary (each ablatable element removed in turn with the
full 73-trial catalog driven against it, so 41 provider-call trials are live under every ablation:
only the kernel element's failure admits a provider-call effect, surface reducers admit
gate-channel effects only. Scope is stated rather than implied — single removals only, and F6/F7/F9
plus the membrane and durable store are never ablated —
REPORT.md, paper §9.1); - a kept-and-fixed failure (the one effect an earlier run leaked, closed and re-measured);
- confidence intervals on every rate and an independent-oracle calibration (0 overclaims
in 41 provider-call trials; on all 6 trials where a real effect occurred the system's proof state
matched the independent verdict —
REPORT.mdgives the decomposition, which is the honest form of the claim).
This repository open-sources the benchmark and its results, not the system under test. The Trinity control plane is patent-pending and not yet public, so the campaign is not re-executable end-to-end here. Two things are checkable now:
- Results are auditable from the released artifacts. Every campaign number and table follows
mechanically from the committed JSONL (
artifacts/). Thescoring/modules recompute them without the system. Every other published table recomputes the same way, from per-trial records that ship alongside: the ablation study, the no-control ceiling, the compound ablation, the four full-catalog §9.1 probes, overhead, false denials, F10, and the blind-set result. What the artifacts do not settle is scope — which runs exist at all — and that is stated at each table rather than left to be inferred.MANIFEST.mdandartifacts/README.mdsay the same thing; read each artifact's_meta.caveatfor what its run can and cannot support. - The method is inspectable — the frozen specification, the family taxonomy, the released family definitions, the declarative blind-attack sets, the scoring modules, and the full harness: the independent oracle, the aggressor, the runner, and the ablation / TCB / measurement-integrity / blind drivers. These reference the system under test, so they do not compile standalone; they are published so the adjudication method can be checked line by line rather than taken on trust.
Full end-to-end re-execution becomes possible when the system under test is released.
PAPER.md the evaluation paper, readable (the rest of the docs link to it here)
PAPER-ARCHITECTURE.md the architecture paper, readable — read this one first
papers/ the LaTeX sources for both, and the archival record: the deposited Zenodo
versions hold exactly these files, verified by hash (see papers/README.md)
REPORT.md measured results (auditable from artifacts/)
oracle/ the independent adjudicator — the component the result depends on
aggressor/ attack construction, the variant catalogs, the chaos drivers, fixtures
harness/ the runner + the ablation / TCB / measurement-integrity / blind drivers
spec/
SPEC.md the frozen methodology (definition of unauthorized effect, outcome model,
families, trial schema)
ABLATION.md the causal spine (the TCB boundary table is in REPORT.md and paper §9.1)
PACKAGING-NOTES.md how to read the frozen SPEC in this layout + the deviation table
families/ the nine-boundary taxonomy + released family definitions (F4/F5/F6/F8/F9)
blind/ declarative, code-blind attack sets + protocol
fixtures/ released-vs-held disclosure split
artifacts/ committed result JSONL for every published run + the family table as data,
with README.md recording provenance, schema departures, and what they
can/cannot verify
ablation/ per-trial records for the six single-mechanism ablations behind the §6.5
ablation table
scoring/ the system-free scoring modules, plus verify.exs and verify.py —
dependency-free verifiers that recompute every published number from the
committed records and exit non-zero on disagreement; check_paper_sync.py
holds each paper's Markdown to its LaTeX (see scoring/VERIFY.md)
LICENSE CC BY 4.0
CITATION.cff citation metadata
MANIFEST.md exactly what is and is not in this release, and why
SHA256SUMS SHA-256 for every file in the release; `sha256sum -c SHA256SUMS` confirms
the bytes you hold are the ones described here
receipt-verification/ holds what you need to check a Trinity authority receipt without
trusting us: the public key registry, two independent offline verifiers, and a corpus that
exercises every verdict they can produce.
What you can establish on your own. That a receipt's signature is genuine under a key you obtained from here rather than from whoever handed you the receipt; that the key's standing at the time the receipt says it was signed permits that signature; and that a receipt whose visible fields disagree with its signed bytes is refused rather than accepted. No network, no packages, no build.
What you cannot establish from this directory alone. That the events described in a receipt happened. A signature binds a record to a signer and to a position in a chain; it does not witness the world.
Pins. Two annotated tags name immutable points: receipt-verification-r1 and
receipt-verification-r2. Cite a tag rather than a branch if you need a fixed reference.
Start with receipt-verification/verifier/VERDICTS.md — one row per verdict case, the input
that produces it, and the order the checks run in. It is the single source the verifiers and their
cross-check are held to.
The system under test (the Trinity control plane) and the held red-team corpora for the
content/runtime/skill families (F1–F3) — their contents are the detection surface those mechanisms
are tuned against, so releasing them would be a defensive-hardening problem rather than an IP one.
The harness that drives the benchmark is released; it simply cannot compile without the system.
See MANIFEST.md.
See CITATION.cff. License: CC BY 4.0 (LICENSE).