Behavior evidence gap: linking agent actions back to BOM components

Context

I'm one of the maintainers of WasmAgent/agentbom — a reference implementation library for describing AI agent inventories (tools, permission scopes, model dependencies, compliance posture, audit trail) as a single verifiable document. We've been tracking the CycloneDX Agent BOM proposal and have been building against it as an early implementer.

Following Steve Springett's suggestion on the existing thread, I'm opening this as a dedicated issue to track the specific gap I raised.

The gap: behavior evidence is not addressable in the current design

The current Blueprints RFC defines runtime evidence — attesting what an agent is (which components are loaded, which version is running). This is valuable and we implement it via our agentbom.evidence_layer.evidence_hashes field.

But in practice, the harder problem for AI agents is attesting what an agent did — specifically:

Which tool calls were made, in what order, with what inputs
Which memory cells were read/written (and whether they were tampered with between reads)
Whether a checkpoint/fork occurred, and from what parent state
Which policy bundle was active at the moment of a decision

These are not runtime inventory facts — they're behavioral trace records that need to be linked back to the BOM so a verifier can say: "the agent that ran this task was operating under this policy bundle, with these tools, and its actions are provably this sequence."

What we've built (implementation experience)

In wasmagent-protocol, we define the Agent Evidence Protocol (AEP) — a set of JSON schemas for:

Schema	What it attests
`aep-record`	A single agent action — links `run_id`, `model_id`, `policy_bundle_digest`, `tool_manifest_digest`, input/output refs with taint labels
`checkpoint-evidence`	Verifiable resume point — `state_digest`, `fork_of` parent checkpoint, tamper-evident via ed25519 signature
`memory-evidence`	Read/write ops with hash-chain (`prior_digest` → `digest`) making historical tampering detectable
`replay-evidence`	Post-hoc replay verification record

In agentbom, each BOM document carries an evidence_layer with aep_references (AEP record IDs or hashes), plus an audit_log of structured events (tool_call, permission_check, prompt_injection_attempt). The BOM thus serves as the static inventory anchor that behavioral trace records reference back to.

The key structural insight from our implementation: behavior evidence doesn't need to live inside the BOM — it needs a stable reference point in the BOM (tool manifest digest, policy bundle digest, model snapshot) so that an external evidence record can prove it was generated under those exact conditions.

Concrete proposal for consideration

Before the August 20 TC54 vote, I'd suggest considering whether the Blueprints design should define:

A behaviorEvidence reference type — a pointer (hash + URI) from a BOM component to an external behavioral trace record, analogous to how SBOM components carry externalReferences
Stable digest anchors — explicit fields in the agent component for toolManifestDigest and policyBundleDigest, so external evidence records can reference a specific BOM snapshot
Evidence envelope conventions — even if the trace format is out of scope for CycloneDX, defining the envelope schema (schema_version, trace_id, created_at_ms, optional signature) would let implementers build interoperable evidence records

These are additive — they don't require the trace format itself to be standardized, just the linking mechanism.

Why this matters for AI agents specifically

Software SBOM consumers already have mature tooling for runtime evidence (CVE scanners, SBOM diff). AI agent consumers face a different threat model: prompt injection, tool misuse, memory poisoning, and checkpoint replay attacks — all of which require behavioral evidence to detect or audit post-hoc. An agent BOM that can only attest "these tools were loaded" but cannot anchor "and these are the actions that were taken under this tool set" leaves a critical gap for the primary AI-specific risk scenarios.

Happy to share our schema files or discuss further.

I ran into this exact gap while building mcp-audit-gateway. Each hash-chained tool call record in my implementation carries a toolManifestDigest and policyDigest, which maps directly to what your aep-record schema does with tool_manifest_digest and policy_bundle_digest. The convergence between our implementations is encouraging.

The part I got stuck on: what does the BOM-side target of those digests actually look like structurally? CycloneDX has service objects with trust boundaries, which covers tool endpoints. But an agent binds a model to tools under specific constraints, and that binding is what behavioral evidence needs to reference back to.

A few open questions from my experience:

Should tools reference existing service BOM entries via ref and just layer agent-specific metadata on top (permissions, side effects, approval gates)? Or does the tool manifest need its own shape?
MCP tools register dynamically via tools/list and can change mid-session. The manifest hash is a point-in-time snapshot. Does your checkpoint-evidence schema already handle epoch transitions here?
Per Steve's decomposition response on # Proposal: AI Behavioral Risk Assertions for CycloneDX 2.0 #956, rate limits / input filtering / budgets are arguably different CycloneDX constructs. But from the evidence-linking perspective, a verifier just needs one hash. Is a single policyBundle digest sufficient, or should each concern be independently digestable?
Since dependencies uses strict typing, delegation scope probably needs its own construct. Could start as registered properties and graduate to schema if proven?

I can share schema files from my implementation if useful for comparison with your AEP work. Would also be glad to collaborate on a PR if this direction has traction before the TC54 vote.

@elang2

@elang2 — answering 3 and 2, because I have measurements for those two specifically rather than an opinion on all four.

On 3, whether one policyBundle digest is sufficient. I ran an experiment on this and the answer came out narrower than I expected: a policy digest pins which policy text ran, but not what input that policy evaluated, and the gap between those two is where the failures live.

The setup holds the action fixed and varies only the resource — both requests are documents:read, both targets are documents, and only the relation between the resource and the delegated scope differs. Four policies, same five checks:

Check	action-only	relation-aware	caller-controlled	agent-bound
in-scope resource allowed	PASS	PASS	PASS	PASS
out-of-scope resource blocked	FAIL	PASS	PASS	PASS
relation present in decision input	FAIL	PASS	PASS	PASS
relation evaluated, not assumed	FAIL	PASS	PASS	PASS
scope survives identifier substitution	FAIL	PASS	PASS	PASS
relation input not caller-controlled	FAIL	PASS	FAIL	PASS

The caller-controlled column is the one that bears on your question. That policy is genuinely relation-aware and genuinely evaluates the relation — it passes every check except the last, because the organization it compares against is read from a caller-supplied field rather than from the grant. It would produce a correct-looking evidence record under a valid policyBundle digest, every time, while evaluating attacker-supplied data.

A later amendment found the same shape one axis over, and this time in the experiment's own reference policy: it checked input.agent_id != "" and input.delegation_id != "", which never establishes that the agent presenting a delegation is the agent it was issued to. Both fields are present in the decision input and both are evaluated — so "relation present" and "relation evaluated" are satisfied by a policy that binds neither. Presence plus evaluation is not binding, and I only caught it because a reviewer pushed the method back at the reference implementation.

So decomposing the policy into independently digestable concerns — rate limits, input filtering, budgets, per Steve's decomposition on #956 — doesn't reach this. What's missing isn't finer resolution on the policy side; it's that the decision input actually evaluated is not addressable at all. If the anchors end at toolManifestDigest and policyBundleDigest, a verifier can prove which rules ran and still not prove what they ran against, nor which fields were genuinely bound rather than merely present. That argues for a third anchor rather than a subdivided second one.

Preregistration, both policies, and runs: https://github.com/Silentpartnercoding/agent-trust-benchmark/blob/main/docs/E006.md

On 2, mid-session tools/list changes against a point-in-time hash. Worth knowing this is live in at least two other places right now. I tested a signed WebMCP manifest against five adversarial cases, and "authorized tool after its schema dictionary changes" denies only when the decision binds a canonical versioned dictionary digest rather than the tool name — signed discovery authenticates the advertised menu, it does not authorize a caller.

More useful to you: openai/codex#36381 hit the identical problem three weeks ago. @sattyamjjain's argument there is the sharpest statement of it I have seen — notifications/tools/list_changed exists precisely so a server can announce an altered surface mid-session, and a server that stays quiet and swaps a description doesn't even need it, so a sealed set stays a subset nominally while the authority behind it widens. @ayanamislover then implemented digests over the normalized definition (namespace/name, description, input schema, execution options) and found a real regression doing it. That thread is further along than mine on the implementation side.

Limits, stated so you can weigh this properly. E006 runs OPA and Cedar policies against a two-organization fixture, not a CycloneDX document — the checks are about whether an engine evaluates a relation, and the mapping onto BOM structure is your and @telleroutlook's question, not something I have measured. The WebMCP result is one live manifest, one observation date.

One note on direction rather than structure: @telleroutlook's framing that behavior evidence needs a stable reference point in the BOM rather than living inside it matches where I ended up on a different protocol's seam — carrying a reference plus enough binding to detect substitution, with the carrier asserting only that the reference was observed and transported, never that it is valid or sufficient for admission. Convergent, and I mention it only because three independent implementations arriving at the same shape in a month is itself worth noting before a vote.

@Silentpartnercoding

@Silentpartnercoding Really appreciate you running actual experiments against this rather than theorizing!

In mcp-audit-gateway the attestation records already carry full call parameters in the signed payload, so the chain proves what was called with what input. But reading your E006 results, I realize the policy side has nothing equivalent. policyBundleDigest proves which rules existed, not what they evaluated. Your caller-controlled column is a subtle failure mode. Passes every check and still evaluates attacker input.

Reading your results, what this looks like to me is a decisionContextDigest on the BOM side. Hash the input frame the policy was given, not the policy itself. In my implementation the tool call IS the decision input (it's what gets signed), but there's no way to reference that from a BOM without a third anchor type.

I put together a draft extending mcp-audit-gateway to emit this as a separate digestable field: elang2/mcp-audit-gateway#1. Happy to have you run E006 against it if useful.

@elang2

Thanks for the quote. One correction to my own argument first, because I had it slightly wrong in a way that matters here.

I said notifications/tools/list_changed "exists precisely so a server can announce" an altered surface. In the 2026-07-28 revision, it is a SHOULD, not a MUST, and it is gated on the client having opened a subscriptions/listen stream with toolsListChanged: true. So a server that never sends it is conformant, and a client that never subscribed will not see it even when the server does send it. The same section says the tool set MUST NOT vary per-connection, and says nothing at all about description stability. That makes the silent-swap case stronger than I put it, not weaker, and @elang2's question 2 has a concrete answer already: treat list_changed as ceiling invalidation rather than as a refresh, and refuse providers that can mutate behaviour behind an unchanged advertised definition. That is what @ayanamislover ended up shipping on codex#36381.

On timing. The TC54 teleconference was today, and #678 merged into 2.0-dev at 15:41 UTC. I grepped the bundled 2.0 schema for toolManifestDigest, policyBundleDigest, behaviorEvidence, and decisionContextDigest. Zero occurrences of each. So this is a 2.0-dev conversation now rather than a pre-vote one, and the thing it has to reconcile with is cyclonedx-evidence-2.0.schema.json, which was merged the same afternoon under #980 and already carries assertion, partyAttribution, analysisMethod, supportingData, and evidenceDataType. A third anchor that does not land inside that model is a parallel mechanism, and parallel mechanisms in evidence schemas tend not to survive.

@elang2 on decisionContextDigest. I shipped that shape yesterday, so rather than say whether I agree, here is what it cost.

FerrumDeck's audit log now records the permission state at decision time rather than at admission time: identity, budget quantities, and a SHA-256 of the policy document, in a content-addressed immutable table. Two things surfaced while doing it, and both were live in every deployment. Escalated tool calls were being written to the audit log as allowed. And the human-approval control was unreachable, with approval_requests empty everywhere. Nothing was wrong about the policy digest. Everything was wrong about the decision the record described. That is your third anchor argued from the failure side, and it is a stronger argument than the design one.

The part I would push on is who asserts it. In mnemo this week, I found the server advertising 2026-07-28 protocol support it does not implement, while actually negotiating 2025-11-25. A version string in a handshake, wrong in my own code, for an unknown length of time, because nothing external checked it. If a component can be wrong about its own protocol version, it can be wrong about its own decisionContextDigest, and a self-asserted anchor inside a signed record is only as good as the component's honesty about its own state. mnemo's answer is a hash chain over agent_events with an offline external verifier, which catches 100% of single-byte mutations over 256 trials; Wilson's 95% lower bound is 98.5%. That detects tampering after the fact. It does not detect a component that was sincerely wrong at write time, and I do not have an answer for that one.

@Silentpartnercoding, one correction on the #956 reference, since your question 3 partly rests on it. Springett's decomposition there is weakness, threat, vulnerability, risk. It is not rate limits, input filtering, and budgets. Applied to your caller-controlled column, that is useful rather than pedantic: a policy that reads the comparison organization from a caller-supplied field is a weakness with a CWE behind it, the injection is the threat, and the specific run is the vulnerability instance. Which means 2.0 may already have somewhere to put that finding without needing a finer policy digest at all.

Last, a deployment number, because all three proposals assume servers that can emit these anchors. agent-audit-kit scanned 2,303 public MCP configurations. Zero serve RFC 9728 discovery, which the 2026-07-28 revision makes a server MUST. 52.3% declare a remote server with no auth, and of the ones using inline auth, 421 of 421 hardcode a static credential. That population is not going to emit a toolManifestDigest because a schema field exists for it. Whatever gets specified needs a story for evidence produced by the host or the gateway rather than by the server being described; otherwise, the field gets populated by exactly the servers that were already careful and stays empty for the ones the threat model is about.

Happy to write up the FerrumDeck decision-record shape against cyclonedx-evidence-2.0 if that is useful to anyone.

@sattyamjjain

@sattyamjjain Yeah, the self-attestation problem is the part I keep circling. mcp-audit-gateway helps partially because it's watching the wire. The gateway doesn't trust the server to describe itself, so for "what was called" you actually get an independent witness.

decisionContextDigest is where it falls apart though. The gateway can't see inside the policy engine. If the engine has a bug like your escalation-logged-as-allowed case, the digest is perfectly valid for a decision that was just wrong. I don't have a clean answer for that. Recording the full input frame so someone can replay against the policy document later is the best I've landed on, but that only catches deterministic bugs.

On deployment population, agreed. Servers aren't going to self-report. Gateway-as-asserter is the only thing that maps to reality there.

The distinction worth drawing is between what the gateway witnesses and what it relays.

For "what was called," you have an independent witness, as you say. For "what was decided," you do not, and the record does not currently mark the difference. That is the shape of the FerrumDeck bug rather than a separate problem from it. The audit log said allowed, and allowed is a claim about the engine's decision, not about the call. The wire was fine. The claim sitting on top of the wire was wrong, and one signature covered both equally.

The third anchor still earns its place; it just answers a different question than the one you are circling. decisionContextDigest tells you what the policy evaluated. It does not tell you who says so.

For that part, I would split the assertion and attribute each half rather than add anything. The gateway asserts it observed this call, with these arguments, at this time. The engine asserts it evaluated this input frame under this policy and returned this effect. One record, two asserters, and a verifier can see at a glance that the second half has no independent backing. cyclonedx-evidence-2.0 already has the slot, since #980 landed partyAttribution and assertion.parties. Marking which parts of an evidence record are witnessed and which are self-reported is what a multi-party attribution field is for, and it is a much smaller change than a new anchor type.

On replay only catching deterministic bugs, I think you are underselling it, and the real problem sits one layer down. The FerrumDeck escalation bug was deterministic. Replay would have caught it on any run. What hid it was that the suites which would have exercised it were reporting green while executing nothing: 135 tests collected, 135 skipped, 0 actually run. The change that surfaced it was a CI gate asserting on executed count rather than collected count. So the case for replay is not that it catches everything. It is that it catches a lot and almost nobody runs it continuously, which is a fixable problem rather than a fundamental one.

For the genuinely non-deterministic remainder, I only have a partial. Two engines, or one engine at two versions, given a byte-identical input frame. A disagreement is a signal that needs no ground truth. It tells you nothing about which one is right, which is a real limit, and I would not oversell it, but "these two disagreed on the same frame" is far cheaper to alert on than correctness, and it does not need an oracle.

Yeah, the single-signature problem is real. Once you flatten the gateway's wire observation and the engine's claim into one signed blob, a verifier can't tell who saw what. I went through my fields and it's not even close. The gateway witnesses 11 of them (tool, args, result, timing, principal, namespace, upstream, chain hash, record id, method, success/error). The engine only contributes one. decisionContextDigest.

I went ahead and implemented it: elang2/mcp-audit-gateway@d8532fa

Each record now carries a parties array that marks who witnessed or asserted which fields. Six test vectors in there covering the cases I could think of, including backward-compat with records that predate the field, empty-array semantics, and whether scope array ordering is normative (it is). There's also a chain vector proving the attribution survives across hash links. JS and Python verifiers both pass.

I went with one record plus attributed parties rather than splitting into two separate assertions. At 11 to 1, the correlation overhead of matching two records doesn't buy verifiers anything they can't already read from the attribution metadata. That changes if the engine eventually gets its own signing key and someone builds a workflow that validates engine claims independently. Which circles back to your differential testing idea as the thing that would actually justify the split.

Three days from the idea to test vectors, and two passing verifiers move faster than the spec conversation will, which is the useful part.

You are right on one record rather than two, and 11 to 1 is a better argument than the one I made. I withdraw the split. Correlating two records buys a verifier nothing that the scope arrays do not already tell it.

Worth reading that ratio as a finding rather than only as a reason to skip the split. Eleven fields have an independent witness. The one that does not is the only field that says whether the call was permitted. So the attribution metadata is not just improving provenance; it is making visible that the whole authorization claim in the record rests on a single self-reported value. That is the FerrumDeck bug stated structurally rather than anecdotally: every byte on the wire was correct, the one claim sitting on top of it was wrong, and a single signature covered both equally. A verifier reading your parties array would have known which half to distrust without knowing anything about my code.

One thing I would check before this goes further, and I am holding myself to something I said upthread. I warned that a third anchor which does not land inside cyclonedx-evidence-2.0 is a parallel mechanism, and that parallel mechanisms in evidence schemas tend not to survive. Your parties array may be one. The question is whether your shape maps onto partyAttribution and assertion.parties from #980 or merely resembles them, and the specific place it will bite is granularity: your attribution is per-field via scope, and if CycloneDX attributes at the assertion level only, then either the scope has to move up into a per-scope assertion or someone has to argue the field-level case into the spec. Cheaper to answer that now than after two implementations have shipped incompatible shapes.

I will do that mapping for the FerrumDeck decision record against evidence-2.0 and post it. I offered it upthread, and nobody needed it then. Yours is the second implementation, and two is when the question stops being hypothetical.

On differential testing, agreed that an independent engine key is what justifies the split. The cheaper version needs neither: same engine, two versions, byte-identical input frame, alert on disagreement. No oracle and no second signer. It tells you nothing about which one is right, which is the real limit. I have not built it, so take that as a proposal rather than a result.

I ran the field-granularity mapping discussed above. The result is narrower, and better, than my initial framing: at the frozen 2.0-dev commit, the distinction is expressible with native CycloneDX constructs and no new evidence property.

Fixture and verifier: https://github.com/Silentpartnercoding/agent-trust-benchmark/blob/82b497e0c642ab1460b2f904c08768e2f263a30f/docs/CDX-AADP-INPUT-PROVENANCE-001.md

Result: EXACT.

The pair holds the tool digest, policy digest, evaluated organization = org-b value, permit verdict, behavior instances, and complete non-provenance projection constant. Both records have valid whole-record Ed25519 JSS signatures. One record sources the exact formulation parameter from authenticated authority state; the other sources it from the caller.

The native mapping is:

formulation task input = decision input field
input source.ref = structural source
source component party = supplying party
citation JSON Pointer = exact field scope
citation attributedTo + process = party and decision-process binding
blueprint behavior instance = observed authorization/action behavior

The verifier returns DERIVABLE when the field citation is removed, AMBIGUOUS when only process attribution remains, and UNREPRESENTED when both source and field attribution disappear. Both JSON records validate with the CycloneDX AJV 2020 setup against commit 1a950b106df221c30cf208b4ffad3e5e1303385f; the upstream schema hashes are frozen in the fixture.

I used AADP-01 only as an independent per-action authorization profile because it explicitly separates self-asserted request source from authenticated authorization context. This is not a claim of AADP adoption or coordination, and the fixture is not a complete AADP wire transcript.

The remaining semantic question for CycloneDX is whether citations are intended to attribute runtime formulation input values this way. If yes, this looks like a compatibility fixture rather than a schema gap. If not, the fixture should expose the smallest mismatch.

Update 2026-08-27: after review by the AADP author, the fixture now distinguishes AADP-01’s explicit re-derivability requirement from the case-derived need for field-level provenance. AADP-01’s minimum evidence entry does not itself enumerate per-field provenance. The result remains EXACT.

feat(attestation): add witness projection for role/party-scoped read …

Navigation Menu

Repository navigation

Behavior evidence gap: linking agent actions back to BOM components #1016

Context

The gap: behavior evidence is not addressable in the current design

What we've built (implementation experience)

Concrete proposal for consideration

Why this matters for AI agents specifically

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Behavior evidence gap: linking agent actions back to BOM components #1016

Description

Context

The gap: behavior evidence is not addressable in the current design

What we've built (implementation experience)

Concrete proposal for consideration

Why this matters for AI agents specifically

Activity

elang2 commented on Aug 19, 2026

Silentpartnercoding commented on Aug 19, 2026

elang2 commented on Aug 20, 2026

sattyamjjain commented on Aug 21, 2026

elang2 commented on Aug 21, 2026

sattyamjjain commented on Aug 22, 2026

elang2 commented on Aug 22, 2026

sattyamjjain commented on Aug 25, 2026

Silentpartnercoding commented on Aug 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions