fix(mcp): a well-formed upstream result is never delivery_uncertain

CHANGELOG.md

@@ -7,6 +7,186 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased] — planned v1.3.0
 
+### Fixed — a well-formed MCP result is never `delivery_uncertain`
+
+- The MCP SDK's `ClientSession.call_tool` validates a result against the
+  upstream's **live** `outputSchema` after the request is answered, and sends a
+  second `tools/list` to fetch it. Because that runs past the dispatch
+  boundary, a schema that had drifted from the result, a failing post-call
+  `tools/list`, or a schema demanding structured content the result lacked
+  raised an exception that discarded the received result. The adapter then
+  signed `delivery_uncertain` for a call that had completed: the debit stayed,
+  the call was never redispatched, the receipt misstated the outcome, and a
+  hostile or drifting upstream could force it (and, with the circuit breaker
+  enabled, count it toward opening the breaker).
+- The adapter now seeds the SDK's per-session schema cache before the dispatch
+  checkpoint (`_bypass_sdk_result_validation`), so the SDK sends no
+  post-dispatch `tools/list` and runs no schema validation. A received result
+  is judged only by the adapter's own shape, size, and secret checks. This also
+  removes a post-dispatch network round trip from every call, and the
+  `$ref`-resolution surface (GHSA-rwrf) from the call path on any SDK version.
+- **Behavior change:** a well-formed result whose `structuredContent` does not
+  match the upstream's advertised output schema is now relayed and charged as a
+  `success` instead of ending in a `504 / -32005` receipt. The gateway still
+  does not validate results against any output schema; drift detection against
+  the startup-pinned schema is not built.
+- Unchanged: a call that yields no result (connection lost, timeout, JSON-RPC
+  error answering `tools/call`) is still `delivery_uncertain`; a tool-level
+  `isError` result is still a refunded `returned_error`; oversized or
+  bearer-echoing results are still `response_rejected`. No new outcome, no
+  schema or API change.
+- The bypass uses a private SDK attribute. If an `mcp` release drops it, the
+  call fails closed before the dispatch claim (generic pre-dispatch failure,
+  logged as `upstream_mcp_sdk_result_validation_not_disabled`) instead of
+  silently reverting to the misclassification.
+- The strict expected-failure test
+  `test_live_output_schema_drift_must_not_turn_a_completed_call_uncertain` is
+  now a plain passing test, with negative-path coverage (drifted schema,
+  failing post-call `tools/list`, no structured content, hostile external
+  `$ref`, unchanged normal success, the production `_session` over HTTP) and a
+  receipt-level test through the governed router. See
+  [`docs/mcp-security-posture.md`](docs/mcp-security-posture.md) §3.2 and
+  [`docs/failure-semantics.md`](docs/failure-semantics.md).
+
+### Fixed — an upstream response the SDK cannot parse is `response_rejected`
+
+- A `tools/call` response the upstream sent but whose envelope the MCP SDK
+  cannot parse (for example `content` that is a string, or a non-boolean
+  `isError`) ended as `delivery_uncertain` (504 / `-32005`) through a real SDK
+  session, although `_call_tool_once` intended `response_rejected`. The SDK's
+  anyio task group rewraps anything raised inside the session block in an
+  `ExceptionGroup`, so the adapter's `ValidationError` branch never matched and
+  the generic handler classified the call by `dispatch_started` alone. Tests
+  that fake the session do not enter the task group, which is why they passed.
+- `_call_tool_once` now records the rejection where the error is raised (the
+  `call_tool` await), before the session unwinds, and only for a
+  `ValidationError` of the `CallToolResult` model, which the SDK raises only
+  after the upstream's JSON-RPC result has arrived. The call ends as
+  `response_rejected` with code `upstream_response_invalid` (502 / `-32006`):
+  the debit stays, the call is never redispatched, and a replay returns the same
+  receipt. A validation error for any other model, such as the request the SDK
+  builds before sending (a non-string key in `arguments`: the claim is
+  committed and nothing reaches the upstream), says nothing about a response
+  and is classified as before, as `delivery_uncertain`.
+- Outcome precedence is unchanged where a handler already applied it: claim
+  contention, then a transport guard rejection, then a known received result,
+  then this rejection, then `delivery_uncertain` only when no response was
+  obtained. The outer `ValidationError` / `JSONDecodeError` branch did not apply
+  the "a result is already in hand" rule, so such an error raised only while the
+  session closes could turn a received result into `response_rejected`, or
+  replace a known rejection's code. It now applies it, and applies the same
+  parse-failure rule as the `call_tool` await (`_is_unparseable_response`): after
+  dispatch, a `CallToolResult` validation error or a JSON decode error is
+  `response_rejected`, and any other validation error is `delivery_uncertain`.
+  That branch is reached only by a session that does not wrap exceptions, where
+  the request-build error above would otherwise have been labeled a rejection
+  again.
+- **Behavior change, `POST /mcp` (where enabled):** `response_rejected` is
+  charged like `delivery_uncertain`, so it is now returned like it: a
+  `tools/call` result with `isError: true`, text saying the upstream answered but
+  the answer was unusable and the effect is unknown, the do-not-resend
+  instruction, the receipt, and `_meta["io.agentmiddleware/outcome"]` with
+  `status: "unknown"` and `reason: "response_rejected"`. It was a bare `-32006`,
+  the code that refunded, retry-safe failures (`upstream_returned_error`,
+  `upstream_pre_dispatch_failed`) also use, so a model shown only the message
+  could resend a call whose charge stands and whose tool probably ran. This
+  applies to any `response_rejected` outcome, because `mcp_standard.py` matches
+  on the reason, not the cause: for example an unparseable envelope, an
+  oversized result, a result echoing the bearer token, a non-identity
+  `Content-Encoding` or malformed `Content-Length` after dispatch, a result that
+  cannot be canonicalized, and a result the persistence layer rejects. Those
+  refunded failures stay JSON-RPC errors. `/mcp/messages` and REST are
+  unchanged. The helper is now `_unknown_outcome_tool_result` (was
+  `_delivery_uncertain_tool_result`).
+- **Behavior change:** a malformed envelope is no longer counted toward the
+  optional circuit breaker. The `response_rejected` path never fed it (an
+  oversized or bearer-echoing result did not either); `delivery_uncertain` does.
+- Documented, not changed: with the optional reconcile worker enabled,
+  `response_rejected` is terminal and is never probed or refunded; only
+  `delivery_uncertain` is leased for a provider status probe.
+- Unchanged: a call with no response (connection dropped, timeout, JSON-RPC
+  error answering `tools/call`) is still `delivery_uncertain`; a malformed
+  `initialize` answer is still a pre-dispatch failure. **Not covered:** a body
+  that is not a JSON-RPC message at all (non-JSON text, `"jsonrpc": "1.0"`,
+  neither `result` nor `error`, or an unknown request id). The SDK transport
+  logs the parse error and forwards it into the session, whose default message
+  handler discards it, so the request is never answered and the call ends as
+  `delivery_uncertain` after the call timeout. The SDK's log line quotes the
+  start of the body; that is unchanged.
+- New tests drive a real SDK session behind the real adapter, in memory and over
+  the production Streamable HTTP client: four envelope shapes, a fixture guard
+  (the envelope must reach the client and fail there), the no-response
+  boundary, a request that cannot be built stays `delivery_uncertain` (also with
+  a session that does not wrap exceptions), a malformed `initialize` stays
+  pre-dispatch, a secret-echo check, the gateway outcome recorded on `/mcp`
+  (`failed` for a rejected response, `unknown` for an undelivered one), and
+  governed-router tests on `/mcp/messages` (one debit, one dispatch, a
+  `response_rejected` receipt, no redispatch on replay, circuit breaker closed,
+  against a JSON-RPC error that opens it) and on `/mcp`. See
+  [`docs/mcp-security-posture.md`](docs/mcp-security-posture.md) §3.2 and
+  [`docs/failure-semantics.md`](docs/failure-semantics.md).
+
+### Changed — documentation re-pinned to the current code
+
+- **Supported endpoint decided.** `POST /mcp/messages` and the REST invoke route
+  are the live governed path in the supported pilot. `POST /mcp` ships off
+  (`ENABLE_STANDARD_MCP_ENDPOINT` defaults to false) and the production runbook
+  keeps it off, because it mints a permit and generates an idempotency key on the
+  caller's behalf. `README.md` and `SECURITY_LIMITATIONS.md` no longer call it the
+  supported or preferred path today; `docs/mcp-security-posture.md` records the
+  decision instead of describing a contradiction; `docs/gateway-overhead.md` no
+  longer calls `/mcp/messages` "not the supported endpoint".
+- `file:line` citations in `docs/mcp-security-posture.md` and
+  `docs/gateway-overhead.md` were written against a base more than a thousand
+  commits older. Of 17, 15 had moved (by 1 to about 970 lines), one SDK range
+  was a line short, and one (`mcp.py:15`) was still right. Each is re-pinned and
+  the claim beside it re-read. Claims that no longer held, corrected: the `idempotency_key_required`
+  refusal applies to a call that names a permit and sends no key (a governed
+  call that names no permit is refused sooner, as `permit_required`,
+  `action_permit_required` or `Tool not found`, and an action-bound call gets a
+  server-derived `act1-` key and is never refused for lack of one); a signed
+  `denied` receipt is issued only when
+  the permit names the caller's wallet; the SDK validation range is
+  `mcp/client/session.py:412-448`; `docs/exit-and-export.md` said no
+  OpenTelemetry instrumentation exists, but optional tracing
+  (`OTEL_TRACING_ENABLED`, default off) does; and the poisoning-test docstring
+  named one call site of `register_configured_upstream_mcp` where
+  `scripts/failure_lab.py` is a second.
+- `docs/gateway-overhead.md` is re-measured twice on this base, with the load
+  average recorded: first call p50 29.5–30.9 ms and replay p50 5.4–5.6 ms
+  (earlier runs: 29.5–31.9 ms and 4.4–4.7 ms), results in
+  `docs/research/gateway-overhead-2026-10-23/`. The public site's figure is not
+  touched.
+- `TROUBLESHOOTING.md` gains a `response_rejected` receipt section.
+- Further claims corrected after a second review. The posture page said
+  operator permit constraints apply only on `/mcp/messages` and REST; `POST /mcp`
+  honours a permit the caller names (`_meta["io.agentmiddleware/permit_id"]` or
+  `mcpContext.permit_id`), enforcing `forbidden_fields` with a signed `denied`
+  receipt, refuses action-bound tools without one, and mints an unconstrained
+  permit only when the reference is omitted (now pinned by a test). It said there
+  is no KMS and no row-level security; an AWS KMS signing backend
+  (`TRUST_SIGNING_BACKEND=aws_kms`) and PostgreSQL row-level security
+  (`TENANT_RLS_MODE=enforce`) exist as opt-ins, default off. The posture page now
+  notes that the refunded `upstream_returned_error` relays the upstream's result
+  payload (`error.data.upstream_result` on `/mcp/messages` and `/mcp`,
+  `detail.upstream_result` in the HTTP 502 body on REST), and that a keyless
+  `/mcp` call's generated key is readable in the signed receipt as the forwarded
+  key. `docs/exit-and-export.md` no longer says traces are not produced.
+  `docs/gateway-overhead.md` quotes only the first-call p99 and max, and
+  records the kernel release rather than a macOS version the result files do not
+  contain. A comment in `mcp_standard.py` that said a keyless call's key is
+  "never returned" is corrected.
+- Endpoint wording, leftovers. The REST invoke route's OpenAPI description no
+  longer tells every integration to call `POST /mcp`; it points at
+  `/.well-known/agent.json` as the `/mcp/messages` description does, and
+  `docs/openapi.json` is regenerated. The Cursor plugin README and skill say the
+  standard endpoint ships off and the plugin works only against a deployment that
+  enables it. `README.md` names the REST invoke route as supported;
+  `docs/authority-required-flow.md` and two test docstrings no longer present
+  `/mcp` as the surface integrations are told to use. The plugin's
+  `mcp.json`, which targets `/mcp` on the first-party host, is deliberately left
+  as is: that is a product decision.
+
 ### Added — per-upstream concurrency limit and circuit breaker (default off)
 
 - `UPSTREAM_MAX_CONCURRENCY_PER_UPSTREAM` (default `0`, unlimited) caps

README.md

@@ -286,12 +286,17 @@ curl -sS -X POST "$API_URL/mcp/messages" \
   }"
 ```
 
-`POST /mcp/messages` is the legacy JSON-RPC transport — deprecated in the
-OpenAPI contract, fully supported, and the only JSON-RPC surface in a default
-local run. `POST /mcp` is the standards-compliant Streamable HTTP endpoint on
-the official MCP SDK; it is opt-in today (`ENABLE_STANDARD_MCP_ENDPOINT=true`,
-default off) and new integrations should prefer it. Both run the same governed
-permit → meter → receipt path.
+`POST /mcp/messages` and the REST invoke route `POST /mcp/tools/{id}/invoke`
+are what the supported pilot runs on — both marked deprecated in the OpenAPI
+contract, both fully supported, and `/mcp/messages` is the only JSON-RPC
+surface in a default local run. `POST /mcp` is the standards-compliant
+Streamable HTTP endpoint on the official MCP SDK. It ships off
+(`ENABLE_STANDARD_MCP_ENDPOINT`, default false) and the production runbook
+([docs/deploy-railway.md](docs/deploy-railway.md)) requires it to stay off,
+because it mints a short-lived, single-tool permit on the caller's behalf and
+generates an idempotency key when the client sends none. It becomes the
+preferred path for new integrations once an operator enables it. Both run the
+same governed permit → meter → receipt path.
 
 An agent holding no authority can ask a human for it with
 `POST /v1/permit-requests`. Full HTTP sequence:

SECURITY_LIMITATIONS.md

@@ -28,6 +28,13 @@ low-sensitivity data: PHI, PCI data, regulated production records, and
 sensitive tool arguments are out of scope. Shared SaaS, customer-VPC/BYOC, and
 customer-operated on-premises deployments are not supported in this pilot.
 
+For the MCP-specific attack classes (tool poisoning, shadowing, confused
+deputy, token passthrough, session hijack), this deployment's exposure to the
+April 2026 MCP SDK STDIO disclosure, and the SDK advisories that touch the
+transports in use, see [`docs/mcp-security-posture.md`](docs/mcp-security-posture.md).
+For what a customer keeps on exit, see
+[`docs/exit-and-export.md`](docs/exit-and-export.md).
+
 High-risk AWI HTTP routes (`/v1/awi/execute`, passkey, rag index/query,
 `dom/sync`) also require `X-Permit-Id` + `Idempotency-Key`; successful metered
 calls emit receipts. Their current abort paths do not have the durable
@@ -357,11 +364,15 @@ production-like boots refuse):
 - `/v1/dev-keys/self-provision` stays runtime-gated by its own flag and is
   advertised in the schema only when that flag is on (never in production).
 
-The legacy invoke entry points `POST /mcp/messages` and
-`POST /mcp/tools/{id}/invoke` remain mounted for existing clients and the
-local proof scripts but are marked `deprecated` in the spec; the standard
-MCP Streamable HTTP endpoint at `POST /mcp` is the supported path. All entry
-points run the same governed permit→meter→receipt path.
+`POST /mcp/messages` and `POST /mcp/tools/{id}/invoke` are the live governed
+path in the supported pilot, and are marked `deprecated` in the spec. The
+standard MCP Streamable HTTP endpoint at `POST /mcp` ships off
+(`ENABLE_STANDARD_MCP_ENDPOINT` defaults to false) and the production runbook
+([`docs/deploy-railway.md`](docs/deploy-railway.md)) requires it to stay off,
+because it mints a short-lived, single-tool permit on the caller's behalf and
+generates an idempotency key when the client sends none. It becomes the
+preferred path for new integrations once an operator enables it. Every entry
+point that is enabled runs the same governed permit→meter→receipt path.
 
 ## Public Health Reporting
 

TROUBLESHOOTING.md

@@ -104,6 +104,11 @@ The gateway claimed a send but has no trustworthy terminal result. Delivery and
 
 On the standard `/mcp` endpoint this outcome arrives as a `tools/call` result with `isError: true` and `_meta["io.agentmiddleware/outcome"].status` set to `"unknown"`, with the do-not-resend instruction in its text, not as a JSON-RPC `-32005` error. `/mcp/messages` and the REST surface keep `-32005`.
 
+### `response_rejected` receipt
+The upstream answered, but the answer was unusable: its envelope could not be parsed, its shape was invalid, it was over the byte cap, or it reflected the upstream bearer token. The call was dispatched and the charge stands, so the action may have taken effect; the gateway does not redispatch, and replaying the same key returns the same receipt (`502` / `-32006`). Do not retry automatically; check the downstream system first. Unlike `delivery_uncertain`, it is never probed or refunded by the optional reconcile worker. `-32006` is also the code of refunded, retry-safe failures (`upstream_returned_error`, `upstream_pre_dispatch_failed`), so tell them apart by `error.message` and the receipt `outcome`, not by the code.
+
+On the standard `/mcp` endpoint this outcome also arrives as a `tools/call` result with `isError: true`, `_meta["io.agentmiddleware/outcome"].status` set to `"unknown"` (reason `response_rejected`) and the do-not-resend instruction in its text. The refunded failures stay JSON-RPC errors there.
+
 ---
 
 ## Examples fail