Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Hey, thanks for working on this. The separation between correlation context and authorization identity is important. I've been testing Keel as an external pre-execution authorization service through OpenShell's Supervisor Middleware, and this connects directly to an issue we encountered during our v0.1.3 qualification. In that experiment, the adapter could associate I see this PR introduces For that next integration step, would the intended approach be to propagate the supervisor-generated That would let an external authorization provider join:
It would also be useful to preserve that identifier across request and response evaluation, including denials and middleware failures. I'm not suggesting that correlation alone proves exact-request binding or downstream effects. Those remain separate guarantees. Happy to help validate the eventual runtime integration against our existing sanitized test cases if helpful. |
|
Written by Claude Code, reviewed by @krishicks
A question about span volume. Every inspected request gets We see supervisor traces as a tool for latency and errors, sampled like any other service, rather than a full record of every request. OCSF already provides that record, and request rates belong in metrics. Sampling for supervisors is tracked in #5362, and supervisor metrics in #5364. In the meantime, we'd like these spans not to assume they're always exported. In particular, could you confirm that the request ID join through |
Compatibility review (advisory)New or updated breaking changes; see the full report for all findings. 1. api-behavior — No explicit risk acceptance foundSuccessful HTTP FORWARD requests without an L7 route now emit a fallback HTTP Activity record instead of the existing Allowed/Success record with the requested path and process.
2. api-behavior — No explicit risk acceptance foundx-openshell-request-id previously passed through HTTP forwarding and allowed middleware mutations; the candidate strips incoming values, replaces upstream values and rejects mutations.
3. api-behavior — No explicit risk acceptance foundInspected HTTP denials replace bounded supervisor.egress.deny spans with unbounded openshell.http.request and policy spans that can inherit workload traceparent.
Review details
Advisory review; publication gates are unchanged. |
krishicks
left a comment
There was a problem hiding this comment.
Written by Claude Code, reviewed by @krishicks
Reviewed at 7e8f7ae. Each point starts with what a user would see; the cause and file references follow. Line numbers refer to that commit.
Audit records
- A SIEM shows requests the supervisor blocked as allowed. A request blocked by a middleware failure (403), admission exhaustion (503), a WebSocket preflight error (502), or a credential preparation failure (502) gets one HTTP Activity record that says
Allowed, with no failure status.- Cause:
emit_primaryreclassifies the record only when the outcome isdenied(l7/correlation.rs:212-220). Theplatform_errorandupstream_erroroutcomes keep the allow record (relay.rs:900-904,relay.rs:1823-1855).
- Cause:
- Allowed plain-HTTP proxy requests no longer say which process made them. On main, every allowed FORWARD request without L7 inspection is recorded with its path, binary, PID, and cmdline. With this PR, it gets a placeholder record: "HTTP request ended before a response was selected", URL
/, no action, no process.- Cause: the correlation is created up front (
proxy.rs:5921-5926), so the empty slot is emitted at the response head. The real allow record arrives after the relay (proxy.rs:7228) and is dropped (correlation.rs:174).
- Cause: the correlation is created up front (
- Requests rejected before policy runs are reported as policy denials. An
Upgraderefusal, a malformed request target, or an invalid GraphQL or JSON-RPC body produces "HTTP request blocked after policy admission", URL/, and a span withpolicy.decision=deny. Policy never ran on these requests, so an operator investigating policy is sent to the wrong place.- Cause: these paths emit only a Network Activity record, so the placeholder record is emitted and relabelled as a denial (
relay.rs:1310-1335,relay.rs:1550-1665,relay.rs:3344-3348).
- Cause: these paths emit only a Network Activity record, so the placeholder record is emitted and relabelled as a denial (
- Some requests still produce extra HTTP Activity records that can't be joined by request ID, and one produces two records that disagree. This happens to any request with response middleware, a request-middleware denial, or a token-grant refusal. A token-grant refusal gives one record saying Denied and another saying Allowed with a 502. Denials on these paths also open a separate
supervisor.egress.denyspan next to the request span.- Cause: these paths still emit through
telemetry::emit_decisionwithoutcorrelation_uid(http_response.rs:698-1251,middleware.rs:1023,token_grant_injection.rs:344). The request ID is already available at the call sites (relay.rs:1810,relay.rs:2655,relay.rs:3138).
- Cause: these paths still emit through
- When credential resolution or MCP reinspection rejects a request, the record no longer shows which rule matched. The surviving record loses the request path, policy rule, and reason. A problem after the response headers are sent produces no correlated record at all.
- Cause:
emit()replaces the primary record with any later HTTP Activity, and silently drops one that arrives after the head is written (correlation.rs:166-176).
- Cause:
Trust and sampling
- A sandboxed workload decides which trace ID appears on supervisor audit records. If it sends a sampled
traceparent, the HTTP Activity record'strace.uidis the workload's trace ID. A workload could point its denials at an unrelated trace. Once #5362 adds a sampling ratio, it could also force its request spans to be exported regardless of that ratio.- Cause: a sampled workload context becomes the parent of
openshell.http.request(propagation.rs:169-174). #4789 documents that workload headers never supply supervisor correlation, and #2640 keeps workload context in a separate field. - Linking sampled context the way unsampled context is already linked would keep the connection to the agent's trace without either effect.
- Cause: a sampled workload context becomes the parent of
Behavior changes beyond correlation
These change what clients see. Could they move to a separate PR, or at least be listed in the PR description?
- Agent HTTP clients stop retrying, so a routine connection close becomes a failed request. When an upstream closes an idle keep-alive connection, main closes the client connection. Go, urllib3, and curl retry that automatically. With this PR the client gets a 502 and surfaces the error.
- Cause: a synthetic 502 on upstream EOF before headers (
http_response.rs:100-110).
- Cause: a synthetic 502 on upstream EOF before headers (
- Some upstream APIs that work today stop working. For example, a server that sends
HTTP/1.1 200without a reason phrase now has its response rejected, and the agent sees the connection close. The audit record still reports a 502.- Cause: new strict response-head validation that applies whether or not a request is correlated (
http_response.rs:425-440).httparseaccepts a missing reason phrase.
- Cause: new strict response-head validation that applies whether or not a request is correlated (
- Agents get a 500 response where the connection used to close. This happens on a missing credential or a stale credential generation.
- Cause:
relay.rs:1064-1080,proxy.rs:6851-6880.
- Cause:
Spans
- Traces don't show where an inspected request's time went. Only policy evaluation appears under
openshell.http.request. Middleware calls, credential preparation, and the upstream dial are not children of the request span, and remote middleware receives the connection's trace context instead of the request's. - WebSocket requests show up as a single span lasting the whole session. The root span stays open until the upgraded connection closes.
- The span marks an error but not which stage failed.
record_error_stageignores its stage argument (correlation.rs:162).
Tests
Most of the record problems above would have been caught by a test asserting exactly one HTTP Activity per request. The dispatch tests use .find() instead. These paths have no coverage:
- placeholder records and rejections before policy runs
- the content of a non-L7 FORWARD record
- token-grant refusal
- request-middleware denial
- upstream EOF before headers
- a second request on a keep-alive connection, which should get a new ID
Process
The branch conflicts with main.
|
Written by Claude Code, reviewed by @krishicks
A follow-up connecting the review with my earlier comment on span volume. The compatibility review's item 3 raises the same concern: span volume. Review point 10 asks for a child span per stage, which adds spans. Nothing bounds that until #5362 adds supervisor sampling, and the #4789 deny-span limiter isn't the right tool, since #5362 removes it. Could the request spans land with or after #5362? This PR would keep the request ID, |
|
Drafted with Codex; reviewed and approved by @HarryMoss. Hi @afourniernv, I’ve read Kris’s latest review and the proposed split between the correlation core and request spans. My local #2640 connection-startup prototype is based on #5214 at Once you’ve settled the split, which revision should I track for those shared paths so I can avoid duplicating your fixes? Context: #2640 (comment) |
|
Drafted by 🤖, signed off by @krishicks A correction to my last comment: the request spans don't need to wait for #5362. They can stay in this PR, along with per-stage children (review point 10). The concern is spans that exist only because a denial happened. A request span that records every outcome is the right shape. Two requirements apply whether or not sampling exists:
The updated diagram:
|
|
Drafted by Codex; @HarryMoss reviewed Thanks @afourniernv for the updates. I reran the two regressions from my earlier review on
I have the small test-only patch ready to share. This is local macOS evidence, rather than a fresh deployed validation. The compatibility advisory also matches what I see in the successful middleware merge path: routine invocation details are dropped rather than retained on the primary record. Would you prefer retaining those audit details, or documenting that loss as an explicit compatibility decision? That question follows Kris's guidance that audit records and joins work independently of trace export. |
Compatibility review (advisory)New or updated breaking changes; see the full report for all findings. 1. api-behavior — No explicit risk acceptance foundSuccessful token grants previously emitted one HTTP Activity record per credential; inspected requests now merge these records into the existing request decision without retaining their messages.
Review details
Advisory review; publication gates are unchanged. |
3f46a50 to
11030e5
Compare
Closes NVIDIA#5393 Refs NVIDIA#5210 Signed-off-by: Alex Fournier <afournier@nvidia.com>
11030e5 to
0f92636
Compare
Signed-off-by: Alex Fournier <afournier@nvidia.com>
@afourniernv I'd like to propose a simpler shape for this PR: keep one HTTP Activity record per control rather than merging them into one primary record per request. This changes my earlier position on #5210. Merging is where the bugs are. Re-reviewing d0bdbe9, most of what remains comes from merging records into one primary: a mid-body truncation recorded as a clean allow, timeouts and transport errors recorded as Denied, granted credential identities lost, and later problems dropped after the first. The compatibility review's findings come from the same place. Each fix so far has added more seed, merge and overlay rules, and every new stage would need its own. We may prefer the granularity anyway. With a record per control, the records for a request show its path through enforcement: policy Allowed under Proposed scope:
That still delivers #5393: from a response, an agent or operator can find every record about the request, with tracing on or off. It also removes the merge machinery, the record-replacement compatibility concerns, and the new response-head validation. What it gives up is a single record carrying both the decision and the status the client received. If we find we need that later, we can agree an outcome classification table first and add it as a separate step. Here's how the features compare: |
Correlate original control records at emission with a request-scoped ID and sequence. Remove primary-record aggregation and broad response-head validation while retaining protected receipts and request tracing. Signed-off-by: Alex Fournier <afournier@nvidia.com>
|
@krishicks Agreed. Updated in 7778e71. Records now stay separate and keep their original fields and timing. The shared request ID and sequence connect them. I removed the record-merging code, fallback summaries and general response-header validation, and kept the request/stage spans. Network and OCSF tests, Clippy and pre-commit pass. The full suite hit the existing VM cleanup failure; its isolated rerun passed. Could you take another look? |
Re-reviewed at 7778e71 against the split proposal. This is the shape we asked for: one record per control with main's content, timing and classification, the request ID stamped at emission, and no new response-head validation. That resolves most of the d0bdbe9 items:
What remains, each starting with what a user would see: Before merge
Worth fixing, not blocking
Tests: the keep-alive test filters to HTTP Activity (
Description and docs:
|
Avoid parsing queries solely for correlation. Carry request attribution through terminal errors and policy reloads, and clear HTTP scope at WebSocket upgrade. Preserve per-control audit records while reporting bounded stage outcomes and filtering receipt trailers consistently. Remove JSON receipt additions and the unused correlation setter. Signed-off-by: Alex Fournier <afournier@nvidia.com>
|
@krishicks Updated in 54a11a1. Removed the extra query parse, kept request attribution through errors and policy reloads, and cleared the HTTP scope at WebSocket upgrade. Also fixed event/stage reporting and receipt trailers, and removed the unused setter and JSON receipt fields. Audit records still keep their original fields and timing. Pre-commit and the full test command pass. The final rebuilt network suite passes 1,710 tests, and eight live Hermes/Relay runs passed, including correlation without supervisor trace export. The explicit-forward query regression, WebSocket/trailer edges and reload boundaries have Rust coverage. The separate transparent-query behavior is unchanged. Ready for another look. |
…ures Signed-off-by: Alex Fournier <afournier@nvidia.com>
|
@krishicks Updated in c871af5. Fixed the four Nextest failures by initializing the emission hook in the direct-response fixtures. Also finished stage status/reason handling, removed audit-derived trace state and redundant helpers, and added the keep-alive parse-rejection regression. The full workspace Nextest rerun passes 9,150 tests with four workers. The independent network suite, five additional test suites, Clippy/pre-commit and docs also pass. The initial unrestricted run hit two timeouts in unchanged VM tests; both passed alone and in the full rerun. CI is waiting for the mirror to advance from 54a11a1. Could you run |
Re-reviewed at c871af5 against the previous review. All three blocking items are fixed:
Stage tags, stage span status and reasons, receipt trailers, JSON bodies and the cleanup items are addressed too. Nothing here blocks merge. The remaining items, each starting with what a user would see:
|
|
/ok to test c871af5 |
Signed-off-by: Alex Fournier <afournier@nvidia.com>
|
@krishicks Fixed the handled-failure reporting gap in 57ec95c. Request and response middleware failures now mark the middleware stage as ERROR even when a local error response was delivered successfully. Valid policy refusals stay OK. HTTP responses and original OCSF records are unchanged. The full workspace Nextest run passes 9,152 tests; the final network run passes 1,718. The additional suites and Clippy/pre-commit also pass. Request fixtures now pin the injected runner correctly and distinguish actual mixed-hook failures, missing services and valid denials. Could you run |
Signed-off-by: Alex Fournier <afournier@nvidia.com>
|
@krishicks Addressed the remaining follow-ups in d6784dd: shared host/sandbox proxy startup, forward terminal-error attribution, the 101 handoff receipt, consistent middleware failure reasons and field documentation. The new regression goes through the actual FORWARD, plaintext CONNECT and TLS CONNECT error emitters. WebSocket tests verify the handshake join, cancellation and frame isolation. One clarification: frame denials are Network Activity records; that path does not create HTTP denial spans today. I preserved that behavior rather than adding new spans. Full Could you run |
|
/ok to test d6784dd |
|
Thanks @krishicks and @afourniernv — good to see this merged. Following the final rework, I built merged commit 8a20212 and ran a local Linux ARM64 qualification using the production standalone network-proxy role, Collector 0.164.0 and Jaeger 2.22.0. The final run passed 21 wire scenarios and 195 assertions. All 49 retained OCSF records passed schema validation, and all 12 captured request traces were retrieved from Jaeger, including the spans referenced by the native OTLP log envelopes. Logs were retained through the Collector’s file sink. The controls covered allowed and denied HTTP requests, HTTP/Finding companions for an authority mismatch, truncated upstream responses, untrusted workload trace context and standalone CONNECT refusals. With trace export disabled, enforcement, request receipts and per-control audit joins remained intact. A brief Collector outage also preserved the wire behaviour; this wasn’t a sustained backpressure or delivery-durability test. Separately, two diagnostic gaps reproduced in test-only probes on d6784dd on both macOS ARM64 and Linux ARM64. The relevant code is unchanged in the merge:
For #2640, I propose adapting onto the merged foundation, preserving HTTP-aware terminal-error attribution and ending CONNECT/transparent startup ownership after the startup audit, before the long-lived relay. This qualifies the local standalone proxy pipeline, not a full managed sandbox, Tempo or real-provider workflow. The #2640 adaptation hasn’t been tested yet. |

Summary
Connect inspected HTTP responses, supervisor traces and OCSF records using the OpenShell-generated request ID. Keep each control's original audit record instead of combining records into a final summary.
Related Issue
Closes #5393. Tracking: #5210. Supervisor sampling remains separate in #5362.
Changes
X-OpenShell-Request-IDheader without adding receipt fields to JSON error bodies. Strip upstream receipt headers and trailers consistently; middleware cannot change the receipt.The public Rust addition is an emission enrichment hook. No gateway RPC, Protobuf field, configuration option or Relay dependency is added.
Release Notes
Receipts are join keys, not authorization or proof of execution or complete delivery. Records and request ID joins work without exported spans. L4 and uninspected traffic do not provide the protected-receipt guarantee.
Inspected request spans include allowed traffic at INFO level and are outside the denial-span budgets. Query and collector-capacity guidance is documented; this PR does not add supervisor sampling.
Testing
mise run testpassed on the final source, including Rust workspace tests, gateway tests with test-support, doctests, Python, TypeScript and tooling tests, with four Rust test threads.mise run pre-commitpassed, including workspace, feature and external-manifest strict Clippy; the commit hook passed again.Earlier live qualification on
54a11a15dpassed eight genuine Hermes/Relay conversations: REST/MCP allow/deny, request/response redaction, response withholding, two-turn recovery, interrupted delivery, authenticated read-only GitHub access, and correlation without supervisor trace export. Verified 207 agent spans, 34 receipts and 308 original per-control records. The current follow-up is covered by the rebuilt Nextest suites above, not a new live qualification.The explicit-forward query fix has Rust coverage. A separate transparent malformed-query probe disconnected before receipt allocation and is not covered by that fix. WebSocket/trailer and reload boundary cases have Rust coverage, not live Hermes coverage.
The first full test attempt exhausted host disk space in unchanged Vault tests. Clearing the previously approved regenerable compiler cache allowed the unchanged full command to pass.
Checklist