feat: add reusable egress policies for agents and sandboxes

[EItanya]

This PR adds reusable EgressPolicy resources for HTTP(S) access beyond an agent's inferred model, MCP, and artifact access. Agent and SandboxTemplate select policies through spec.egressPolicyRefs in the same namespace. Authentication uses a header, optional literal prefix, and Secret key reference:

authentication:
  header: Authorization
  prefix: "Bearer "
  secretKeyRef:
    name: ticket-auth
    key: token

The elected Agent and SandboxTemplate reconcilers resolve policies from cached Kubernetes inputs and persist grants on the source definition. Effective policy merges those grants with a revision's pinned integration access when read. Create, resume, and checkpoint forks use that persisted state without translating Kubernetes inputs. Explicit grants stay outside the revision hash.

Every preparation applies persisted policy to the source's running and suspended actors across retained revisions, with at most 16 updates in parallel. Unchanged policies skip the Substrate write. Failures retry through the existing preparation queue, and startup replays every source. Slow actor updates therefore delay that preparation queue. Source finalizers preserve deletion through controller downtime and remain until actor revocation succeeds. Substrate version preconditions order competing applications, and creation rechecks policy after applying it so concurrent updates are not missed. Convergence remains asynchronous.

New instances can use the latest successful runtime revision while edits are pending or failed. Grants continue updating even when a Harness, template, or WorkerPool cannot be resolved. Missing or invalid policies revoke explicit grants while preserving pinned integration access. Recreating a source does not transfer grants to its old instances. Conflicts with pinned credentials, token passthrough origins, or the merged rule limit withhold explicit grants and emit warnings.

Credential injection requires an exact HTTPS origin, and Secret values stay out of runtime revisions. Includes CRDs, clients, Helm packaging, and API and lifecycle documentation.

E2E coverage uses the same OCI skill and shell probe through kagent, Codex, Claude, and BYO Go ADK. It checks live grant/revocation, creation from the last successful revision, resume, and checkpoint forks. A separate HTTPS scenario checks actual Secret header replacement, with and without a prefix. Standalone sandboxes have lifecycle coverage through their process API.

Validation: the full Go test suite with the race detector, PostgreSQL store and migration tests (including inline SQL preparation), and Go lint passed locally. Runtime E2E validation for this refactor is pending CI; the available local cluster uses the old API.

[fiunchinho]

This applies now to the RemoteMCPServer.spec.headersFrom[].name field. Maybe we could add the CEL rule to validate this into the CRD. WDYT?

BTW, this breaks existing RemoteMCPServer objects that use any of the not allowed headers, so let's make sure this is released as a breaking change (PR title, PR description, etc).

[codyhartsook]

🤖 AI-generated review.

[ernoaapa]

Thanks, this is what we need for per-agent egress. Would you accept a bare * as an origin, for allow-all? Substrate already supports * hostnames, and some of our agents (developers building code with many package registries) need open egress. Exact-origin rules with credentials would still win for their hosts. Happy to send a follow-up PR if that fits.

[EItanya]

Thanks, this is what we need for per-agent egress. Would you accept a bare * as an origin, for allow-all? Substrate already supports * hostnames, and some of our agents (developers building code with many package registries) need open egress. Exact-origin rules with credentials would still win for their hosts. Happy to send a follow-up PR if that fits.

Makes sense, let's do a follow-up