Skip to content

[FEATURE] Credential headers on Agent egress entries #3093

Open

Description

@thomasbarrett

Problem

#3019 lets an Agent reach extra origins, but not authenticate to them. kagent binds gateway credentials only for ModelConfig keys and RemoteMCPServer headers.

Proposal

Make Agent.spec.egress entries objects, with optional headers the egress gateway sets. The runtime sends each header with a placeholder and never holds the value.

spec:
  egress:
    - origin: https://proxy.golang.org        # reach only, as in #3019

    # A Secret's value
    - origin: https://api.internal.example
      headers:
        - name: Authorization
          prefix: "Bearer "
          valueFrom:
            secretKeyRef: {name: internal-api, key: token}
          # OAuth 2.0 client credentials
          valueFrom:
            oauth2:
              tokenURL: https://auth.example/oauth/token
              audiences: [https://api.internal.example]
              scopes: [read]
              clientAuth:
                method: clientSecretPost
                secretRef: {name: internal-api-client}   # client-id, client-secret
          # On-Behalf-Of: the caller's token exchanged for this API (RFC 8693)
          valueFrom:
            oauth2:
              tokenURL: https://auth.example/oauth/token
              audiences: [https://api.internal.example]
              clientAuth:
                method: clientSecretPost
                secretRef: {name: internal-api-client}
              tokenExchange:
                subject: Caller
  • headers only on exact https hosts, where Substrate injects.
  • valueFrom is a union, so new sources don't change existing Agents.
  • A missing Secret is reported on ResolvedRefs.

secretKeyRef works with Substrate as is. oauth2 needs a Substrate credential provider that runs the grant and caches tokens (as kagent-dev/substrate#47 does for Google). On-Behalf-Of also needs kagent to pass each turn's caller token to that provider.

Alternative: a separate EgressEndpoint resource

Define the origin and its headers once, in their own resource, and have Agents reference it by name, the way they reference a ModelConfig:

kind: EgressEndpoint
metadata: {name: internal-api, namespace: team-a}
spec:
  origin: https://api.internal.example
  headers:
    - name: Authorization
      prefix: "Bearer "
      valueFrom:
        oauth2: {...}               # same sources as above
---
kind: Agent
spec:
  egress: [https://proxy.golang.org]   # #3019, unchanged
  egressEndpointRefs:
    - name: internal-api               # its origin is opened automatically

This is safer for a platform. With headers on the Agent, anyone who can write an Agent can bind any Secret in the namespace to any host they list. With a separate resource:

  • Platform owners decide where each credential may go, and with what scopes. Agent authors can only pick from those endpoints.
  • Kubernetes RBAC can split the two: create on EgressEndpoint for the platform, on Agent for users. kagent's API and UI would manage Agents but not endpoints, so the split holds even though the API authorizes no one.
  • One endpoint serves many Agents, and changes to where credentials go are reviewed in one place.
  • feat(api): Agent.spec.egress — HTTP(S) origins an Agent may reach #3019's egress stays a list of strings.

Later, an endpoint could also limit which Agents may reference it (for example by label), like Gateway API's ReferenceGrant.

Activity

  1. changed the title [-][FEATURE] Secret-backed headers on Agent egress entries[/-] [+][FEATURE] Credential headers on Agent egress entries: Secrets, OAuth client credentials, On-Behalf-Of[/+] on Oct 8, 2026
  2. changed the title [-][FEATURE] Credential headers on Agent egress entries: Secrets, OAuth client credentials, On-Behalf-Of[/-] [+][FEATURE] Credential headers on Agent egress entries[/+] on Oct 8, 2026
  3. added 4 commits that reference this issue on Oct 8, 2026
    499cdf0
    dfbeeb8
    500058d
    5ff4c12
  4. fiunchinho commented on Oct 8, 2026

    @fiunchinho

    +1 for the EgressEndpoint alternative.

    • It's the shape kagent already uses for credentials. ModelConfig and RemoteMCPServer each hold a destination and its credentials, and an Agent references them by name. An EgressEndpoint would be the same kind of resource for any other HTTPS destination.
    • Agent.spec.egress from feat(api): Agent.spec.egress — HTTP(S) origins an Agent may reach #3019 stays a list of strings for hosts that need no credentials.
    • The runtime side mostly exists. The compiler already turns ModelConfig keys and Secret-backed RemoteMCPServer headers into destination-scoped bindings that Substrate's gateway injects (docs/architecture/credential-injection.md). A first version with secretKeyRef headers could reuse that path, and oauth2 could follow once Substrate has a provider for it.

    A few points I think a first version should settle:

    • Conflicts. Substrate rejects different credentials for the same hostname and header. An endpoint that conflicts with a ModelConfig, a RemoteMCPServer or another endpoint on the same Agent should fail compilation with a clear error.
    • Namespaces. Same namespace as the Agent to start. Cross-namespace use could come later through allowedNamespaces, as RemoteMCPServer does.
    • Status. A missing Secret on ResolvedRefs of the endpoint, and an unresolved egressEndpointRefs entry on the Agent's conditions.

    @thomasbarrett are you planning to implement this? If not, I'm happy to pick it up once #3019 lands. It would be good to hear from the maintainers whether a new CRD is the direction they want before either of us starts.

  5. thomasbarrett commented on Oct 8, 2026

    @thomasbarrett
    ContributorAuthor

    I was planning on implementing this on Fluidstack internal fork to test on some of our agent workloads and contributing upstream, but would love help @fiunchinho . I'm going to pull up to community meeting next week to discuss

  6. thomasbarrett commented on Oct 8, 2026

    @thomasbarrett
    ContributorAuthor

    Another consideration is that we likely want authenticated egress from Sandbox actor as well so that the agents can upload and download files to S3, internal / external services etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions