Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-4732.docs.buildwithfern.com/openshell |
db063a9 to
8dfbd78
Compare
a876a1a to
437619a
Compare
3da8000 to
5fbece5
Compare
5fbece5 to
1899b5b
Compare
96ba1b7 to
e0b6cec
Compare
b4c1940 to
b733887
Compare
b733887 to
8e6786b
Compare
8e6786b to
6dd2cbb
Compare
6dd2cbb to
85cba03
Compare
a78134c to
03c6d6e
Compare
6b0466e to
16f61f5
Compare
Compatibility review (advisory)New or updated breaking changes; see the full report for all findings. 1. rust-sdk — No explicit risk acceptance foundGetSandboxProviderAccessToken adds a required get_sandbox_provider_access_token method to openshell_sdk::raw::proto::open_shell_server::OpenShell without a default implementation.
2. go-sdk — No explicit risk acceptance foundopenshellv1.OpenShellClient now requires GetSandboxProviderAccessToken in addition to its baseline method set.
3. typescript-sdk — No explicit risk acceptance foundRegeneration makes getSandboxProviderAccessToken mandatory on the injected Client<typeof OpenShell> accepted by SandboxClient and SandboxTemplateClient constructors.
Review details
Advisory review; publication gates are unchanged. |
|
Looks good to me! |
|
Team, I second @EmilienM's review by adding my personal feedback based on testing this PR on OpenShift. The biggest issue I am facing is the required On OpenShift, SPIRE is deployed via the SPIFFE/SPIRE operator (ZTWIM), which does not expose the admin socket outside the agent pod. Even if it did, mounting it via |
|
This pull request has had no activity for 14 days and is now marked stale. |
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Preserve the shared supervisor pool and recovery implementation from main while wiring gateway-brokered sandbox identities into shared hosts. Refresh the public schema inventory, authentication fixtures, and affected test-workspace lockfiles. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
|
@akram Following up on the OpenShift deployment concern: commits 89aad0f and 17c8b7b switch this implementation to the SPIFFE Broker API over mTLS. The gateway obtains its own X.509-SVID through the CSI Workload API socket. It no longer mounts a hostPath or requires a supplemental group for SPIRE access. For sandbox issuance, it supplies the Sandbox CR UID from its persisted compute-runtime identity. SPIRE resolves and attests that object, checks the broker identity allowlist, and enforces scoped Kubernetes authorization through Local validation passed 36 Kubernetes conformance tests and all 8 Broker/provider-token tests, including credential renewal, abrupt supervisor takeover, and stale-session rejection. The Broker suite also passed against the retained Helm dev deployment. The gateway Pod specification passed server-side dry-run admission in a namespace enforcing Kubernetes Restricted Pod Security. OpenShift |
|
Following up on the compatibility advisory for the current head, 17c8b7b: We are keeping The Rust generated server-trait requirement and TypeScript injected-client/mock shape still need reviewer disposition: either mitigation or explicit acceptance of the source break. The current TypeScript SDK CI task passes all 206 tests, but that does not prove compatibility with downstream handwritten mocks or server implementations. The Go finding should be reevaluated against Please reevaluate these findings against the current API/SDK baseline. Keeping the RPC on this service is the proposed API placement; compatibility risk acceptance remains a separate review decision. |
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Preserve the SPIFFE RPC alongside service discovery and update the combined public schema inventory without changing durable fingerprints. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
EmilienM
left a comment
There was a problem hiding this comment.
Another iteration needed: 4 major, 19 minor (inline, 1 below)
The Broker API rework removed the hostPath and node-coupling problems from round 1, and the cache-binding fixes hold up. The main problem now is that the Broker connection pins one SPIRE agent's ID, so the documented Service setup only works on a single-node cluster. The other majors are per-request costs on the shared-supervisor path (catalog scans, the global gRPC rate limiter) and the wire format of the new RPC's credential_key.
Outside the diff:
- Minor:
docs/how-it-works/providers/profiles.mdx:705, still says token grants need the sandbox supervisor's own Workload API socket, as dodocs/kubernetes/access-control.mdx:37anddeploy/helm/openshell/README.md.gotmpl:535; a short note linking to the new Broker section would cover shared supervisors.
How this was reviewed
- Commit
d7b0c95, round 2: a full pass, since the branch was rebased after round 1's9b60d8d - Deep review, because it touches
AGENTS.md,.agents/skills/helm-dev-environment/SKILL.mdandcrates/openshell-core/src/dynamic_credential_key.rs - Passes: architecture, performance, security, scalability, rust, kubernetes and supply-chain, then correctness, then an advanced defensive review
- Independent skeptics checked each candidate finding (3 with a reproducer for blocking and major ones, 2 for architecture and scalability ones, 1 for minor ones); 19 of 23 held up
- Repo rules read from
main: AGENTS.md, CLAUDE.md, CONTRIBUTING.md, docs/CONTRIBUTING.mdx - Not covered: nothing skipped
| .dangerous() | ||
| .with_custom_certificate_verifier(openshell_crypto::tls::spiffe::server_verifier( | ||
| roots, | ||
| config.server_spiffe_id.clone(), |
There was a problem hiding this comment.
Major: the verifier accepts exactly one agent SPIFFE ID, but the documented endpoint (spire-broker.spire.svc) is a Service over the agent DaemonSet, and each k8s_psat agent ID carries its own node UID. On a multi-node cluster most new connections land on another agent and fail the handshake as unavailable, and gateway replicas or a rescheduled gateway can't share one value. The e2e passes because kind has one node (with-kube-gateway.sh:659 pins items[0]). Either accept agents by a trust-domain path prefix or a set, or document and test one supported topology (node-local agent with a per-node ID) and make configuration.mdx, NOTES.txt and the skill row agree.
| .await | ||
| .map_err(|_| token_grant_store_status())? | ||
| .ok_or_else(invalid)?; | ||
| let catalog = state |
There was a problem hiding this comment.
Major: snapshot_catalog lists, normalizes and hashes every platform and workspace profile, and obtain calls grant_configuration twice per request (here and at :319), cache hits included. The shared supervisor calls this RPC for every proxied request, so gateway DB and CPU cost now scales with request rate times profile count. A point lookup of the one profile, or a catalog cached by catalog.revision() with the closing re-check comparing revisions, keeps the fail-closed checks without the scans.
| &'a self, | ||
| request: TokenGrantRequest<'a>, | ||
| ) -> Pin<Box<dyn Future<Output = Result<String, TokenGrantFailure>> + Send + 'a>> { | ||
| Box::pin(async move { |
There was a problem hiding this comment.
Major: with no supervisor cache, every proxied request becomes a GetSandboxProviderAccessToken call, and they all go through the single process-wide GrpcRateLimiter (multiplex.rs:350). With grpc_rate_limit_* set, one busy sandbox can drain the bucket and every other sandbox, supervisor control RPC and CLI call gets RESOURCE_EXHAUSTED. I'm not reopening the per-request authorization choice from round 1. Exempting this RPC from the global limiter and giving it a per-sandbox limit (or at least documenting the interaction) would contain it.
| // session JWT. No caller-selected identity, endpoint, audience, or scope. | ||
| message GetSandboxProviderAccessTokenRequest { | ||
| // Endpoint-bound dynamic credential key supplied by gateway configuration. | ||
| string credential_key = 1; |
There was a problem hiding this comment.
Major: credential_key carries the supervisor's internal tab-separated scoped key, rev:N and installation:X included, and the gateway reverse-parses it with unscoped_owned_key. That ties a public message to an internal delimiter format and the supervisor's cache-scoping scheme, so changing either becomes a wire break. Modelling the selector as fields (as ExchangeProviderSubjectTokenRequest does with provider + credential_key), or sending the unscoped configured key and documenting its format here, keeps the contract explicit.
|
|
||
| /// The reference is derived only from the driver's immutable identity retained | ||
| /// by the gateway, not mutable workload labels or caller-provided selectors. | ||
| fn reference(runtime: &str) -> Result<WorkloadReference, TokenGrantFailure> { |
There was a problem hiding this comment.
Minor: reference() re-parses the Kubernetes driver's private kubernetes://ns/uid/component encoding and repeats its CRD group and plural, and SandboxIdentityProvider::fetch_jwt_svid doesn't document runtime_identity. It fails closed, but a driver change would break issuance with no compile-time signal. A typed reference exported next to parse_kubernetes_runtime_identity gives it one owner.
| .min(svid.expires_at.saturating_mul(1000)) | ||
| .min(subject_expiry.unwrap_or(i64::MAX)); | ||
| let serve_until_ms = | ||
| serve_until_ms(&token, ttl_override, obtained_at_ms, identity_expires_at_ms).ok_or_else( |
There was a problem hiding this comment.
Minor: authorize accepts a session up to the 30 s leeway past exp, so a miss in that window runs the Broker fetch and token exchanges, then serve_until_ms returns None and the request fails as unclassified at token_endpoint. Checking session.expires_at against now before acquiring fails fast with an identity rejection instead.
| let session = authority | ||
| .verify_gateway_token(bearer) | ||
| .map_err(|error| error.status)?; | ||
| if session.expires_at == 0 { |
There was a problem hiding this comment.
Minor: sandboxes launched before gateway_jwt.ttl_secs was set hold exp = 0 bearers that are never renewed, so once an operator enables spiffe_broker they fail here until relaunched, while their other RPCs keep working. Worth a line in the configuration docs or a startup warning.
| /// DNS names are not used for SPIFFE authentication. Revocation checking is | ||
| /// not enabled; deployments rely on short-lived SVIDs. | ||
| #[must_use] | ||
| pub fn server_verifier(roots: RootCertStore, expected: String) -> Arc<dyn ServerCertVerifier> { |
There was a problem hiding this comment.
Minor: this verifier gates the Broker connection but has no unit tests for its failure paths (wrong URI SAN, extra URI SANs, missing SAN, untrusted chain). CRYPTO.md asks for failure-handling tests, and the probe-based checks don't run in CI.
| cleanup_vault_fixture | ||
| fi | ||
|
|
||
| cleanup_spire_fixture |
There was a problem hiding this comment.
Minor: this uninstalls SPIRE, and its CSI driver with it, before the OpenShell release whose gateway and supervisor Pods mount csi.spiffe.io is removed. On an existing cluster those Pods and the namespace stay Terminating, and the next run hits "namespace is being terminated". Moving the call after the release and namespace teardown fixes it.
| tonic = { workspace = true, features = ["channel", "tls-native-roots"] } | ||
| tonic-reflection = { workspace = true } | ||
| prost = { workspace = true } | ||
| tonic-prost = { workspace = true } |
There was a problem hiding this comment.
Minor: tonic-prost is added here, but nothing in openshell-server uses it.
| tonic-prost = { workspace = true } |
Summary
Enable shared-supervisor Kubernetes sandboxes to obtain provider access tokens using stable sandbox SPIFFE identities. The gateway obtains sandbox JWT-SVIDs through SPIRE's experimental SPIFFE Broker API and performs provider-token exchanges. Supervisors authenticate with each sandbox generation's credentials and receive only final provider access tokens; workloads receive no SPIFFE socket or credentials.
Related Issue
Closes #4731
Changes
GetSandboxProviderAccessTokento the OpenShell service. Validate the signed session against persisted runtime generation, credential epoch, token lineage, and active state. Resolve provider attachment, credential key, endpoint, audience, and scopes from gateway configuration.<prefix>/gateway/<gateway-id>/sandbox/<sandbox-id>and request identity using the Sandbox resource's persisted Kubernetes UID. Callers cannot supply workload references or selectors.openshell-crypto, then verify returned JWT-SVID signatures, audience, expiry, and exact sandbox subject.openshell-spiffeand the Broker adapter intoopenshell-spire. Update published configuration docs, Helm guidance, troubleshooting, and integration fixtures.Token flow
sequenceDiagram participant W as Sandbox workload participant S as Shared supervisor participant G as Gateway participant A as SPIRE Workload API participant B as SPIFFE Broker participant K as Kubernetes API participant T as Provider token service participant U as Upstream service W->>S: Request requiring provider credentials S->>G: Request token with sandbox session JWT G->>G: Verify session and active sandbox generation G->>G: Resolve attached provider and configured grant alt Valid cached token G->>G: Read token bound to sandbox, grant and subject else Cache miss G->>G: Derive stable SPIFFE ID and Sandbox UID G->>A: Fetch current gateway X.509-SVID and trust bundles A-->>G: Gateway credentials and bundles G->>B: Connect with mTLS and verify Broker identity G->>B: Request sandbox JWT-SVID using Sandbox UID B->>B: Check authorized broker identity B->>K: Attest Sandbox and check gateway authorization K-->>B: Attestation and authorization results B-->>G: Sandbox JWT-SVID G->>B: Fetch JWT verification bundles B-->>G: Bundles G->>G: Verify signature, audience, expiry and identity opt RFC 8693 token exchange G->>A: Fetch gateway JWT-SVID A-->>G: Gateway JWT-SVID G->>T: Exchange provider subject credential T-->>G: Intermediate token bound to sandbox identity end G->>T: Request provider token using sandbox JWT-SVID T-->>G: Provider access token end G->>G: Recheck configuration and session G-->>S: Final access token and remaining lifetime S->>U: Forward request with injected token U-->>S: Response S-->>W: ResponseConfiguration and scope
The feature is disabled unless
[openshell.gateway.spiffe_broker]is configured; there is no compile-time feature flag. Gateway session JWTs with a positive TTL are required.Operators must enable SPIRE's experimental Broker endpoint, authorize the gateway identity, configure Kubernetes attestation with
access_policy = "enforced", and provision registrations for sandbox identities. The agent needs Sandbox lookup and SubjectAccessReview permissions; the gateway identity needs narrowly scopedimpersonate-via-spireauthorization. The configured server SPIFFE ID must match the Broker endpoint's agent certificate exactly.The gateway uses a CSI Workload API mount. Shared supervisors and sandbox workloads require no SPIFFE mounts. RFC 8693 profiles also require the gateway's existing Workload API configuration for its component JWT-SVID (
server.providerTokenGrants.spiffe.enabled=truein Helm).This change supports Kubernetes shared supervision. It does not automatically provision SPIRE registrations or enable other compute drivers. Already-issued provider bearer tokens retain their normal expiration. Kubernetes Restricted Pod Security admission was validated; OpenShift
restricted-v2SCC compatibility has not been validated.The public RPC adds a method to generated Rust server traits and the full TypeScript client interface; downstream handwritten implementations and mocks may require updates.
Connection pooling, broader cache improvements, generated upstream Broker bindings, and inclusion of the SPIRE fixture in default CI remain follow-ups.
Testing
Validation of the implementation at
17c8b7b2c:The subsequent commit
4532524dconly corrects troubleshooting advice and a test description;git diff --checkpassed.