Skip to content

feat(gateway): expose workspace service route snapshots and watches - #4941

Merged
TaylorMutch merged 6 commits into
mainfrom
feat/4667-service-discovery/TaylorMutch
Nov 16, 2026
Merged

TaylorMutch merged 6 commits into
mainfrom
feat/4667-service-discovery/TaylorMutch

Conversation

@TaylorMutch

@TaylorMutch TaylorMutch commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Expose accepted supervisor installations through the separate openshell.v1.ServiceDiscovery service: GetServiceRouteSnapshot and WatchServiceRoutes for one explicit authorized workspace. Independent proxies and infrastructure controllers consume exact hostnames and trusted supervisor endpoints through the same API.

Based on main at e8ba30fde.

Related Issue

Closes #4667. Part of RFC 0017.

Changes

  • Keep discovery separate from the management OpenShell service. Register it with gateway auth, configured interceptors, and reflection. Existing management server implementations need no discovery stubs; Rust SDK consumers use raw_service_discovery() / raw_service_discovery_fresh().

  • Reuse collection-local provider catalog, global settings, and bounded bulk service declarations while retaining per-sandbox validation. Recheck shared dependencies before publication; unsuitable bulk inputs fall back to individual reads.

  • Reread accepted installation receipts before publication, recheck live ownership, and revalidate renewed configuration. Return the latest accepted expiry for unchanged identities; omit withdrawn or replaced installations. Reads never extend leases.

  • Share collection results for one second per workspace, cache at most 16 workspaces, and allow four simultaneous collections. Never return expired cached routes.

  • Bound unary requests to 16 total, four per principal, eight per workspace. Bound watches to 32 total, four per principal, 16 per workspace; release all budgets on cancellation/error/expiry.

  • Recheck bearer authentication, use refreshed identity roles, and reauthorize workspace access before publishing watch replacements. Offline JWT revocation follows verifier semantics.

  • Gate snapshot and watch RPCs on supervisor_app_listener.enabled; disabled listeners return FAILED_PRECONDITION before collecting routes or allocating a watch.

  • Publish accepted installation records from feat(supervisor): advertise accepted application route installations #4940, which resolves trusted supervisor endpoints through the private ResolveSupervisorAppEndpoint compute RPC. External consumers use the public discovery RPCs; ordinary GetSandbox observations do not resolve routes.

  • Return complete replacements with installation versions, declaration revisions, expiry, and current owner endpoints.

  • Recheck session, runtime, credential epoch, and shared pool authority before publishing routes.

  • Bound collection, watches, buffering, send time, and lifetime; periodically reauthorize watches and fail closed on storage errors.

  • Document client cache expiry and stable installation versions across lease-only renewal.

Release Notes

Discovery uses /openshell.v1.ServiceDiscovery/GetServiceRouteSnapshot and /openshell.v1.ServiceDiscovery/WatchServiceRoutes. Message shapes are unchanged.

Discovery and Kubernetes request flow

Discovery returns an application hostname and a trusted supervisor endpoint
(host and port), together with installation identity and expiry. name is the
sandbox route's service name; an empty value identifies the unnamed/default
service. It is not a Kubernetes Service resource name.

The endpoint is the owning supervisor Pod IP and app-listener port. Consumers
preserve the application hostname as TLS SNI and HTTP Host/authority. Multiple
routes can share one supervisor endpoint; the hostname selects the sandbox
instance and service. The authoritative public application URL is separate and
can use a different public port.

flowchart TB
    subgraph Control["Control plane: build and publish discovery"]
        SUP["Supervisor<br/>Reports installed routes against a lease"]
        GW["OpenShell gateway<br/>Validates installation and ownership"]
        KD["Kubernetes compute driver<br/>ResolveSupervisorAppEndpoint"]
        KUBE["Kubernetes objects + supervisor ownership<br/>Find and validate owning supervisor Pod"]
        API["ServiceDiscovery service<br/>Snapshot / watch<br/>hostname + Pod IP:8443 + expiry"]
        CONSUMER["Sandbox proxy or<br/>example Kubernetes HTTPRoute controller"]

        SUP -->|"Installed-route advertisement"| GW
        GW -->|"Resolve trusted endpoint"| KD
        KD --> KUBE
        KUBE -->|"Validated Pod IP and listener port"| KD
        KD --> GW
        GW --> API
        API -->|"Snapshot / watch"| CONSUMER
    end

    subgraph Data["Application data plane"]
        CLIENT["Client<br/>https://team--demo--web.apps.example.com"]
        EDGE["Configured proxy or Kubernetes Gateway<br/>Public listener, e.g. port 443"]
        LISTENER["Owning supervisor Pod<br/>10.42.1.27:8443<br/>TLS app listener"]
        DISPATCH["Select sandbox instance + named service<br/>using SNI / Host"]
        CONNECTOR["Selected instance's sandbox connector"]
        APP["Application inside sandbox<br/>127.0.0.1:8080"]

        CLIENT --> EDGE
        EDGE -->|"Connect to discovered endpoint<br/>preserve application SNI / Host"| LISTENER
        LISTENER --> DISPATCH
        DISPATCH --> CONNECTOR
        CONNECTOR --> APP
    end

    CONSUMER -.->|"Configure forwarding"| EDGE
Loading

Addresses, domain names, and ports above are illustrative. Discovery describes
accepted installations, not application health or external ingress readiness.
The Kubernetes driver resolves trusted endpoints; a separate consumer configures
HTTPRoutes or proxy forwarding. Application traffic bypasses the OpenShell
gateway. The proxy and example controller are separate consumers, not part of
this PR's implementation.

Testing

  • Rebased onto main e8ba30fde: standalone discovery passed 2,725 server tests (42 PostgreSQL-dependent tests skipped locally).
  • Combined stack passed 2,761 server tests (43 PostgreSQL-dependent tests skipped), nine proxy tests, and 19 supervisor listener regressions. Strict all-target server/supervisor/proxy Clippy passed.
  • TypeScript SDK validation passed: protobuf generation/lint, lint, type checking, 207 tests, coverage, and build.
  • Helm lint and all three chart unit suites passed, including gateway TOML parsing, resource coherence, ownership, and supervisor scaling checks. Markdown lint and Rust formatting passed.
  • Compatibility regressions cover recorded replay URLs across origin/port changes and disabled ingress, existing TypeScript object shapes, auth-disabled loopback schemes, application-cookie preservation, and real-TLS discovery-certificate opt-in behavior.

Earlier local k3d validation covered dedicated/shared owner changes, HTTP/WebSockets, network-policy isolation, discovery withdrawal, and two proxy replicas. This revision has no new live-cluster or production-load qualification. New-head CI is pending.

Qualification limits

Live HA validation exercised supervisor process/Pod loss on a local k3d cluster. Whole-node failure, real network partitions, metric-driven HPA, sustained production load, and published release artifacts remain unqualified. Grant-file projection contributes to revocation propagation time. Main now includes #4842 for gateway-mediated hashed hostnames. Direct application-listener/discovery hostnames remain readable; extending hashed naming to that path and installation-status presentation (#4670) remain separate work.

The example Kubernetes HTTPRoute controller is a deferred compatibility POC outside this stack.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Relevant implementation, configuration, and API documentation updated

PR dependency order: #4941 → #4956 → #4963 → #4964 → #4957.

@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

@TaylorMutch
TaylorMutch added this pull request to stack #4942 October 26, 2026 22:35
@TaylorMutch
TaylorMutch force-pushed the feat/4667-service-discovery/TaylorMutch branch from c724036 to 932643d Compare October 27, 2026 03:15
@TaylorMutch
TaylorMutch force-pushed the feat/4667-service-discovery/TaylorMutch branch from 932643d to 6c83b66 Compare October 27, 2026 05:42
@TaylorMutch
TaylorMutch removed this pull request from stack #4942 October 27, 2026 05:45
@TaylorMutch
TaylorMutch added this pull request to stack #4965 October 27, 2026 05:46
@TaylorMutch
TaylorMutch removed this pull request from stack #4965 October 27, 2026 05:47
@TaylorMutch
TaylorMutch added this pull request to stack #4966 October 27, 2026 06:24
@TaylorMutch
TaylorMutch force-pushed the feat/4667-service-discovery/TaylorMutch branch from 6c83b66 to 755e4d1 Compare October 27, 2026 06:24
@TaylorMutch
TaylorMutch force-pushed the feat/4667-service-discovery/TaylorMutch branch from 755e4d1 to 0944ea2 Compare October 27, 2026 06:25
@TaylorMutch
TaylorMutch force-pushed the feat/4667-service-discovery/TaylorMutch branch from 0944ea2 to 8504067 Compare October 27, 2026 06:46
@TaylorMutch
TaylorMutch force-pushed the feat/4667-service-discovery/TaylorMutch branch from 8504067 to feac6b4 Compare October 27, 2026 21:06
@purp purp added topic:api Public APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards. topic:authnz Authentication, authorization, identity, sessions, credential protection, and policy integrity. topic:documentation Architecture docs, READMEs, project website updates, and automated documentation generation. topic:networking General network transport, connectivity, and proxy behavior. topic:performance Startup time, latency, throughput, overhead, scalability, benchmarking, and rate limiting. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts. topic:sdk:rust Rust SDK implementation, maintenance, interfaces, and distribution. topic:sdk:typescript TypeScript SDK implementation, maintenance, interfaces, and distribution. labels Nov 15, 2026
@TaylorMutch
TaylorMutch requested a review from pimlock November 16, 2026 17:16
pimlock
pimlock previously approved these changes Nov 16, 2026

@pimlock pimlock left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

@TaylorMutch
TaylorMutch added this pull request to the merge queue Nov 16, 2026
@TaylorMutch
TaylorMutch removed this pull request from the merge queue due to a manual request Nov 16, 2026
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
…nsion

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch
TaylorMutch force-pushed the feat/4667-service-discovery/TaylorMutch branch from fd8af36 to 29c0d05 Compare November 16, 2026 19:55
@TaylorMutch
TaylorMutch added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@TaylorMutch
TaylorMutch added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit bf9f55c Nov 16, 2026
94 of 95 checks passed
@TaylorMutch
TaylorMutch deleted the feat/4667-service-discovery/TaylorMutch branch November 16, 2026 21:36
jhjaggars added a commit to jhjaggars/OpenShell that referenced this pull request Nov 17, 2026
Notes and live-trial evidence for the workspace service-route discovery
API (PR NVIDIA#4941): snapshot/watch behavior on a k3d cluster, the TLS data
plane to discovered supervisor endpoints, and a shared-certificate edge
PoC (Envoy Gateway + HTTPRoute + BackendTLSPolicy) re-encrypting to a
discovered endpoint. Includes design options and upstream feedback for
shared-cert service exposure.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:platform SDKs, CLIs, APIs, gateway infrastructure, persistence, configuration, and session management. topic:api Public APIs, gRPC/protobuf, SDK interfaces, versioning, compatibility, and API design standards. topic:authnz Authentication, authorization, identity, sessions, credential protection, and policy integrity. topic:documentation Architecture docs, READMEs, project website updates, and automated documentation generation. topic:networking General network transport, connectivity, and proxy behavior. topic:performance Startup time, latency, throughput, overhead, scalability, benchmarking, and rate limiting. topic:reliability High availability, recovery, retries, graceful degradation, and correctness across restarts. topic:sdk:rust Rust SDK implementation, maintenance, interfaces, and distribution. topic:sdk:typescript TypeScript SDK implementation, maintenance, interfaces, and distribution.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(gateway): expose workspace service route snapshots and watches

3 participants