Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 51 additions & 1 deletion deploy/helm/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,55 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays:
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development

### Sandbox service routing

Gateway-mediated sandbox service routing is disabled by default in this Kubernetes
chart to avoid an unintended alternate application access path. Control-plane API
access is unchanged.

For service ingress directly through supervisors, enable
`[openshell.supervisor.app_listener].enabled` and pair it with an ingress solution
that routes requests to the app listener on the supervisor owning each sandbox:

```yaml
gatewayConfig:
openshell.gateway:
service_routing_enabled: false
openshell.supervisor.app_listener:
enabled: true
port: 9443
base_url: https://services.example.com
openshell.drivers.kubernetes.sandbox_runtime.app_listener:
tls_secret_name: openshell-app-tls
```

Provision `openshell-app-tls` with a certificate for `*.services.example.com`.
Pair the listener with a TLS-passthrough proxy or a compatible Kubernetes
ingress/Gateway API integration. The integration consumes service discovery to
map each advertised hostname to the current owning supervisor's IP and configured
app-listener port, preserving the application hostname as Host and TLS SNI.
Configure DNS and NetworkPolicy for that path. Enabling the listener alone does
not provision DNS or external ingress routes, and the chart does not enable it
automatically.

To use gateway-mediated service routing instead, explicitly enable forwarding;
that path does not require the app listener:

```yaml
gatewayConfig:
openshell.gateway:
service_routing_enabled: true
```

Existing Helm installations that rely on gateway service URLs must set this
explicitly before upgrading. Missing saved values receive the disabled chart
default; explicit choices are preserved. If neither routing mode is enabled,
startup logs warn that sandbox service routing is disabled.

The core runtime continues to default gateway routing to enabled, preserving
Docker, Podman, and VM deployments outside this chart. Gateway service
authentication remains independently opt-in.

### Additional destination CAs

Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach
Expand Down Expand Up @@ -577,7 +626,8 @@ discovery endpoint or its TLS CA.
| gateway.image.registry | string | `""` | Gateway image registry. Empty uses global.image.registry. |
| gateway.image.repository | string | `"openshell/gateway"` | Gateway image repository. |
| gateway.image.tag | string | `""` | Gateway image tag. Defaults to the chart appVersion when empty. |
| gatewayConfig | object | `{"openshell":{"version":2}}` | Non-secret gateway application configuration. Top-level keys name TOML tables and are rendered into the mounted gateway.toml file. Kubernetes resource inputs remain outside this map; template expressions derive the corresponding runtime values from their resource owner. |
| gatewayConfig | object | `{"openshell":{"version":2},"openshell.gateway":{"service_routing_enabled":false}}` | Non-secret gateway application configuration. Top-level keys name TOML tables and are rendered into the mounted gateway.toml file. Kubernetes resource inputs remain outside this map; template expressions derive the corresponding runtime values from their resource owner. |
| gatewayConfig."openshell.gateway".service_routing_enabled | bool | `false` | Opt in to gateway-mediated sandbox service URLs and forwarding. Disabled for Kubernetes; configure application-listener ingress or explicitly enable this path. |
| global.image.pullPolicy | string | `"IfNotPresent"` | Shared OpenShell image pull policy. Individual image pull policies take precedence. |
| global.image.registry | string | `"ghcr.io/nvidia"` | Shared OpenShell image registry. Individual image registries take precedence. |
| global.image.tag | string | `""` | Shared OpenShell image tag. Defaults to the chart appVersion when empty. |
Expand Down
49 changes: 49 additions & 0 deletions deploy/helm/openshell/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,55 @@ See [`values.yaml`](values.yaml) for source defaults. Selected overlays:
- [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants
- [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development

### Sandbox service routing

Gateway-mediated sandbox service routing is disabled by default in this Kubernetes
chart to avoid an unintended alternate application access path. Control-plane API
access is unchanged.

For service ingress directly through supervisors, enable
`[openshell.supervisor.app_listener].enabled` and pair it with an ingress solution
that routes requests to the app listener on the supervisor owning each sandbox:

```yaml
gatewayConfig:
openshell.gateway:
service_routing_enabled: false
openshell.supervisor.app_listener:
enabled: true
port: 9443
base_url: https://services.example.com
openshell.drivers.kubernetes.sandbox_runtime.app_listener:
tls_secret_name: openshell-app-tls
```

Provision `openshell-app-tls` with a certificate for `*.services.example.com`.
Pair the listener with a TLS-passthrough proxy or a compatible Kubernetes
ingress/Gateway API integration. The integration consumes service discovery to
map each advertised hostname to the current owning supervisor's IP and configured
app-listener port, preserving the application hostname as Host and TLS SNI.
Configure DNS and NetworkPolicy for that path. Enabling the listener alone does
not provision DNS or external ingress routes, and the chart does not enable it
automatically.

To use gateway-mediated service routing instead, explicitly enable forwarding;
that path does not require the app listener:

```yaml
gatewayConfig:
openshell.gateway:
service_routing_enabled: true
```

Existing Helm installations that rely on gateway service URLs must set this
explicitly before upgrading. Missing saved values receive the disabled chart
default; explicit choices are preserved. If neither routing mode is enabled,
startup logs warn that sandbox service routing is disabled.

The core runtime continues to default gateway routing to enabled, preserving
Docker, Podman, and VM deployments outside this chart. Gateway service
authentication remains independently opt-in.

### Additional destination CAs

Create an operator-owned ConfigMap with a `ca.crt` key when sandboxes must reach
Expand Down
4 changes: 4 additions & 0 deletions deploy/helm/openshell/templates/_toml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,10 @@ Top-level lists represent TOML arrays of tables and preserve their YAML order. *
{{- $config := deepCopy (.Values.gatewayConfig | default dict) -}}
{{- $legacyServer := .Values.server | default dict -}}
{{- $gateway := get $config "openshell.gateway" | default dict -}}
{{/* Keep the Kubernetes routing default when saved values omit it or use null. */}}
{{- if or (not (hasKey $gateway "service_routing_enabled")) (eq (get $gateway "service_routing_enabled") nil) -}}
{{- $_ := set $gateway "service_routing_enabled" false -}}
{{- end -}}
{{- if not (hasKey $gateway "name") -}}{{- $_ := set $gateway "name" (get $legacyServer "name" | default (include "openshell.fullname" .)) -}}{{- end -}}
{{/* The Service and probes own listener ports, so keep runtime listeners aligned. */}}
{{- $_ := set $gateway "bind_address" (printf "0.0.0.0:%v" .Values.service.port) -}}
Expand Down
31 changes: 26 additions & 5 deletions deploy/helm/openshell/tests/gateway_config_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,40 @@ release:
namespace: my-namespace

tests:
- it: leaves gateway service routing enabled by the runtime default when omitted
- it: disables gateway service routing by default on Kubernetes
template: templates/gateway-config.yaml
asserts:
- notMatchRegex:
- matchRegex:
path: data["gateway.toml"]
pattern: 'service_routing_enabled\s*='
pattern: '(?m)^service_routing_enabled\s*=\s*false$'

- it: allows explicitly enabling gateway service routing
template: templates/gateway-config.yaml
set:
gatewayConfig:
openshell.gateway:
service_routing_enabled: true
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '(?m)^service_routing_enabled\s*=\s*true$'

- it: defaults gateway routing off when saved configuration omits the gateway table
template: templates/gateway-config.yaml
set:
gatewayConfig:
openshell.gateway: null
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '(?m)^service_routing_enabled\s*=\s*false$'

- it: disables gateway service routing through canonical configuration
- it: defaults gateway routing off when saved configuration omits the flag
template: templates/gateway-config.yaml
set:
gatewayConfig:
openshell.gateway:
service_routing_enabled: false
service_routing_enabled: null
asserts:
- matchRegex:
path: data["gateway.toml"]
Expand Down
3 changes: 3 additions & 0 deletions deploy/helm/openshell/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,9 @@ topologySpreadConstraints: []
gatewayConfig:
openshell:
version: 2
openshell.gateway:
# -- Opt in to gateway-mediated sandbox service URLs and forwarding. Disabled for Kubernetes; configure application-listener ingress or explicitly enable this path.
service_routing_enabled: false

# Server configuration
server:
Expand Down
24 changes: 23 additions & 1 deletion docs/how-it-works/gateways/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -607,7 +607,9 @@ and `https://<workspace>--<sandbox>.<base-domain>/` for the unnamed endpoint.
The base domain is the first wildcard DNS SAN in `server_sans`; gateways
without SANs use the built-in `openshell.localhost`.

`service_routing_enabled` defaults to `true`. Set it to `false` to stop publishing
The core runtime defaults `service_routing_enabled` to `true`; Docker, Podman,
and VM gateway behavior is unchanged. The Kubernetes Helm chart defaults it to
`false`. Set it to `false` to stop publishing
gateway service URLs and reject gateway-mediated application HTTP requests and
WebSocket upgrades with `404`. Control-plane APIs, service declarations, direct
supervisor listeners, and discovery remain available. The setting also accepts
Expand All @@ -621,6 +623,26 @@ gatewayConfig:
service_routing_enabled: false
```

For Kubernetes service ingress directly through supervisors, set
`[openshell.supervisor.app_listener].enabled = true` and configure the
[supervisor application listener](#supervisor-application-listener). Pair it with
an ingress solution, such as a TLS-passthrough proxy or a compatible Kubernetes
ingress/Gateway API integration, that consumes discovery and routes each
advertised hostname to the current owning supervisor's app-listener IP and port.
Preserve the application hostname as Host and TLS SNI. Enabling the listener
alone does not provision DNS or external ingress routes; the chart does not
enable either the listener or the ingress integration automatically.

If both routing modes are disabled, startup logs warn that sandbox service
routing is disabled. Gateway-mediated forwarding does not require the app
listener.

When upgrading a Helm installation that uses gateway service URLs, explicitly
set `service_routing_enabled: true` in the canonical table above before the
upgrade. Missing saved values receive the chart's `false` default; explicit
`true` values remain enabled. This changes application forwarding only, not
control-plane API access or the service-authentication default.

With gateway routing disabled, `gateway_url` is empty and `gateway_service_urls`
is empty. A configured application origin supplies the preferred URL; without
one, preferred URLs are empty. Idempotent retries preserve recorded preferred
Expand Down
5 changes: 3 additions & 2 deletions e2e/kubernetes/app-listener/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,9 @@ differs from the repository's mise environment.
`OPENSHELL_E2E_APP_DELIVERY` accepts `push` (default) or `poll`;
`OPENSHELL_E2E_APP_TAKEOVER` accepts `graceful` (default) or `abrupt` for shared
supervisors. Dedicated runs include gateway-outage expiry. Every run also
disables the listener and verifies absent app sockets/TLS mounts/Services and
continued gateway application ingress. Logs, events, and resource summaries
disables the listener, explicitly enables gateway routing for the fallback
check, and verifies absent app sockets/TLS mounts/Services and continued gateway
application ingress. Logs, events, and resource summaries
are written under `results/app-listener/`, without TLS private keys or Secrets.

GitHub Actions runs four matrix cases under the existing `test:e2e` label:
Expand Down
3 changes: 2 additions & 1 deletion e2e/kubernetes/app-listener/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,8 @@ def wait_for_shared_hosts():
json.dumps(
{
"gatewayConfig": {
"openshell.supervisor.app_listener": {"enabled": False}
"openshell.supervisor.app_listener": {"enabled": False},
"openshell.gateway": {"service_routing_enabled": True},
}
}
)
Expand Down
6 changes: 5 additions & 1 deletion skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -1760,7 +1760,11 @@ Watches recheck bearer credentials and membership every second. Allow one refres
interval for withdrawals; route lease expiry remains independently enforced.

For incorrect public service URLs, inspect
`[openshell.gateway].service_routing_enabled` first. When false, gateway URL fields
`[openshell.gateway].service_routing_enabled` first. The Helm chart defaults it
to false; the core runtime and non-Kubernetes deployments retain true. To retain
gateway forwarding in a Helm deployment or upgrade, set
`gatewayConfig."openshell.gateway".service_routing_enabled: true` explicitly.
When false, gateway URL fields
are empty and gateway application requests return `404`; direct listener and
discovery behavior is independent. Then inspect
`[openshell.supervisor.app_listener].base_url`. It is the external HTTPS base origin
Expand Down
Loading