From 721dba7d6508bf2ac8c3620798ec5c465b0d495f Mon Sep 17 00:00:00 2001 From: Drew Newberry Date: Wed, 30 Sep 2026 22:49:37 -0700 Subject: [PATCH 1/4] feat(docker): select supervisor networking for ECI workloads Signed-off-by: Drew Newberry --- crates/openshell-driver-docker/README.md | 35 ++- crates/openshell-driver-docker/src/lib.rs | 139 +++++++++++- crates/openshell-driver-docker/src/tests.rs | 201 +++++++++++++++++- crates/openshell-supervisor/src/main.rs | 88 +++++++- docs/about/support-matrix.mdx | 2 +- docs/how-it-works/gateways/configuration.mdx | 37 +++- docs/how-it-works/gateways/overview.mdx | 2 +- docs/how-it-works/sandboxes/runtimes.mdx | 10 +- e2e/rust/Cargo.toml | 5 + .../tests/docker_supervisor_networking.rs | 127 +++++++++++ e2e/with-docker-gateway.sh | 17 +- skills/debug-openshell-cluster/SKILL.md | 16 +- tasks/test.toml | 8 + 13 files changed, 653 insertions(+), 34 deletions(-) create mode 100644 e2e/rust/tests/docker_supervisor_networking.rs diff --git a/crates/openshell-driver-docker/README.md b/crates/openshell-driver-docker/README.md index ee329f8dea..feb94e68d6 100644 --- a/crates/openshell-driver-docker/README.md +++ b/crates/openshell-driver-docker/README.md @@ -41,9 +41,11 @@ mediates every supported TCP and DNS operation, attributes it to the calling binary, and sends the request across the private channel. The supervisor authorizes the request before it opens an upstream connection. Docker's absent workload network is the mandatory outer fence if mediation fails or is -bypassed. The trusted supervisor companion uses Docker host networking, where -it reaches the gateway's primary loopback listener and originates approved -egress. +bypassed. The trusted supervisor companion originates approved egress. Its +operator-owned `supervisor_network_mode` defaults to `auto`: each launch +inspects the existing workload container and chooses bridge for `sysbox-runc` +(the Docker Desktop ECI runtime), or host for other runtimes. Explicit `host` +and `bridge` values override this heuristic. No probe container is created. The driver copies trusted runtime bytes from the configured supervisor image through the Docker archive API. No workload launch depends on a host bind @@ -158,7 +160,7 @@ openshell sandbox create \ `/openshell-sandbox` binary. The driver extracts that binary as bytes and stages it into the stopped workload. `supervisor_image` contains the dynamically linked glibc `/openshell-supervisor` binary that runs in the -host-networked supervisor container. Release and gateway image builds bake +supervisor companion container. Release and gateway image builds bake matching image tags into the binary. ## Gateway session and TLS @@ -172,14 +174,31 @@ When no endpoint is configured, the supervisor connects to Docker daemon host. A configured HTTPS server certificate must include the endpoint host in its subject alternative names. -The driver publishes host loopback as the backend address for +In host mode, the driver publishes host loopback as the backend address for `host.openshell.internal`. Policy DNS resolves that reserved name through the mediated path, so policies can reach host services without a Docker bridge, container DNS alias, or another gateway listener. -Docker Engine on Linux supports host networking directly. Docker Desktop -requires host networking to be enabled in Settings and does not support it -when Enhanced Container Isolation is enabled. +Host mode on Docker Desktop requires host networking enabled in Settings and +ECI disabled. Bridge mode uses Docker-resolved `host-gateway` entries for both +host aliases. The supervisor's `--host-gateway-from-hosts` option pins the +numeric address from its own driver-owned `/etc/hosts` into the runtime +descriptor before attaching the workload; workload DNS and hosts files cannot +select this address. Gateway endpoints and host aliases are independent in +bridge mode. + +When `grpc_endpoint` is omitted, bridge mode uses `host.docker.internal` on +Desktop or for a wildcard gateway listener, and the concrete bind address for +other non-loopback Linux listeners. The port is the gateway bind port; remapped +container publications require an explicit endpoint. Native Linux loopback-only +listeners cannot serve a bridge supervisor and produce an actionable error. +Explicit endpoints retain TLS verification and are never rewritten; bridge +rejects loopback/unspecified endpoint addresses. Host services bound only to +loopback are also unreachable from a native Linux bridge supervisor. + +Selecting bridge does not qualify full ECI support. Landlock and seccomp probes +remain mandatory, and a gateway container under ECI needs an administrator's +Docker socket exception. The supervisor owns these security-critical variables: diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index d4a1f2f71d..95e4fd5c17 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -152,6 +152,17 @@ fn provisioning_span( span } +/// Operator-owned networking mode for the trusted supervisor companion. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum DockerSupervisorNetworkMode { + /// Use bridge for Sysbox/ECI workloads, otherwise use host networking. + #[default] + Auto, + Host, + Bridge, +} + /// Gateway-local configuration for the Docker compute driver. #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] #[serde(default, deny_unknown_fields)] @@ -177,6 +188,9 @@ pub struct DockerComputeConfig { /// Gateway gRPC endpoint the sandbox connects back to. pub grpc_endpoint: String, + /// Networking mode for the supervisor only. Workloads always use network=none. + pub supervisor_network_mode: DockerSupervisorNetworkMode, + /// Image containing the trusted `openshell-sandbox` binary. pub sandbox_runtime_image: Option, @@ -273,6 +287,7 @@ impl Default for DockerComputeConfig { image_pull_policy: ImagePullPolicy::default(), sandbox_label: "default".to_string(), grpc_endpoint: String::new(), + supervisor_network_mode: DockerSupervisorNetworkMode::Auto, sandbox_runtime_image: None, supervisor_bin: None, supervisor_image: None, @@ -309,6 +324,9 @@ struct DockerDriverRuntimeConfig { sandbox_binary: Arc>, supervisor_image_id: String, supervisor_grpc_endpoint: String, + supervisor_network_mode: DockerSupervisorNetworkMode, + supervisor_endpoint_explicit: bool, + bridge_default_endpoint: Option, ssh_socket_path: String, guest_tls: Option, gpu: DockerGpuRuntimeCapabilities, @@ -881,6 +899,14 @@ impl DockerComputeDriver { }; validate_docker_app_armor_profile(docker_config.app_armor_profile.as_ref(), &info)?; let gateway_port = gateway_bind_address.port(); + let supervisor_endpoint_explicit = !docker_config.grpc_endpoint.trim().is_empty(); + let bridge_default_endpoint = default_docker_bridge_endpoint( + gateway_bind_address, + docker_guest_tls_configured(docker_config), + info.operating_system + .as_deref() + .is_some_and(|os| os.to_ascii_lowercase().contains("docker desktop")), + ); let mut docker_config = docker_config.clone(); if docker_config.grpc_endpoint.trim().is_empty() { docker_config.grpc_endpoint = default_docker_supervisor_grpc_endpoint( @@ -938,6 +964,9 @@ impl DockerComputeDriver { sandbox_binary, supervisor_image_id, supervisor_grpc_endpoint, + supervisor_network_mode: docker_config.supervisor_network_mode, + supervisor_endpoint_explicit, + bridge_default_endpoint, ssh_socket_path: docker_config.ssh_socket_path.clone(), guest_tls, gpu, @@ -4562,10 +4591,9 @@ async fn prepare_docker_boundary_files( server_name: tls.server_name.clone(), trust_anchor_pem: tls.trust_anchor_pem.clone(), }, - // Pin the reserved host alias to the same address used by the - // host-networked supervisor. This is normally loopback, but container - // CI reaches the gateway and host fixtures through the job - // container's bridge address. + // Host mode pins the alias to the gateway address, normally loopback. + // In bridge mode the companion replaces this value from its own + // Docker-injected hosts file before attaching the descriptor. host_gateway_ip: docker_supervisor_host_address(&config.supervisor_grpc_endpoint), workload_identity: workload_identity.clone(), child_env: docker_child_environment(sandbox), @@ -4976,14 +5004,34 @@ fn docker_auxiliary_container_labels( ]) } -fn docker_supervisor_host_config(mounts: Vec, grpc_endpoint: &str) -> HostConfig { +fn docker_supervisor_host_config( + mounts: Vec, + grpc_endpoint: &str, + network_mode: DockerSupervisorNetworkMode, +) -> HostConfig { HostConfig { // The supervisor is trusted infrastructure and originates every // approved upstream connection. Host networking lets it reach the // configured gateway and host-side services directly; the workload // remains fenced by network=none. - network_mode: Some("host".to_string()), - extra_hosts: docker_supervisor_host_aliases(grpc_endpoint), + network_mode: Some( + if network_mode == DockerSupervisorNetworkMode::Bridge { + "bridge" + } else { + "host" + } + .to_string(), + ), + extra_hosts: if network_mode == DockerSupervisorNetworkMode::Bridge { + // Docker resolves host-gateway on the daemon, including Desktop's VM. + // The gateway endpoint may be remote and must not define host aliases. + Some(vec![ + format!("{HOST_OPEN_SHELL_INTERNAL}:host-gateway"), + format!("{HOST_DOCKER_INTERNAL}:host-gateway"), + ]) + } else { + docker_supervisor_host_aliases(grpc_endpoint) + }, mounts: Some(mounts), cap_drop: Some(vec!["ALL".to_string()]), cap_add: None, @@ -5039,6 +5087,18 @@ async fn spawn_docker_control_process( config: &DockerDriverRuntimeConfig, failure_context: DockerRuntimeFailureContext, ) -> Result { + // Inspect the existing workload on every supervisor launch, including restart. + // This adds no diagnostic container and observes the actual selected runtime. + let (network_mode, supervisor_endpoint) = Box::pin(async { + let workload = docker + .inspect_container(&failure_context.container_id, None) + .await + .map_err(|error| internal_status("inspect Docker workload runtime", error))?; + docker_supervisor_networking(config, &workload) + }) + .await?; + info!(sandbox_id = %sandbox.id, ?network_mode, endpoint = %supervisor_endpoint, + "Selected Docker supervisor networking"); let directory = docker_boundary_state_dir(sandbox, config)?; let main_process_spec = tokio::fs::read_to_string(directory.join(MAIN_PROCESS_SPEC_FILE)) .await @@ -5067,7 +5127,7 @@ async fn spawn_docker_control_process( format!( "{}={}", openshell_core::sandbox_env::ENDPOINT, - config.supervisor_grpc_endpoint + supervisor_endpoint ), format!("{}={}", openshell_core::sandbox_env::SANDBOX_ID, sandbox.id), format!("{}={}", openshell_core::sandbox_env::SANDBOX, sandbox.name), @@ -5144,6 +5204,9 @@ async fn spawn_docker_control_process( workspace_root, format!("--health-socket-path={SUPERVISOR_HEALTH_SOCKET_PATH}"), ]; + if network_mode == DockerSupervisorNetworkMode::Bridge { + command.push("--host-gateway-from-hosts".to_string()); + } command.extend(docker_upstream_proxy_cli_args( &config.upstream_proxy, config.proxy_ca_bundle.is_some(), @@ -5207,7 +5270,8 @@ async fn spawn_docker_control_process( }), host_config: Some(docker_supervisor_host_config( supervisor_mounts, - &config.supervisor_grpc_endpoint, + &supervisor_endpoint, + network_mode, )), ..Default::default() }; @@ -6514,6 +6578,63 @@ fn default_docker_supervisor_grpc_endpoint(gateway_port: u16, tls: bool) -> Stri format!("{scheme}://127.0.0.1:{gateway_port}") } +fn default_docker_bridge_endpoint( + gateway_bind_address: SocketAddr, + tls: bool, + desktop: bool, +) -> Option { + let scheme = if tls { "https" } else { "http" }; + if desktop || gateway_bind_address.ip().is_unspecified() { + Some(format!( + "{scheme}://{HOST_DOCKER_INTERNAL}:{}", + gateway_bind_address.port() + )) + } else if !gateway_bind_address.ip().is_loopback() { + Some(format!("{scheme}://{gateway_bind_address}")) + } else { + None + } +} + +fn docker_supervisor_networking( + config: &DockerDriverRuntimeConfig, + workload: &bollard::models::ContainerInspectResponse, +) -> Result<(DockerSupervisorNetworkMode, String), Status> { + let sysbox = workload + .host_config + .as_ref() + .and_then(|host| host.runtime.as_deref()) + .is_some_and(|runtime| runtime == "sysbox-runc"); + let mode = match config.supervisor_network_mode { + DockerSupervisorNetworkMode::Auto if sysbox => DockerSupervisorNetworkMode::Bridge, + DockerSupervisorNetworkMode::Auto => DockerSupervisorNetworkMode::Host, + mode => mode, + }; + let endpoint = if mode == DockerSupervisorNetworkMode::Bridge + && !config.supervisor_endpoint_explicit + { + config.bridge_default_endpoint.clone().ok_or_else(|| Status::failed_precondition( + "Docker bridge supervisor cannot reach a Linux gateway bound only to loopback; configure a bridge-reachable gateway bind_address and grpc_endpoint, or use supervisor_network_mode = 'host' when the runtime permits host networking", + ))? + } else { + config.supervisor_grpc_endpoint.clone() + }; + if mode == DockerSupervisorNetworkMode::Bridge { + let url = Url::parse(&endpoint).map_err(|error| { + Status::failed_precondition(format!("invalid Docker supervisor endpoint: {error}")) + })?; + if matches!(url.host(), Some(url::Host::Domain(host)) if host.eq_ignore_ascii_case("localhost")) + || matches!(url.host(), Some(url::Host::Ipv4(ip)) if ip.is_loopback() || ip.is_unspecified()) + || matches!(url.host(), Some(url::Host::Ipv6(ip)) if ip.is_loopback() || ip.is_unspecified()) + { + return Err(Status::failed_precondition( + "Docker bridge supervisor grpc_endpoint must be reachable outside the gateway's loopback namespace; use host.docker.internal or a reachable gateway address", + )); + } + } + Ok((mode, endpoint)) +} + pub(crate) fn docker_guest_tls_paths( docker_config: &DockerComputeConfig, ) -> CoreResult> { diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index bc747312c3..bcb86a62a1 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -185,6 +185,9 @@ fn runtime_config() -> DockerDriverRuntimeConfig { sandbox_binary: Arc::new(b"\x7fELFtest".to_vec()), supervisor_image_id: "sha256:supervisor-test".to_string(), supervisor_grpc_endpoint: "https://host.openshell.internal:8443".to_string(), + supervisor_network_mode: DockerSupervisorNetworkMode::Auto, + supervisor_endpoint_explicit: true, + bridge_default_endpoint: Some("https://host.docker.internal:8443".to_string()), ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(), guest_tls: Some(DockerGuestTlsPaths { ca: PathBuf::from("/tmp/ca.crt"), @@ -2888,7 +2891,11 @@ fn build_container_create_body_disables_docker_networking() { #[test] fn docker_supervisor_uses_host_network() { - let host = docker_supervisor_host_config(Vec::new(), "https://127.0.0.1:17670"); + let host = docker_supervisor_host_config( + Vec::new(), + "https://127.0.0.1:17670", + DockerSupervisorNetworkMode::Host, + ); assert_eq!(host.network_mode.as_deref(), Some("host")); assert_eq!( @@ -2904,7 +2911,11 @@ fn docker_supervisor_uses_host_network() { #[test] fn docker_supervisor_maps_host_aliases_to_the_gateway_address() { - let host = docker_supervisor_host_config(Vec::new(), "https://172.20.0.4:17670"); + let host = docker_supervisor_host_config( + Vec::new(), + "https://172.20.0.4:17670", + DockerSupervisorNetworkMode::Host, + ); assert_eq!( host.extra_hosts, @@ -2921,7 +2932,11 @@ fn docker_supervisor_maps_host_aliases_to_the_gateway_address() { #[test] fn docker_supervisor_leaves_named_gateway_hosts_to_dns() { - let host = docker_supervisor_host_config(Vec::new(), "https://gateway.example.com:17670"); + let host = docker_supervisor_host_config( + Vec::new(), + "https://gateway.example.com:17670", + DockerSupervisorNetworkMode::Host, + ); assert_eq!(host.extra_hosts, None); assert_eq!( @@ -2942,6 +2957,186 @@ fn docker_supervisor_defaults_to_the_primary_loopback_endpoint() { ); } +#[test] +fn docker_supervisor_selects_bridge_for_sysbox_and_honors_operator_mode() { + for (runtime, requested, expected) in [ + ( + Some("sysbox-runc"), + DockerSupervisorNetworkMode::Auto, + DockerSupervisorNetworkMode::Bridge, + ), + ( + Some("runc"), + DockerSupervisorNetworkMode::Auto, + DockerSupervisorNetworkMode::Host, + ), + ( + None, + DockerSupervisorNetworkMode::Auto, + DockerSupervisorNetworkMode::Host, + ), + ( + Some("sysbox-runc"), + DockerSupervisorNetworkMode::Host, + DockerSupervisorNetworkMode::Host, + ), + ( + Some("runc"), + DockerSupervisorNetworkMode::Bridge, + DockerSupervisorNetworkMode::Bridge, + ), + ] { + let mut config = runtime_config(); + config.supervisor_network_mode = requested; + config.supervisor_endpoint_explicit = false; + config.supervisor_grpc_endpoint = "https://127.0.0.1:8443".into(); + let inspected = bollard::models::ContainerInspectResponse { + host_config: Some(HostConfig { + runtime: runtime.map(str::to_string), + ..Default::default() + }), + ..Default::default() + }; + let (mode, endpoint) = docker_supervisor_networking(&config, &inspected).unwrap(); + assert_eq!(mode, expected); + assert_eq!( + endpoint, + if mode == DockerSupervisorNetworkMode::Bridge { + "https://host.docker.internal:8443" + } else { + "https://127.0.0.1:8443" + } + ); + } +} + +#[test] +fn docker_bridge_preserves_explicit_endpoints_and_rejects_loopback() { + let mut config = runtime_config(); + config.supervisor_network_mode = DockerSupervisorNetworkMode::Bridge; + for endpoint in [ + "https://gateway.example.com:9443", + "http://172.20.0.4:8080", + "https://[fd00::2]:9443", + ] { + config.supervisor_grpc_endpoint = endpoint.into(); + let (_, selected) = docker_supervisor_networking( + &config, + &bollard::models::ContainerInspectResponse::default(), + ) + .unwrap(); + assert_eq!(selected, endpoint); + } + for endpoint in [ + "https://127.0.0.1:8443", + "http://localhost:8080", + "https://[::1]:8443", + "http://0.0.0.0:8080", + "http://[::]:8080", + ] { + config.supervisor_grpc_endpoint = endpoint.into(); + let error = docker_supervisor_networking( + &config, + &bollard::models::ContainerInspectResponse::default(), + ) + .unwrap_err(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert!(error.message().contains("grpc_endpoint")); + } +} + +#[test] +fn docker_bridge_defaults_respect_desktop_and_linux_listener_reachability() { + assert_eq!( + default_docker_bridge_endpoint("127.0.0.1:17670".parse().unwrap(), true, true).as_deref(), + Some("https://host.docker.internal:17670") + ); + assert_eq!( + default_docker_bridge_endpoint("0.0.0.0:8080".parse().unwrap(), false, false).as_deref(), + Some("http://host.docker.internal:8080") + ); + assert_eq!( + default_docker_bridge_endpoint("192.168.1.2:8080".parse().unwrap(), true, false).as_deref(), + Some("https://192.168.1.2:8080") + ); + assert_eq!( + default_docker_bridge_endpoint("[fd00::2]:8080".parse().unwrap(), true, false).as_deref(), + Some("https://[fd00::2]:8080") + ); + assert!( + default_docker_bridge_endpoint("127.0.0.1:17670".parse().unwrap(), true, false).is_none() + ); + let mut config = runtime_config(); + config.supervisor_network_mode = DockerSupervisorNetworkMode::Bridge; + config.supervisor_endpoint_explicit = false; + config.bridge_default_endpoint = None; + let error = docker_supervisor_networking( + &config, + &bollard::models::ContainerInspectResponse::default(), + ) + .unwrap_err(); + assert!(error.message().contains("bind_address")); +} + +#[test] +fn docker_bridge_host_aliases_are_independent_of_gateway_endpoint() { + let host = docker_supervisor_host_config( + Vec::new(), + "https://gateway.example.com:9443", + DockerSupervisorNetworkMode::Bridge, + ); + assert_eq!(host.network_mode.as_deref(), Some("bridge")); + assert_eq!( + host.extra_hosts, + Some(vec![ + "host.openshell.internal:host-gateway".into(), + "host.docker.internal:host-gateway".into() + ]) + ); + assert_eq!(host.cap_drop, Some(vec!["ALL".into()])); + assert_eq!(host.cap_add, None); + assert_eq!( + host.security_opt, + Some(vec!["no-new-privileges:true".into()]) + ); + assert_eq!( + build_container_create_body(&test_sandbox(), &runtime_config()) + .unwrap() + .host_config + .unwrap() + .network_mode + .as_deref(), + Some("none") + ); +} + +#[test] +fn docker_supervisor_network_mode_is_an_operator_config_enum() { + assert_eq!( + toml::from_str::("") + .unwrap() + .supervisor_network_mode, + DockerSupervisorNetworkMode::Auto + ); + for (value, expected) in [ + ("auto", DockerSupervisorNetworkMode::Auto), + ("host", DockerSupervisorNetworkMode::Host), + ("bridge", DockerSupervisorNetworkMode::Bridge), + ] { + let config: DockerComputeConfig = + toml::from_str(&format!("supervisor_network_mode = '{value}'")).unwrap(); + assert_eq!(config.supervisor_network_mode, expected); + } + assert!(toml::from_str::("supervisor_network_mode = 'none'").is_err()); + assert!( + serde_json::from_value::(serde_json::json!({ + "supervisor_network_mode": "host" + })) + .is_err(), + "workloads cannot select supervisor networking" + ); +} + #[test] fn build_container_create_body_limits_writable_runtime_storage_to_supervisor_ca() { let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); diff --git a/crates/openshell-supervisor/src/main.rs b/crates/openshell-supervisor/src/main.rs index 4bfb469891..8026969e5e 100644 --- a/crates/openshell-supervisor/src/main.rs +++ b/crates/openshell-supervisor/src/main.rs @@ -113,6 +113,10 @@ struct Args { #[arg(long)] backend_descriptor_file: Option, + /// Resolve the Docker host alias from this supervisor's driver-owned hosts file. + #[arg(long)] + host_gateway_from_hosts: bool, + /// Protected gateway-issued credentials for this exact sandbox launch. #[arg(long)] auth_bundle_file: Option, @@ -180,14 +184,68 @@ fn backend_descriptor(args: &Args) -> Result { let path = args.backend_descriptor_file.as_deref().ok_or_else(|| { miette::miette!("--backend-descriptor-file is required for --role=isolation-backend") })?; - let payload = std::fs::read(path) + let mut payload = std::fs::read(path) .map_err(|error| miette::miette!("read backend descriptor {}: {error}", path.display()))?; + if args.host_gateway_from_hosts { + let hosts = std::fs::read_to_string("/etc/hosts") + .map_err(|error| miette::miette!("read supervisor Docker host aliases: {error}"))?; + let ip = docker_host_gateway_from_hosts(&hosts)?; + let mut descriptor: openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor = + serde_json::from_slice(&payload).into_diagnostic()?; + // Resolve before any workload is admitted, from the supervisor's mount + // namespace. Workload /etc/hosts and DNS never supply this trusted IP. + descriptor.host_gateway_ip = Some(ip); + payload = serde_json::to_vec(&descriptor).into_diagnostic()?; + } Ok(BackendDescriptor { backend_name: openshell_sandbox_backend::BACKEND_NAME.to_string(), payload, }) } +fn docker_host_gateway_from_hosts(hosts: &str) -> Result { + let mut ips = std::collections::BTreeSet::new(); + for line in hosts.lines() { + let mut fields = line.split('#').next().unwrap_or("").split_whitespace(); + let Some(ip) = fields + .next() + .and_then(|value| value.parse::().ok()) + else { + continue; + }; + if fields.any(|host| host.eq_ignore_ascii_case("host.openshell.internal")) { + ips.insert(ip); + } + } + // Docker may expand host-gateway to both families. Prefer IPv4 consistently. + let ipv4 = ips + .iter() + .copied() + .filter(std::net::IpAddr::is_ipv4) + .collect::>(); + let candidates = if ipv4.is_empty() { + ips.into_iter().collect() + } else { + ipv4 + }; + let [ip] = candidates.as_slice() else { + return Err(miette::miette!( + "expected one Docker host-gateway address per family in supervisor /etc/hosts" + )); + }; + if ip.is_loopback() + || ip.is_unspecified() + || ip.is_multicast() + || matches!(ip, std::net::IpAddr::V4(ip) if ip.is_link_local() || ip.is_broadcast()) + || matches!(ip, std::net::IpAddr::V6(ip) if ip.is_unicast_link_local() || ip.to_ipv4_mapped().is_some()) + { + return Err(miette::miette!( + "unsafe Docker host-gateway address in supervisor /etc/hosts" + )); + } + Ok(*ip) +} + fn auth_bundle(args: &Args) -> Result { let path = args.auth_bundle_file.as_deref().ok_or_else(|| { miette::miette!("--auth-bundle-file is required for --role=isolation-backend") @@ -232,6 +290,7 @@ fn validate_role_arguments(args: &Args) -> Result<()> { } SupervisorRole::NetworkProxy => { if args.backend_descriptor_file.is_some() + || args.host_gateway_from_hosts || args.auth_bundle_file.is_some() || args.sandbox_id.is_some() || args.sandbox.is_some() @@ -482,6 +541,33 @@ fn main() -> Result<()> { mod tests { use super::*; + #[test] + fn docker_host_gateway_uses_supervisor_alias_and_prefers_ipv4() { + let hosts = "127.0.0.1 localhost\n192.168.65.254 host.openshell.internal host.docker.internal\nfd00::1 host.openshell.internal\n192.168.65.254 host.openshell.internal # duplicate\n"; + assert_eq!( + docker_host_gateway_from_hosts(hosts).unwrap(), + "192.168.65.254".parse::().unwrap() + ); + assert_eq!( + docker_host_gateway_from_hosts("fd00::1 host.openshell.internal").unwrap(), + "fd00::1".parse::().unwrap() + ); + } + + #[test] + fn docker_host_gateway_rejects_missing_ambiguous_and_unsafe_aliases() { + for hosts in [ + "192.168.65.254 host.docker.internal", + "192.168.65.254 host.openshell.internal\n172.17.0.1 host.openshell.internal", + "127.0.0.1 host.openshell.internal", + "0.0.0.0 host.openshell.internal", + "169.254.169.254 host.openshell.internal", + "::ffff:127.0.0.1 host.openshell.internal", + ] { + assert!(docker_host_gateway_from_hosts(hosts).is_err(), "{hosts}"); + } + } + #[test] fn isolation_backend_is_the_default_role() { let directory = tempfile::tempdir().expect("temporary runtime descriptor directory"); diff --git a/docs/about/support-matrix.mdx b/docs/about/support-matrix.mdx index 9cace1a708..e5a491e613 100644 --- a/docs/about/support-matrix.mdx +++ b/docs/about/support-matrix.mdx @@ -112,7 +112,7 @@ The gateway can manage sandboxes through several runtimes. | Runtime | Status | Notes | |---|---|---| -| Docker | Supported for local development and single-machine gateways. | Requires Docker Desktop or Docker Engine on the gateway host. | +| Docker | Supported for local development and single-machine gateways. | Requires an accessible Docker API socket. Supervisor networking defaults to host, or bridge for detected Sysbox/ECI workloads. See [Docker runtime setup](/how-it-works/sandboxes/runtimes#docker-driver) for endpoint and isolation requirements. | | Podman | Supported for rootless local and workstation workflows. | Requires a Podman-compatible socket and rootless networking setup. | | Kubernetes | Supported through the [OpenShell Helm chart](https://github.com/NVIDIA/OpenShell/blob/main/deploy/helm/openshell/README.md). | Requires a Kubernetes cluster supplied by the operator. | | MicroVM | Supported for VM-backed sandboxes. | Uses the VM compute driver and libkrun-based runtime. | diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 3b3c3aaaf0..73fdd6a2ee 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -910,9 +910,11 @@ default_image = "nvcr.io/nvidia/base/ubuntu:24.04" image_pull_policy = "if_not_present" # Value assigned to the openshell.sandbox_namespace label on sandbox containers. sandbox_label = "docker-dev" -# Optional override. When omitted, the host-networked supervisor uses the -# gateway's primary loopback endpoint. -grpc_endpoint = "https://127.0.0.1:17670" +# auto uses bridge for Sysbox/ECI workloads, otherwise host networking. +# Operator-owned; sandbox driver config cannot override this choice. +supervisor_network_mode = "auto" +# Optional override. Omit for automatic host or bridge callback selection. +# grpc_endpoint = "https://gateway.example.com:17670" # Workload-side runtime. Defaults to the gateway version. # sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:" # Supervisor runtime. Defaults to the gateway version. @@ -948,6 +950,35 @@ provider_spiffe_workload_api_socket = "/run/spire/agent.sock" Use `sandbox_label` for Docker configurations. The legacy `sandbox_namespace` key is rejected. +`supervisor_network_mode` accepts `auto` (the default), `host`, or `bridge`. +On each supervisor launch, including restart, `auto` inspects the existing +workload container's runtime. `sysbox-runc`, used by Docker Desktop Enhanced +Container Isolation (ECI), selects bridge networking; other runtimes select +host networking. This heuristic creates no diagnostic container. An explicit +mode overrides it. The workload always uses `network=none`. + +When `grpc_endpoint` is omitted, host mode uses the gateway's primary loopback +endpoint. Bridge mode uses `host.docker.internal` and the gateway's bind port +on Docker Desktop or for a wildcard listener. On native Linux, a concrete +non-loopback listener uses its bind address. A Linux gateway bound only to +loopback requires a bridge-reachable listener and endpoint; OpenShell does not +automatically widen the listener. A gateway container with a remapped published +port requires an explicit endpoint using that host port. Explicit endpoints +retain their scheme, host, port, and TLS verification; bridge mode rejects +loopback and unspecified endpoint addresses. HTTPS certificates must cover +the selected endpoint hostname or IP. + +In bridge mode, Docker resolves `host-gateway` for the supervisor's +`host.openshell.internal` and `host.docker.internal` aliases independently of +the gateway endpoint. The supervisor pins that driver-owned address before +admitting the workload. On native Linux, host services listening only on +loopback are unreachable through the bridge; expose them on a bridge-reachable +interface or use host mode when permitted. Host mode on Docker Desktop requires +host networking enabled and ECI disabled. Bridge selection removes that +networking conflict but does not establish full ECI compatibility: the runtime +must still pass OpenShell's Landlock and seccomp qualification. A containerized +gateway under ECI also requires an administrator-approved Docker socket exception. + Docker accepts `http://` and `https://` proxy URLs in explicit `scheme://host:port` form. `no_proxy` bypasses only the corporate proxy; OpenShell policy still applies. Proxy URLs cannot embed credentials. Supply a diff --git a/docs/how-it-works/gateways/overview.mdx b/docs/how-it-works/gateways/overview.mdx index ee1f735fa3..fe12dc6b1f 100644 --- a/docs/how-it-works/gateways/overview.mdx +++ b/docs/how-it-works/gateways/overview.mdx @@ -26,7 +26,7 @@ A gateway provisions sandboxes through the compute driver configured for that ga | Runtime | Where sandboxes run | Best for | |---|---|---| -| Docker | Containers on the gateway host. | Solo development, quick iteration, and single-machine gateways. | +| Docker | Containers on the Docker daemon host. | Solo development, quick iteration, and single-machine gateways. | | Podman | Rootless containers on the gateway host. | Workstations that avoid a rootful Docker daemon. | | Kubernetes | Pods in an operator-managed cluster. | Shared clusters and cloud environments. | | MicroVM | VM-backed sandboxes. | Workflows that need VM-backed isolation. | diff --git a/docs/how-it-works/sandboxes/runtimes.mdx b/docs/how-it-works/sandboxes/runtimes.mdx index f52a571bd2..8d75749c91 100644 --- a/docs/how-it-works/sandboxes/runtimes.mdx +++ b/docs/how-it-works/sandboxes/runtimes.mdx @@ -82,7 +82,7 @@ To pin specific GPUs, pass `cdi_devices` for Docker or Podman, or `gpu_device_id ## Docker Driver -[Docker](https://www.docker.com/get-started/) runs sandboxes as containers on the gateway host. Docker is also required to build sandbox images from local directories or Dockerfiles. +[Docker](https://www.docker.com/get-started/) runs sandboxes as containers on the Docker daemon host. The gateway accesses the daemon through its configured Unix socket. Docker is also required to build sandbox images from local directories or Dockerfiles. ```toml [openshell.gateway] @@ -92,9 +92,13 @@ compute_driver = "docker" socket_path = "/var/run/docker.sock" ``` -Common options in `[openshell.drivers.docker]` are `socket_path`, `grpc_endpoint`, `sandbox_runtime_image`, `supervisor_image`, `image_pull_policy`, and `sandbox_pids_limit`. When `socket_path` is unset, the driver uses the socket found by auto-detection. +Common options in `[openshell.drivers.docker]` are `socket_path`, `grpc_endpoint`, `supervisor_network_mode`, `sandbox_runtime_image`, `supervisor_image`, `image_pull_policy`, and `sandbox_pids_limit`. When `socket_path` is unset, the driver uses the socket found by auto-detection. -Docker Desktop must have host networking enabled, and it cannot use Enhanced Container Isolation. Set `grpc_endpoint` when sandboxes cannot reach the gateway on host loopback. For GPU sandboxes, configure Docker CDI before starting the gateway. +`supervisor_network_mode` defaults to `auto`: the driver uses bridge networking for workloads whose inspected runtime is `sysbox-runc` (Docker Desktop Enhanced Container Isolation), and host networking otherwise. Set `host` or `bridge` to override the heuristic. The workload always retains `network=none`. + +Host mode on Docker Desktop requires host networking enabled and Enhanced Container Isolation disabled. Bridge mode automatically uses `host.docker.internal` for a gateway on the Desktop host. On native Linux, the gateway and host services must listen on bridge-reachable addresses. Set `grpc_endpoint` for remote gateways, remapped published ports, or other custom layouts. Endpoint selection preserves TLS verification and does not change the gateway's listener. See [Docker configuration](/how-it-works/gateways/configuration#docker). + +Bridge selection does not establish full Enhanced Container Isolation compatibility; the sandbox must still pass its Landlock and seccomp qualification. A gateway running in an ECI container requires an administrator-approved Docker socket exception. For GPU sandboxes, configure Docker CDI before starting the gateway. ### Docker Corporate Proxy Egress diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index 6b1fc73723..d779904e59 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -83,6 +83,11 @@ name = "docker_corporate_proxy" path = "tests/docker_corporate_proxy.rs" required-features = ["e2e-docker"] +[[test]] +name = "docker_supervisor_networking" +path = "tests/docker_supervisor_networking.rs" +required-features = ["e2e-docker"] + [[test]] name = "driver_config_volume" path = "tests/driver_config_volume.rs" diff --git a/e2e/rust/tests/docker_supervisor_networking.rs b/e2e/rust/tests/docker_supervisor_networking.rs new file mode 100644 index 0000000000..88b825b4bc --- /dev/null +++ b/e2e/rust/tests/docker_supervisor_networking.rs @@ -0,0 +1,127 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +#![cfg(feature = "e2e-docker")] + +//! Run with both the default auto mode and forced bridge mode to exercise +//! authenticated gateway callbacks, interactive access, and supervisor restart. + +use std::time::Duration; + +use openshell_e2e::harness::cli::{run_cli, wait_for_sandbox_phase}; +use openshell_e2e::harness::sandbox::SandboxGuard; + +async fn inspect_role(name: &str, role: &str) -> serde_json::Value { + let listed = tokio::process::Command::new("docker") + .args([ + "ps", + "-aq", + "--filter", + &format!("label=openshell.ai/sandbox-name={name}"), + "--filter", + &format!("label=openshell.ai/isolation-role={role}"), + ]) + .output() + .await + .expect("list sandbox containers"); + assert!(listed.status.success()); + let id = String::from_utf8(listed.stdout).unwrap(); + let id = id.trim(); + assert!(!id.is_empty(), "missing {role} container for {name}"); + let inspected = tokio::process::Command::new("docker") + .args(["inspect", id]) + .output() + .await + .expect("inspect sandbox container"); + assert!(inspected.status.success()); + serde_json::from_slice::>(&inspected.stdout) + .unwrap() + .remove(0) +} + +#[tokio::test] +async fn supervisor_networking_preserves_boundary_and_survives_restart() { + let policy = tempfile::NamedTempFile::new().expect("create policy file"); + std::fs::write( + policy.path(), + r"version: 1 +filesystem_policy: + include_workdir: true + read_only: [/usr, /bin, /lib, /proc, /dev/urandom, /app, /etc, /var/log] + read_write: [/sandbox, /tmp, /dev/null] +landlock: + compatibility: best_effort +network_policies: + host_gateway: + name: host_gateway + endpoints: + - {host: host.openshell.internal, port: 8080, protocol: tcp} + binaries: + - {path: /usr/bin/getent} +", + ) + .expect("write policy file"); + let mut sandbox = SandboxGuard::create(&["--policy", policy.path().to_str().unwrap()]) + .await + .expect("create sandbox"); + let workload = inspect_role(&sandbox.name, "sandbox").await; + let requested = std::env::var("OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE") + .unwrap_or_else(|_| "auto".into()); + let expected = if requested == "bridge" + || (requested == "auto" && workload["HostConfig"]["Runtime"] == "sysbox-runc") + { + "bridge" + } else { + "host" + }; + assert_eq!(workload["HostConfig"]["NetworkMode"], "none"); + assert!( + workload["NetworkSettings"]["Networks"] + .as_object() + .is_none_or(|networks| networks.keys().all(|name| name == "none")) + ); + let supervisor = inspect_role(&sandbox.name, "supervisor").await; + assert_eq!(supervisor["HostConfig"]["NetworkMode"], expected); + assert!( + sandbox + .exec(&["sh", "-c", "echo networking-ready"]) + .await + .unwrap() + .contains("networking-ready") + ); + assert!( + !sandbox + .exec(&["/usr/bin/getent", "hosts", "host.openshell.internal"]) + .await + .expect("resolve policy-approved host alias") + .trim() + .is_empty() + ); + let (output, code) = run_cli(&["sandbox", "stop", &sandbox.name]).await; + assert_eq!(code, 0, "{output}"); + wait_for_sandbox_phase(&sandbox.name, "Stopped", Duration::from_secs(60)) + .await + .unwrap(); + let (output, code) = run_cli(&["sandbox", "start", &sandbox.name]).await; + assert_eq!(code, 0, "{output}"); + assert!( + sandbox + .exec(&["sh", "-c", "echo networking-restarted"]) + .await + .unwrap() + .contains("networking-restarted") + ); + assert!( + !sandbox + .exec(&["/usr/bin/getent", "hosts", "host.openshell.internal"]) + .await + .expect("resolve host alias after restart") + .trim() + .is_empty() + ); + assert_eq!( + inspect_role(&sandbox.name, "supervisor").await["HostConfig"]["NetworkMode"], + expected + ); + sandbox.cleanup().await; +} diff --git a/e2e/with-docker-gateway.sh b/e2e/with-docker-gateway.sh index 4b96693ab1..c569779d39 100755 --- a/e2e/with-docker-gateway.sh +++ b/e2e/with-docker-gateway.sh @@ -22,6 +22,8 @@ # SUPERVISOR_IMAGE=... (common test-wrapper override) # OPENSHELL_SUPERVISOR_IMAGE=... (existing compatibility override) # OPENSHELL_DOCKER_SUPERVISOR_IMAGE=... (Docker-specific override) +# OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE=auto|host|bridge +# OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT=1 omits the callback override. # # The default sandbox image uses a mutable tag. This wrapper refreshes it # before starting the gateway, while the Docker driver defaults to @@ -569,6 +571,11 @@ if connect_current_container_to_docker_network "${DOCKER_NETWORK_NAME}"; then SUPERVISOR_GATEWAY_HOST="${GATEWAY_HOST_ALIAS_IP}" else GATEWAY_HOST_ALIAS_IP="" + if [ "${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-auto}" = "bridge" ]; then + # This ephemeral test gateway deliberately admits bridge-side connections. + GATEWAY_BIND_IP="0.0.0.0" + SUPERVISOR_GATEWAY_HOST="host.docker.internal" + fi fi PKI_DIR="${WORKDIR}/pki" @@ -620,7 +627,10 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" else printf 'allow_driver_config = true\n' printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")" - printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" + if [ "${OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT:-0}" != "1" ]; then + printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" + fi + printf 'supervisor_network_mode = %s\n' "$(toml_string "${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-auto}")" printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")" printf 'image_pull_policy = %s\n' "$(toml_string "${SANDBOX_IMAGE_PULL_POLICY}")" printf 'enable_bind_mounts = true\n' @@ -634,7 +644,10 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then { printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")" - printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" + if [ "${OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT:-0}" != "1" ]; then + printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" + fi + printf 'supervisor_network_mode = %s\n' "$(toml_string "${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-auto}")" printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")" printf 'image_pull_policy = %s\n' "$(toml_string "${SANDBOX_IMAGE_PULL_POLICY}")" printf 'guest_tls_ca = %s\n' "$(toml_string "${PKI_DIR}/ca.crt")" diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 6382428946..20447b1391 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -287,9 +287,19 @@ Common findings: - Sandbox never registers: check gateway logs and the supervisor's gateway endpoint. - Calls to an external tool server fail while the sandbox is Ready: inspect `Tool server connections` in `openshell sandbox get `. For configured MCP-over-HTTP endpoints, JSON output exposes each address together with `last_result` and `last_reported_at` in `endpoint_statuses`. Select the endpoint by host, path, and ports, then check the reported failure boundary. `last_reported_at` records gateway acceptance time and can advance when retained evidence is accepted after a reset. Results do not expire or prove current availability; `HttpResponseReceived` can still contain a tool error. If several paths share a host and port, a failure before the path is known remains in logs. Verify the actual operation when current tool availability matters. - On Docker Desktop, repeated `Policy fetch failed after 5 attempts` messages - can mean host networking is disabled. Enable host networking in Docker - Desktop, ensure Enhanced Container Isolation is disabled, and verify the - gateway's primary endpoint is reachable from a host-networked container. + can mean the supervisor's selected network mode cannot reach the gateway. + Check the gateway's `Selected Docker supervisor networking` log. The default + `supervisor_network_mode = "auto"` selects bridge for inspected `sysbox-runc` + workloads (ECI), and host otherwise. Host mode requires Desktop host + networking enabled and ECI disabled; set `bridge` explicitly when neither + feature is enabled. Bridge mode uses `host.docker.internal` for the Desktop + host and requires a bridge-reachable listener on native Linux. Explicit + `grpc_endpoint` values must be reachable in the selected mode, including any + remapped published port, with matching TLS certificate names. Mode selection + does not bypass Landlock/seccomp qualification. A gateway container under ECI + also needs an administrator-approved Docker socket exception. See the + published [Docker configuration](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration.md) + for the current networking options. - Sandbox runtime image exits before printing `openshell-sandbox --version`: verify the configured image contains a static executable at `/openshell-sandbox`. - A sandbox with explicit `protocol: tcp` endpoints fails before workload readiness: confirm the selected isolation backend advertises TCP mediation, then inspect the sandbox and supervisor logs for protected-channel setup or listener failures. A driver that cannot supply the required outer egress fence and authenticated runtime channel must reject the policy before starting the agent. - Supervisor runtime validation fails: verify `supervisor_image` contains an `/openshell-supervisor` executable from the same release as the sandbox runtime, and that the dynamic loader and shared libraries it links against are available inside that image. `docker run --rm --network none --entrypoint /openshell-supervisor --version` should print that release; a `no such file or directory` error for a binary that exists means the loader or a library is missing. The supervisor runs from its own image and does not need to be static; only `/openshell-sandbox` must be. diff --git a/tasks/test.toml b/tasks/test.toml index c45826f1db..0ad5c59177 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -290,6 +290,14 @@ description = "Run Docker conformance and Rust e2e tests against a standalone ga depends = ["e2e:conformance:build"] run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh" +["e2e:docker:supervisor-networking"] +description = "Exercise Docker supervisor networking and restart in auto and bridge modes" +depends = ["e2e:workload:build"] +run = [ + "e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_supervisor_networking", + "OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE=bridge OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT=1 e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_supervisor_networking", +] + ["e2e:mechanistic-existing-endpoint"] description = "Run #2821 existing inspected-endpoint auto-approval regression" run = [ From b77155e6ca29101a22b4394a8990c0c9a7c4495f Mon Sep 17 00:00:00 2001 From: Drew Newberry Date: Thu, 1 Oct 2026 12:30:48 -0700 Subject: [PATCH 2/4] test(docker): use automatic networking in e2e Signed-off-by: Drew Newberry --- e2e/rust/tests/docker_supervisor_networking.rs | 8 ++------ e2e/with-docker-gateway.sh | 14 +++----------- tasks/test.toml | 7 ++----- 3 files changed, 7 insertions(+), 22 deletions(-) diff --git a/e2e/rust/tests/docker_supervisor_networking.rs b/e2e/rust/tests/docker_supervisor_networking.rs index 88b825b4bc..0c1bfe01a4 100644 --- a/e2e/rust/tests/docker_supervisor_networking.rs +++ b/e2e/rust/tests/docker_supervisor_networking.rs @@ -3,7 +3,7 @@ #![cfg(feature = "e2e-docker")] -//! Run with both the default auto mode and forced bridge mode to exercise +//! Exercise automatic networking selection using the workload runtime, along with //! authenticated gateway callbacks, interactive access, and supervisor restart. use std::time::Duration; @@ -65,11 +65,7 @@ network_policies: .await .expect("create sandbox"); let workload = inspect_role(&sandbox.name, "sandbox").await; - let requested = std::env::var("OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE") - .unwrap_or_else(|_| "auto".into()); - let expected = if requested == "bridge" - || (requested == "auto" && workload["HostConfig"]["Runtime"] == "sysbox-runc") - { + let expected = if workload["HostConfig"]["Runtime"] == "sysbox-runc" { "bridge" } else { "host" diff --git a/e2e/with-docker-gateway.sh b/e2e/with-docker-gateway.sh index c569779d39..512b632697 100755 --- a/e2e/with-docker-gateway.sh +++ b/e2e/with-docker-gateway.sh @@ -22,8 +22,6 @@ # SUPERVISOR_IMAGE=... (common test-wrapper override) # OPENSHELL_SUPERVISOR_IMAGE=... (existing compatibility override) # OPENSHELL_DOCKER_SUPERVISOR_IMAGE=... (Docker-specific override) -# OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE=auto|host|bridge -# OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT=1 omits the callback override. # # The default sandbox image uses a mutable tag. This wrapper refreshes it # before starting the gateway, while the Docker driver defaults to @@ -571,11 +569,6 @@ if connect_current_container_to_docker_network "${DOCKER_NETWORK_NAME}"; then SUPERVISOR_GATEWAY_HOST="${GATEWAY_HOST_ALIAS_IP}" else GATEWAY_HOST_ALIAS_IP="" - if [ "${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-auto}" = "bridge" ]; then - # This ephemeral test gateway deliberately admits bridge-side connections. - GATEWAY_BIND_IP="0.0.0.0" - SUPERVISOR_GATEWAY_HOST="host.docker.internal" - fi fi PKI_DIR="${WORKDIR}/pki" @@ -627,10 +620,10 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" else printf 'allow_driver_config = true\n' printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")" - if [ "${OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT:-0}" != "1" ]; then + # Container jobs need their bridge address; host runs exercise discovery. + if [ -n "${GATEWAY_HOST_ALIAS_IP}" ]; then printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" fi - printf 'supervisor_network_mode = %s\n' "$(toml_string "${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-auto}")" printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")" printf 'image_pull_policy = %s\n' "$(toml_string "${SANDBOX_IMAGE_PULL_POLICY}")" printf 'enable_bind_mounts = true\n' @@ -644,10 +637,9 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then { printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")" - if [ "${OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT:-0}" != "1" ]; then + if [ -n "${GATEWAY_HOST_ALIAS_IP}" ]; then printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" fi - printf 'supervisor_network_mode = %s\n' "$(toml_string "${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-auto}")" printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")" printf 'image_pull_policy = %s\n' "$(toml_string "${SANDBOX_IMAGE_PULL_POLICY}")" printf 'guest_tls_ca = %s\n' "$(toml_string "${PKI_DIR}/ca.crt")" diff --git a/tasks/test.toml b/tasks/test.toml index 0ad5c59177..8295950d79 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -291,12 +291,9 @@ depends = ["e2e:conformance:build"] run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh" ["e2e:docker:supervisor-networking"] -description = "Exercise Docker supervisor networking and restart in auto and bridge modes" +description = "Exercise automatic Docker supervisor networking and restart" depends = ["e2e:workload:build"] -run = [ - "e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_supervisor_networking", - "OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE=bridge OPENSHELL_E2E_DOCKER_AUTOMATIC_ENDPOINT=1 e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_supervisor_networking", -] +run = "e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_supervisor_networking" ["e2e:mechanistic-existing-endpoint"] description = "Run #2821 existing inspected-endpoint auto-approval regression" From b2a0214fa5f80771275149632964fa5f0d32aa67 Mon Sep 17 00:00:00 2001 From: Drew Newberry Date: Thu, 1 Oct 2026 12:40:25 -0700 Subject: [PATCH 3/4] test(docker): rely on existing e2e networking coverage Signed-off-by: Drew Newberry --- e2e/rust/Cargo.toml | 5 - .../tests/docker_supervisor_networking.rs | 123 ------------------ tasks/test.toml | 5 - 3 files changed, 133 deletions(-) delete mode 100644 e2e/rust/tests/docker_supervisor_networking.rs diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index d779904e59..6b1fc73723 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -83,11 +83,6 @@ name = "docker_corporate_proxy" path = "tests/docker_corporate_proxy.rs" required-features = ["e2e-docker"] -[[test]] -name = "docker_supervisor_networking" -path = "tests/docker_supervisor_networking.rs" -required-features = ["e2e-docker"] - [[test]] name = "driver_config_volume" path = "tests/driver_config_volume.rs" diff --git a/e2e/rust/tests/docker_supervisor_networking.rs b/e2e/rust/tests/docker_supervisor_networking.rs deleted file mode 100644 index 0c1bfe01a4..0000000000 --- a/e2e/rust/tests/docker_supervisor_networking.rs +++ /dev/null @@ -1,123 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -#![cfg(feature = "e2e-docker")] - -//! Exercise automatic networking selection using the workload runtime, along with -//! authenticated gateway callbacks, interactive access, and supervisor restart. - -use std::time::Duration; - -use openshell_e2e::harness::cli::{run_cli, wait_for_sandbox_phase}; -use openshell_e2e::harness::sandbox::SandboxGuard; - -async fn inspect_role(name: &str, role: &str) -> serde_json::Value { - let listed = tokio::process::Command::new("docker") - .args([ - "ps", - "-aq", - "--filter", - &format!("label=openshell.ai/sandbox-name={name}"), - "--filter", - &format!("label=openshell.ai/isolation-role={role}"), - ]) - .output() - .await - .expect("list sandbox containers"); - assert!(listed.status.success()); - let id = String::from_utf8(listed.stdout).unwrap(); - let id = id.trim(); - assert!(!id.is_empty(), "missing {role} container for {name}"); - let inspected = tokio::process::Command::new("docker") - .args(["inspect", id]) - .output() - .await - .expect("inspect sandbox container"); - assert!(inspected.status.success()); - serde_json::from_slice::>(&inspected.stdout) - .unwrap() - .remove(0) -} - -#[tokio::test] -async fn supervisor_networking_preserves_boundary_and_survives_restart() { - let policy = tempfile::NamedTempFile::new().expect("create policy file"); - std::fs::write( - policy.path(), - r"version: 1 -filesystem_policy: - include_workdir: true - read_only: [/usr, /bin, /lib, /proc, /dev/urandom, /app, /etc, /var/log] - read_write: [/sandbox, /tmp, /dev/null] -landlock: - compatibility: best_effort -network_policies: - host_gateway: - name: host_gateway - endpoints: - - {host: host.openshell.internal, port: 8080, protocol: tcp} - binaries: - - {path: /usr/bin/getent} -", - ) - .expect("write policy file"); - let mut sandbox = SandboxGuard::create(&["--policy", policy.path().to_str().unwrap()]) - .await - .expect("create sandbox"); - let workload = inspect_role(&sandbox.name, "sandbox").await; - let expected = if workload["HostConfig"]["Runtime"] == "sysbox-runc" { - "bridge" - } else { - "host" - }; - assert_eq!(workload["HostConfig"]["NetworkMode"], "none"); - assert!( - workload["NetworkSettings"]["Networks"] - .as_object() - .is_none_or(|networks| networks.keys().all(|name| name == "none")) - ); - let supervisor = inspect_role(&sandbox.name, "supervisor").await; - assert_eq!(supervisor["HostConfig"]["NetworkMode"], expected); - assert!( - sandbox - .exec(&["sh", "-c", "echo networking-ready"]) - .await - .unwrap() - .contains("networking-ready") - ); - assert!( - !sandbox - .exec(&["/usr/bin/getent", "hosts", "host.openshell.internal"]) - .await - .expect("resolve policy-approved host alias") - .trim() - .is_empty() - ); - let (output, code) = run_cli(&["sandbox", "stop", &sandbox.name]).await; - assert_eq!(code, 0, "{output}"); - wait_for_sandbox_phase(&sandbox.name, "Stopped", Duration::from_secs(60)) - .await - .unwrap(); - let (output, code) = run_cli(&["sandbox", "start", &sandbox.name]).await; - assert_eq!(code, 0, "{output}"); - assert!( - sandbox - .exec(&["sh", "-c", "echo networking-restarted"]) - .await - .unwrap() - .contains("networking-restarted") - ); - assert!( - !sandbox - .exec(&["/usr/bin/getent", "hosts", "host.openshell.internal"]) - .await - .expect("resolve host alias after restart") - .trim() - .is_empty() - ); - assert_eq!( - inspect_role(&sandbox.name, "supervisor").await["HostConfig"]["NetworkMode"], - expected - ); - sandbox.cleanup().await; -} diff --git a/tasks/test.toml b/tasks/test.toml index 8295950d79..c45826f1db 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -290,11 +290,6 @@ description = "Run Docker conformance and Rust e2e tests against a standalone ga depends = ["e2e:conformance:build"] run = "OPENSHELL_CONFORMANCE_BIN=\"${OPENSHELL_CONFORMANCE_BIN:-$PWD/target/debug/openshell-conformance}\" e2e/rust/e2e-docker.sh" -["e2e:docker:supervisor-networking"] -description = "Exercise automatic Docker supervisor networking and restart" -depends = ["e2e:workload:build"] -run = "e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test docker_supervisor_networking" - ["e2e:mechanistic-existing-endpoint"] description = "Run #2821 existing inspected-endpoint auto-approval regression" run = [ From d6335628db3087455e7feebeea56898de91e4890 Mon Sep 17 00:00:00 2001 From: Drew Newberry Date: Thu, 1 Oct 2026 12:59:11 -0700 Subject: [PATCH 4/4] docs(docker): shorten supervisor networking configuration Signed-off-by: Drew Newberry --- docs/how-it-works/gateways/configuration.mdx | 42 +++++++------------- 1 file changed, 15 insertions(+), 27 deletions(-) diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 73fdd6a2ee..ca840409cd 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -951,33 +951,21 @@ Use `sandbox_label` for Docker configurations. The legacy `sandbox_namespace` key is rejected. `supervisor_network_mode` accepts `auto` (the default), `host`, or `bridge`. -On each supervisor launch, including restart, `auto` inspects the existing -workload container's runtime. `sysbox-runc`, used by Docker Desktop Enhanced -Container Isolation (ECI), selects bridge networking; other runtimes select -host networking. This heuristic creates no diagnostic container. An explicit -mode overrides it. The workload always uses `network=none`. - -When `grpc_endpoint` is omitted, host mode uses the gateway's primary loopback -endpoint. Bridge mode uses `host.docker.internal` and the gateway's bind port -on Docker Desktop or for a wildcard listener. On native Linux, a concrete -non-loopback listener uses its bind address. A Linux gateway bound only to -loopback requires a bridge-reachable listener and endpoint; OpenShell does not -automatically widen the listener. A gateway container with a remapped published -port requires an explicit endpoint using that host port. Explicit endpoints -retain their scheme, host, port, and TLS verification; bridge mode rejects -loopback and unspecified endpoint addresses. HTTPS certificates must cover -the selected endpoint hostname or IP. - -In bridge mode, Docker resolves `host-gateway` for the supervisor's -`host.openshell.internal` and `host.docker.internal` aliases independently of -the gateway endpoint. The supervisor pins that driver-owned address before -admitting the workload. On native Linux, host services listening only on -loopback are unreachable through the bridge; expose them on a bridge-reachable -interface or use host mode when permitted. Host mode on Docker Desktop requires -host networking enabled and ECI disabled. Bridge selection removes that -networking conflict but does not establish full ECI compatibility: the runtime -must still pass OpenShell's Landlock and seccomp qualification. A containerized -gateway under ECI also requires an administrator-approved Docker socket exception. +`auto` selects bridge for Docker Desktop Enhanced Container Isolation (ECI) +workloads using `sysbox-runc`, otherwise host, on every launch and restart. +The workload always uses `network=none`. + +Omit `grpc_endpoint` for automatic selection: host mode uses loopback; bridge +uses `host.docker.internal` on Desktop or with a wildcard listener, otherwise +the Linux listener's non-loopback address. Both use the gateway's bind port. +Bridge requires a reachable listener; OpenShell does not widen it. Set an +explicit endpoint for remapped ports. HTTPS certificates must cover the endpoint. + +Bridge resolves both host aliases through Docker's `host-gateway`; native Linux +services bound only to loopback remain unreachable. Desktop host mode requires +host networking enabled and ECI disabled. ECI still requires Landlock and seccomp +qualification, plus an administrator-approved Docker socket exception for +containerized gateways. Docker accepts `http://` and `https://` proxy URLs in explicit `scheme://host:port` form. `no_proxy` bypasses only the corporate proxy;