diff --git a/crates/openshell-driver-docker/README.md b/crates/openshell-driver-docker/README.md index ee329f8dea..feb94e68d6 100644 --- a/crates/openshell-driver-docker/README.md +++ b/crates/openshell-driver-docker/README.md @@ -41,9 +41,11 @@ mediates every supported TCP and DNS operation, attributes it to the calling binary, and sends the request across the private channel. The supervisor authorizes the request before it opens an upstream connection. Docker's absent workload network is the mandatory outer fence if mediation fails or is -bypassed. The trusted supervisor companion uses Docker host networking, where -it reaches the gateway's primary loopback listener and originates approved -egress. +bypassed. The trusted supervisor companion originates approved egress. Its +operator-owned `supervisor_network_mode` defaults to `auto`: each launch +inspects the existing workload container and chooses bridge for `sysbox-runc` +(the Docker Desktop ECI runtime), or host for other runtimes. Explicit `host` +and `bridge` values override this heuristic. No probe container is created. The driver copies trusted runtime bytes from the configured supervisor image through the Docker archive API. No workload launch depends on a host bind @@ -158,7 +160,7 @@ openshell sandbox create \ `/openshell-sandbox` binary. The driver extracts that binary as bytes and stages it into the stopped workload. `supervisor_image` contains the dynamically linked glibc `/openshell-supervisor` binary that runs in the -host-networked supervisor container. Release and gateway image builds bake +supervisor companion container. Release and gateway image builds bake matching image tags into the binary. ## Gateway session and TLS @@ -172,14 +174,31 @@ When no endpoint is configured, the supervisor connects to Docker daemon host. A configured HTTPS server certificate must include the endpoint host in its subject alternative names. -The driver publishes host loopback as the backend address for +In host mode, the driver publishes host loopback as the backend address for `host.openshell.internal`. Policy DNS resolves that reserved name through the mediated path, so policies can reach host services without a Docker bridge, container DNS alias, or another gateway listener. -Docker Engine on Linux supports host networking directly. Docker Desktop -requires host networking to be enabled in Settings and does not support it -when Enhanced Container Isolation is enabled. +Host mode on Docker Desktop requires host networking enabled in Settings and +ECI disabled. Bridge mode uses Docker-resolved `host-gateway` entries for both +host aliases. The supervisor's `--host-gateway-from-hosts` option pins the +numeric address from its own driver-owned `/etc/hosts` into the runtime +descriptor before attaching the workload; workload DNS and hosts files cannot +select this address. Gateway endpoints and host aliases are independent in +bridge mode. + +When `grpc_endpoint` is omitted, bridge mode uses `host.docker.internal` on +Desktop or for a wildcard gateway listener, and the concrete bind address for +other non-loopback Linux listeners. The port is the gateway bind port; remapped +container publications require an explicit endpoint. Native Linux loopback-only +listeners cannot serve a bridge supervisor and produce an actionable error. +Explicit endpoints retain TLS verification and are never rewritten; bridge +rejects loopback/unspecified endpoint addresses. Host services bound only to +loopback are also unreachable from a native Linux bridge supervisor. + +Selecting bridge does not qualify full ECI support. Landlock and seccomp probes +remain mandatory, and a gateway container under ECI needs an administrator's +Docker socket exception. The supervisor owns these security-critical variables: diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index d4a1f2f71d..95e4fd5c17 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -152,6 +152,17 @@ fn provisioning_span( span } +/// Operator-owned networking mode for the trusted supervisor companion. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum DockerSupervisorNetworkMode { + /// Use bridge for Sysbox/ECI workloads, otherwise use host networking. + #[default] + Auto, + Host, + Bridge, +} + /// Gateway-local configuration for the Docker compute driver. #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] #[serde(default, deny_unknown_fields)] @@ -177,6 +188,9 @@ pub struct DockerComputeConfig { /// Gateway gRPC endpoint the sandbox connects back to. pub grpc_endpoint: String, + /// Networking mode for the supervisor only. Workloads always use network=none. + pub supervisor_network_mode: DockerSupervisorNetworkMode, + /// Image containing the trusted `openshell-sandbox` binary. pub sandbox_runtime_image: Option, @@ -273,6 +287,7 @@ impl Default for DockerComputeConfig { image_pull_policy: ImagePullPolicy::default(), sandbox_label: "default".to_string(), grpc_endpoint: String::new(), + supervisor_network_mode: DockerSupervisorNetworkMode::Auto, sandbox_runtime_image: None, supervisor_bin: None, supervisor_image: None, @@ -309,6 +324,9 @@ struct DockerDriverRuntimeConfig { sandbox_binary: Arc>, supervisor_image_id: String, supervisor_grpc_endpoint: String, + supervisor_network_mode: DockerSupervisorNetworkMode, + supervisor_endpoint_explicit: bool, + bridge_default_endpoint: Option, ssh_socket_path: String, guest_tls: Option, gpu: DockerGpuRuntimeCapabilities, @@ -881,6 +899,14 @@ impl DockerComputeDriver { }; validate_docker_app_armor_profile(docker_config.app_armor_profile.as_ref(), &info)?; let gateway_port = gateway_bind_address.port(); + let supervisor_endpoint_explicit = !docker_config.grpc_endpoint.trim().is_empty(); + let bridge_default_endpoint = default_docker_bridge_endpoint( + gateway_bind_address, + docker_guest_tls_configured(docker_config), + info.operating_system + .as_deref() + .is_some_and(|os| os.to_ascii_lowercase().contains("docker desktop")), + ); let mut docker_config = docker_config.clone(); if docker_config.grpc_endpoint.trim().is_empty() { docker_config.grpc_endpoint = default_docker_supervisor_grpc_endpoint( @@ -938,6 +964,9 @@ impl DockerComputeDriver { sandbox_binary, supervisor_image_id, supervisor_grpc_endpoint, + supervisor_network_mode: docker_config.supervisor_network_mode, + supervisor_endpoint_explicit, + bridge_default_endpoint, ssh_socket_path: docker_config.ssh_socket_path.clone(), guest_tls, gpu, @@ -4562,10 +4591,9 @@ async fn prepare_docker_boundary_files( server_name: tls.server_name.clone(), trust_anchor_pem: tls.trust_anchor_pem.clone(), }, - // Pin the reserved host alias to the same address used by the - // host-networked supervisor. This is normally loopback, but container - // CI reaches the gateway and host fixtures through the job - // container's bridge address. + // Host mode pins the alias to the gateway address, normally loopback. + // In bridge mode the companion replaces this value from its own + // Docker-injected hosts file before attaching the descriptor. host_gateway_ip: docker_supervisor_host_address(&config.supervisor_grpc_endpoint), workload_identity: workload_identity.clone(), child_env: docker_child_environment(sandbox), @@ -4976,14 +5004,34 @@ fn docker_auxiliary_container_labels( ]) } -fn docker_supervisor_host_config(mounts: Vec, grpc_endpoint: &str) -> HostConfig { +fn docker_supervisor_host_config( + mounts: Vec, + grpc_endpoint: &str, + network_mode: DockerSupervisorNetworkMode, +) -> HostConfig { HostConfig { // The supervisor is trusted infrastructure and originates every // approved upstream connection. Host networking lets it reach the // configured gateway and host-side services directly; the workload // remains fenced by network=none. - network_mode: Some("host".to_string()), - extra_hosts: docker_supervisor_host_aliases(grpc_endpoint), + network_mode: Some( + if network_mode == DockerSupervisorNetworkMode::Bridge { + "bridge" + } else { + "host" + } + .to_string(), + ), + extra_hosts: if network_mode == DockerSupervisorNetworkMode::Bridge { + // Docker resolves host-gateway on the daemon, including Desktop's VM. + // The gateway endpoint may be remote and must not define host aliases. + Some(vec![ + format!("{HOST_OPEN_SHELL_INTERNAL}:host-gateway"), + format!("{HOST_DOCKER_INTERNAL}:host-gateway"), + ]) + } else { + docker_supervisor_host_aliases(grpc_endpoint) + }, mounts: Some(mounts), cap_drop: Some(vec!["ALL".to_string()]), cap_add: None, @@ -5039,6 +5087,18 @@ async fn spawn_docker_control_process( config: &DockerDriverRuntimeConfig, failure_context: DockerRuntimeFailureContext, ) -> Result { + // Inspect the existing workload on every supervisor launch, including restart. + // This adds no diagnostic container and observes the actual selected runtime. + let (network_mode, supervisor_endpoint) = Box::pin(async { + let workload = docker + .inspect_container(&failure_context.container_id, None) + .await + .map_err(|error| internal_status("inspect Docker workload runtime", error))?; + docker_supervisor_networking(config, &workload) + }) + .await?; + info!(sandbox_id = %sandbox.id, ?network_mode, endpoint = %supervisor_endpoint, + "Selected Docker supervisor networking"); let directory = docker_boundary_state_dir(sandbox, config)?; let main_process_spec = tokio::fs::read_to_string(directory.join(MAIN_PROCESS_SPEC_FILE)) .await @@ -5067,7 +5127,7 @@ async fn spawn_docker_control_process( format!( "{}={}", openshell_core::sandbox_env::ENDPOINT, - config.supervisor_grpc_endpoint + supervisor_endpoint ), format!("{}={}", openshell_core::sandbox_env::SANDBOX_ID, sandbox.id), format!("{}={}", openshell_core::sandbox_env::SANDBOX, sandbox.name), @@ -5144,6 +5204,9 @@ async fn spawn_docker_control_process( workspace_root, format!("--health-socket-path={SUPERVISOR_HEALTH_SOCKET_PATH}"), ]; + if network_mode == DockerSupervisorNetworkMode::Bridge { + command.push("--host-gateway-from-hosts".to_string()); + } command.extend(docker_upstream_proxy_cli_args( &config.upstream_proxy, config.proxy_ca_bundle.is_some(), @@ -5207,7 +5270,8 @@ async fn spawn_docker_control_process( }), host_config: Some(docker_supervisor_host_config( supervisor_mounts, - &config.supervisor_grpc_endpoint, + &supervisor_endpoint, + network_mode, )), ..Default::default() }; @@ -6514,6 +6578,63 @@ fn default_docker_supervisor_grpc_endpoint(gateway_port: u16, tls: bool) -> Stri format!("{scheme}://127.0.0.1:{gateway_port}") } +fn default_docker_bridge_endpoint( + gateway_bind_address: SocketAddr, + tls: bool, + desktop: bool, +) -> Option { + let scheme = if tls { "https" } else { "http" }; + if desktop || gateway_bind_address.ip().is_unspecified() { + Some(format!( + "{scheme}://{HOST_DOCKER_INTERNAL}:{}", + gateway_bind_address.port() + )) + } else if !gateway_bind_address.ip().is_loopback() { + Some(format!("{scheme}://{gateway_bind_address}")) + } else { + None + } +} + +fn docker_supervisor_networking( + config: &DockerDriverRuntimeConfig, + workload: &bollard::models::ContainerInspectResponse, +) -> Result<(DockerSupervisorNetworkMode, String), Status> { + let sysbox = workload + .host_config + .as_ref() + .and_then(|host| host.runtime.as_deref()) + .is_some_and(|runtime| runtime == "sysbox-runc"); + let mode = match config.supervisor_network_mode { + DockerSupervisorNetworkMode::Auto if sysbox => DockerSupervisorNetworkMode::Bridge, + DockerSupervisorNetworkMode::Auto => DockerSupervisorNetworkMode::Host, + mode => mode, + }; + let endpoint = if mode == DockerSupervisorNetworkMode::Bridge + && !config.supervisor_endpoint_explicit + { + config.bridge_default_endpoint.clone().ok_or_else(|| Status::failed_precondition( + "Docker bridge supervisor cannot reach a Linux gateway bound only to loopback; configure a bridge-reachable gateway bind_address and grpc_endpoint, or use supervisor_network_mode = 'host' when the runtime permits host networking", + ))? + } else { + config.supervisor_grpc_endpoint.clone() + }; + if mode == DockerSupervisorNetworkMode::Bridge { + let url = Url::parse(&endpoint).map_err(|error| { + Status::failed_precondition(format!("invalid Docker supervisor endpoint: {error}")) + })?; + if matches!(url.host(), Some(url::Host::Domain(host)) if host.eq_ignore_ascii_case("localhost")) + || matches!(url.host(), Some(url::Host::Ipv4(ip)) if ip.is_loopback() || ip.is_unspecified()) + || matches!(url.host(), Some(url::Host::Ipv6(ip)) if ip.is_loopback() || ip.is_unspecified()) + { + return Err(Status::failed_precondition( + "Docker bridge supervisor grpc_endpoint must be reachable outside the gateway's loopback namespace; use host.docker.internal or a reachable gateway address", + )); + } + } + Ok((mode, endpoint)) +} + pub(crate) fn docker_guest_tls_paths( docker_config: &DockerComputeConfig, ) -> CoreResult> { diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index bc747312c3..bcb86a62a1 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -185,6 +185,9 @@ fn runtime_config() -> DockerDriverRuntimeConfig { sandbox_binary: Arc::new(b"\x7fELFtest".to_vec()), supervisor_image_id: "sha256:supervisor-test".to_string(), supervisor_grpc_endpoint: "https://host.openshell.internal:8443".to_string(), + supervisor_network_mode: DockerSupervisorNetworkMode::Auto, + supervisor_endpoint_explicit: true, + bridge_default_endpoint: Some("https://host.docker.internal:8443".to_string()), ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(), guest_tls: Some(DockerGuestTlsPaths { ca: PathBuf::from("/tmp/ca.crt"), @@ -2888,7 +2891,11 @@ fn build_container_create_body_disables_docker_networking() { #[test] fn docker_supervisor_uses_host_network() { - let host = docker_supervisor_host_config(Vec::new(), "https://127.0.0.1:17670"); + let host = docker_supervisor_host_config( + Vec::new(), + "https://127.0.0.1:17670", + DockerSupervisorNetworkMode::Host, + ); assert_eq!(host.network_mode.as_deref(), Some("host")); assert_eq!( @@ -2904,7 +2911,11 @@ fn docker_supervisor_uses_host_network() { #[test] fn docker_supervisor_maps_host_aliases_to_the_gateway_address() { - let host = docker_supervisor_host_config(Vec::new(), "https://172.20.0.4:17670"); + let host = docker_supervisor_host_config( + Vec::new(), + "https://172.20.0.4:17670", + DockerSupervisorNetworkMode::Host, + ); assert_eq!( host.extra_hosts, @@ -2921,7 +2932,11 @@ fn docker_supervisor_maps_host_aliases_to_the_gateway_address() { #[test] fn docker_supervisor_leaves_named_gateway_hosts_to_dns() { - let host = docker_supervisor_host_config(Vec::new(), "https://gateway.example.com:17670"); + let host = docker_supervisor_host_config( + Vec::new(), + "https://gateway.example.com:17670", + DockerSupervisorNetworkMode::Host, + ); assert_eq!(host.extra_hosts, None); assert_eq!( @@ -2942,6 +2957,186 @@ fn docker_supervisor_defaults_to_the_primary_loopback_endpoint() { ); } +#[test] +fn docker_supervisor_selects_bridge_for_sysbox_and_honors_operator_mode() { + for (runtime, requested, expected) in [ + ( + Some("sysbox-runc"), + DockerSupervisorNetworkMode::Auto, + DockerSupervisorNetworkMode::Bridge, + ), + ( + Some("runc"), + DockerSupervisorNetworkMode::Auto, + DockerSupervisorNetworkMode::Host, + ), + ( + None, + DockerSupervisorNetworkMode::Auto, + DockerSupervisorNetworkMode::Host, + ), + ( + Some("sysbox-runc"), + DockerSupervisorNetworkMode::Host, + DockerSupervisorNetworkMode::Host, + ), + ( + Some("runc"), + DockerSupervisorNetworkMode::Bridge, + DockerSupervisorNetworkMode::Bridge, + ), + ] { + let mut config = runtime_config(); + config.supervisor_network_mode = requested; + config.supervisor_endpoint_explicit = false; + config.supervisor_grpc_endpoint = "https://127.0.0.1:8443".into(); + let inspected = bollard::models::ContainerInspectResponse { + host_config: Some(HostConfig { + runtime: runtime.map(str::to_string), + ..Default::default() + }), + ..Default::default() + }; + let (mode, endpoint) = docker_supervisor_networking(&config, &inspected).unwrap(); + assert_eq!(mode, expected); + assert_eq!( + endpoint, + if mode == DockerSupervisorNetworkMode::Bridge { + "https://host.docker.internal:8443" + } else { + "https://127.0.0.1:8443" + } + ); + } +} + +#[test] +fn docker_bridge_preserves_explicit_endpoints_and_rejects_loopback() { + let mut config = runtime_config(); + config.supervisor_network_mode = DockerSupervisorNetworkMode::Bridge; + for endpoint in [ + "https://gateway.example.com:9443", + "http://172.20.0.4:8080", + "https://[fd00::2]:9443", + ] { + config.supervisor_grpc_endpoint = endpoint.into(); + let (_, selected) = docker_supervisor_networking( + &config, + &bollard::models::ContainerInspectResponse::default(), + ) + .unwrap(); + assert_eq!(selected, endpoint); + } + for endpoint in [ + "https://127.0.0.1:8443", + "http://localhost:8080", + "https://[::1]:8443", + "http://0.0.0.0:8080", + "http://[::]:8080", + ] { + config.supervisor_grpc_endpoint = endpoint.into(); + let error = docker_supervisor_networking( + &config, + &bollard::models::ContainerInspectResponse::default(), + ) + .unwrap_err(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert!(error.message().contains("grpc_endpoint")); + } +} + +#[test] +fn docker_bridge_defaults_respect_desktop_and_linux_listener_reachability() { + assert_eq!( + default_docker_bridge_endpoint("127.0.0.1:17670".parse().unwrap(), true, true).as_deref(), + Some("https://host.docker.internal:17670") + ); + assert_eq!( + default_docker_bridge_endpoint("0.0.0.0:8080".parse().unwrap(), false, false).as_deref(), + Some("http://host.docker.internal:8080") + ); + assert_eq!( + default_docker_bridge_endpoint("192.168.1.2:8080".parse().unwrap(), true, false).as_deref(), + Some("https://192.168.1.2:8080") + ); + assert_eq!( + default_docker_bridge_endpoint("[fd00::2]:8080".parse().unwrap(), true, false).as_deref(), + Some("https://[fd00::2]:8080") + ); + assert!( + default_docker_bridge_endpoint("127.0.0.1:17670".parse().unwrap(), true, false).is_none() + ); + let mut config = runtime_config(); + config.supervisor_network_mode = DockerSupervisorNetworkMode::Bridge; + config.supervisor_endpoint_explicit = false; + config.bridge_default_endpoint = None; + let error = docker_supervisor_networking( + &config, + &bollard::models::ContainerInspectResponse::default(), + ) + .unwrap_err(); + assert!(error.message().contains("bind_address")); +} + +#[test] +fn docker_bridge_host_aliases_are_independent_of_gateway_endpoint() { + let host = docker_supervisor_host_config( + Vec::new(), + "https://gateway.example.com:9443", + DockerSupervisorNetworkMode::Bridge, + ); + assert_eq!(host.network_mode.as_deref(), Some("bridge")); + assert_eq!( + host.extra_hosts, + Some(vec![ + "host.openshell.internal:host-gateway".into(), + "host.docker.internal:host-gateway".into() + ]) + ); + assert_eq!(host.cap_drop, Some(vec!["ALL".into()])); + assert_eq!(host.cap_add, None); + assert_eq!( + host.security_opt, + Some(vec!["no-new-privileges:true".into()]) + ); + assert_eq!( + build_container_create_body(&test_sandbox(), &runtime_config()) + .unwrap() + .host_config + .unwrap() + .network_mode + .as_deref(), + Some("none") + ); +} + +#[test] +fn docker_supervisor_network_mode_is_an_operator_config_enum() { + assert_eq!( + toml::from_str::("") + .unwrap() + .supervisor_network_mode, + DockerSupervisorNetworkMode::Auto + ); + for (value, expected) in [ + ("auto", DockerSupervisorNetworkMode::Auto), + ("host", DockerSupervisorNetworkMode::Host), + ("bridge", DockerSupervisorNetworkMode::Bridge), + ] { + let config: DockerComputeConfig = + toml::from_str(&format!("supervisor_network_mode = '{value}'")).unwrap(); + assert_eq!(config.supervisor_network_mode, expected); + } + assert!(toml::from_str::("supervisor_network_mode = 'none'").is_err()); + assert!( + serde_json::from_value::(serde_json::json!({ + "supervisor_network_mode": "host" + })) + .is_err(), + "workloads cannot select supervisor networking" + ); +} + #[test] fn build_container_create_body_limits_writable_runtime_storage_to_supervisor_ca() { let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); diff --git a/crates/openshell-supervisor/src/main.rs b/crates/openshell-supervisor/src/main.rs index 4bfb469891..8026969e5e 100644 --- a/crates/openshell-supervisor/src/main.rs +++ b/crates/openshell-supervisor/src/main.rs @@ -113,6 +113,10 @@ struct Args { #[arg(long)] backend_descriptor_file: Option, + /// Resolve the Docker host alias from this supervisor's driver-owned hosts file. + #[arg(long)] + host_gateway_from_hosts: bool, + /// Protected gateway-issued credentials for this exact sandbox launch. #[arg(long)] auth_bundle_file: Option, @@ -180,14 +184,68 @@ fn backend_descriptor(args: &Args) -> Result { let path = args.backend_descriptor_file.as_deref().ok_or_else(|| { miette::miette!("--backend-descriptor-file is required for --role=isolation-backend") })?; - let payload = std::fs::read(path) + let mut payload = std::fs::read(path) .map_err(|error| miette::miette!("read backend descriptor {}: {error}", path.display()))?; + if args.host_gateway_from_hosts { + let hosts = std::fs::read_to_string("/etc/hosts") + .map_err(|error| miette::miette!("read supervisor Docker host aliases: {error}"))?; + let ip = docker_host_gateway_from_hosts(&hosts)?; + let mut descriptor: openshell_sandbox_backend::boundary_protocol::SandboxRuntimeDescriptor = + serde_json::from_slice(&payload).into_diagnostic()?; + // Resolve before any workload is admitted, from the supervisor's mount + // namespace. Workload /etc/hosts and DNS never supply this trusted IP. + descriptor.host_gateway_ip = Some(ip); + payload = serde_json::to_vec(&descriptor).into_diagnostic()?; + } Ok(BackendDescriptor { backend_name: openshell_sandbox_backend::BACKEND_NAME.to_string(), payload, }) } +fn docker_host_gateway_from_hosts(hosts: &str) -> Result { + let mut ips = std::collections::BTreeSet::new(); + for line in hosts.lines() { + let mut fields = line.split('#').next().unwrap_or("").split_whitespace(); + let Some(ip) = fields + .next() + .and_then(|value| value.parse::().ok()) + else { + continue; + }; + if fields.any(|host| host.eq_ignore_ascii_case("host.openshell.internal")) { + ips.insert(ip); + } + } + // Docker may expand host-gateway to both families. Prefer IPv4 consistently. + let ipv4 = ips + .iter() + .copied() + .filter(std::net::IpAddr::is_ipv4) + .collect::>(); + let candidates = if ipv4.is_empty() { + ips.into_iter().collect() + } else { + ipv4 + }; + let [ip] = candidates.as_slice() else { + return Err(miette::miette!( + "expected one Docker host-gateway address per family in supervisor /etc/hosts" + )); + }; + if ip.is_loopback() + || ip.is_unspecified() + || ip.is_multicast() + || matches!(ip, std::net::IpAddr::V4(ip) if ip.is_link_local() || ip.is_broadcast()) + || matches!(ip, std::net::IpAddr::V6(ip) if ip.is_unicast_link_local() || ip.to_ipv4_mapped().is_some()) + { + return Err(miette::miette!( + "unsafe Docker host-gateway address in supervisor /etc/hosts" + )); + } + Ok(*ip) +} + fn auth_bundle(args: &Args) -> Result { let path = args.auth_bundle_file.as_deref().ok_or_else(|| { miette::miette!("--auth-bundle-file is required for --role=isolation-backend") @@ -232,6 +290,7 @@ fn validate_role_arguments(args: &Args) -> Result<()> { } SupervisorRole::NetworkProxy => { if args.backend_descriptor_file.is_some() + || args.host_gateway_from_hosts || args.auth_bundle_file.is_some() || args.sandbox_id.is_some() || args.sandbox.is_some() @@ -482,6 +541,33 @@ fn main() -> Result<()> { mod tests { use super::*; + #[test] + fn docker_host_gateway_uses_supervisor_alias_and_prefers_ipv4() { + let hosts = "127.0.0.1 localhost\n192.168.65.254 host.openshell.internal host.docker.internal\nfd00::1 host.openshell.internal\n192.168.65.254 host.openshell.internal # duplicate\n"; + assert_eq!( + docker_host_gateway_from_hosts(hosts).unwrap(), + "192.168.65.254".parse::().unwrap() + ); + assert_eq!( + docker_host_gateway_from_hosts("fd00::1 host.openshell.internal").unwrap(), + "fd00::1".parse::().unwrap() + ); + } + + #[test] + fn docker_host_gateway_rejects_missing_ambiguous_and_unsafe_aliases() { + for hosts in [ + "192.168.65.254 host.docker.internal", + "192.168.65.254 host.openshell.internal\n172.17.0.1 host.openshell.internal", + "127.0.0.1 host.openshell.internal", + "0.0.0.0 host.openshell.internal", + "169.254.169.254 host.openshell.internal", + "::ffff:127.0.0.1 host.openshell.internal", + ] { + assert!(docker_host_gateway_from_hosts(hosts).is_err(), "{hosts}"); + } + } + #[test] fn isolation_backend_is_the_default_role() { let directory = tempfile::tempdir().expect("temporary runtime descriptor directory"); diff --git a/docs/about/support-matrix.mdx b/docs/about/support-matrix.mdx index 9cace1a708..e5a491e613 100644 --- a/docs/about/support-matrix.mdx +++ b/docs/about/support-matrix.mdx @@ -112,7 +112,7 @@ The gateway can manage sandboxes through several runtimes. | Runtime | Status | Notes | |---|---|---| -| Docker | Supported for local development and single-machine gateways. | Requires Docker Desktop or Docker Engine on the gateway host. | +| Docker | Supported for local development and single-machine gateways. | Requires an accessible Docker API socket. Supervisor networking defaults to host, or bridge for detected Sysbox/ECI workloads. See [Docker runtime setup](/how-it-works/sandboxes/runtimes#docker-driver) for endpoint and isolation requirements. | | Podman | Supported for rootless local and workstation workflows. | Requires a Podman-compatible socket and rootless networking setup. | | Kubernetes | Supported through the [OpenShell Helm chart](https://github.com/NVIDIA/OpenShell/blob/main/deploy/helm/openshell/README.md). | Requires a Kubernetes cluster supplied by the operator. | | MicroVM | Supported for VM-backed sandboxes. | Uses the VM compute driver and libkrun-based runtime. | diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index 3b3c3aaaf0..ca840409cd 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -910,9 +910,11 @@ default_image = "nvcr.io/nvidia/base/ubuntu:24.04" image_pull_policy = "if_not_present" # Value assigned to the openshell.sandbox_namespace label on sandbox containers. sandbox_label = "docker-dev" -# Optional override. When omitted, the host-networked supervisor uses the -# gateway's primary loopback endpoint. -grpc_endpoint = "https://127.0.0.1:17670" +# auto uses bridge for Sysbox/ECI workloads, otherwise host networking. +# Operator-owned; sandbox driver config cannot override this choice. +supervisor_network_mode = "auto" +# Optional override. Omit for automatic host or bridge callback selection. +# grpc_endpoint = "https://gateway.example.com:17670" # Workload-side runtime. Defaults to the gateway version. # sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:" # Supervisor runtime. Defaults to the gateway version. @@ -948,6 +950,23 @@ provider_spiffe_workload_api_socket = "/run/spire/agent.sock" Use `sandbox_label` for Docker configurations. The legacy `sandbox_namespace` key is rejected. +`supervisor_network_mode` accepts `auto` (the default), `host`, or `bridge`. +`auto` selects bridge for Docker Desktop Enhanced Container Isolation (ECI) +workloads using `sysbox-runc`, otherwise host, on every launch and restart. +The workload always uses `network=none`. + +Omit `grpc_endpoint` for automatic selection: host mode uses loopback; bridge +uses `host.docker.internal` on Desktop or with a wildcard listener, otherwise +the Linux listener's non-loopback address. Both use the gateway's bind port. +Bridge requires a reachable listener; OpenShell does not widen it. Set an +explicit endpoint for remapped ports. HTTPS certificates must cover the endpoint. + +Bridge resolves both host aliases through Docker's `host-gateway`; native Linux +services bound only to loopback remain unreachable. Desktop host mode requires +host networking enabled and ECI disabled. ECI still requires Landlock and seccomp +qualification, plus an administrator-approved Docker socket exception for +containerized gateways. + Docker accepts `http://` and `https://` proxy URLs in explicit `scheme://host:port` form. `no_proxy` bypasses only the corporate proxy; OpenShell policy still applies. Proxy URLs cannot embed credentials. Supply a diff --git a/docs/how-it-works/gateways/overview.mdx b/docs/how-it-works/gateways/overview.mdx index ee1f735fa3..fe12dc6b1f 100644 --- a/docs/how-it-works/gateways/overview.mdx +++ b/docs/how-it-works/gateways/overview.mdx @@ -26,7 +26,7 @@ A gateway provisions sandboxes through the compute driver configured for that ga | Runtime | Where sandboxes run | Best for | |---|---|---| -| Docker | Containers on the gateway host. | Solo development, quick iteration, and single-machine gateways. | +| Docker | Containers on the Docker daemon host. | Solo development, quick iteration, and single-machine gateways. | | Podman | Rootless containers on the gateway host. | Workstations that avoid a rootful Docker daemon. | | Kubernetes | Pods in an operator-managed cluster. | Shared clusters and cloud environments. | | MicroVM | VM-backed sandboxes. | Workflows that need VM-backed isolation. | diff --git a/docs/how-it-works/sandboxes/runtimes.mdx b/docs/how-it-works/sandboxes/runtimes.mdx index f52a571bd2..8d75749c91 100644 --- a/docs/how-it-works/sandboxes/runtimes.mdx +++ b/docs/how-it-works/sandboxes/runtimes.mdx @@ -82,7 +82,7 @@ To pin specific GPUs, pass `cdi_devices` for Docker or Podman, or `gpu_device_id ## Docker Driver -[Docker](https://www.docker.com/get-started/) runs sandboxes as containers on the gateway host. Docker is also required to build sandbox images from local directories or Dockerfiles. +[Docker](https://www.docker.com/get-started/) runs sandboxes as containers on the Docker daemon host. The gateway accesses the daemon through its configured Unix socket. Docker is also required to build sandbox images from local directories or Dockerfiles. ```toml [openshell.gateway] @@ -92,9 +92,13 @@ compute_driver = "docker" socket_path = "/var/run/docker.sock" ``` -Common options in `[openshell.drivers.docker]` are `socket_path`, `grpc_endpoint`, `sandbox_runtime_image`, `supervisor_image`, `image_pull_policy`, and `sandbox_pids_limit`. When `socket_path` is unset, the driver uses the socket found by auto-detection. +Common options in `[openshell.drivers.docker]` are `socket_path`, `grpc_endpoint`, `supervisor_network_mode`, `sandbox_runtime_image`, `supervisor_image`, `image_pull_policy`, and `sandbox_pids_limit`. When `socket_path` is unset, the driver uses the socket found by auto-detection. -Docker Desktop must have host networking enabled, and it cannot use Enhanced Container Isolation. Set `grpc_endpoint` when sandboxes cannot reach the gateway on host loopback. For GPU sandboxes, configure Docker CDI before starting the gateway. +`supervisor_network_mode` defaults to `auto`: the driver uses bridge networking for workloads whose inspected runtime is `sysbox-runc` (Docker Desktop Enhanced Container Isolation), and host networking otherwise. Set `host` or `bridge` to override the heuristic. The workload always retains `network=none`. + +Host mode on Docker Desktop requires host networking enabled and Enhanced Container Isolation disabled. Bridge mode automatically uses `host.docker.internal` for a gateway on the Desktop host. On native Linux, the gateway and host services must listen on bridge-reachable addresses. Set `grpc_endpoint` for remote gateways, remapped published ports, or other custom layouts. Endpoint selection preserves TLS verification and does not change the gateway's listener. See [Docker configuration](/how-it-works/gateways/configuration#docker). + +Bridge selection does not establish full Enhanced Container Isolation compatibility; the sandbox must still pass its Landlock and seccomp qualification. A gateway running in an ECI container requires an administrator-approved Docker socket exception. For GPU sandboxes, configure Docker CDI before starting the gateway. ### Docker Corporate Proxy Egress diff --git a/e2e/with-docker-gateway.sh b/e2e/with-docker-gateway.sh index 4b96693ab1..512b632697 100755 --- a/e2e/with-docker-gateway.sh +++ b/e2e/with-docker-gateway.sh @@ -620,7 +620,10 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" else printf 'allow_driver_config = true\n' printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")" - printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" + # Container jobs need their bridge address; host runs exercise discovery. + if [ -n "${GATEWAY_HOST_ALIAS_IP}" ]; then + printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" + fi printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")" printf 'image_pull_policy = %s\n' "$(toml_string "${SANDBOX_IMAGE_PULL_POLICY}")" printf 'enable_bind_mounts = true\n' @@ -634,7 +637,9 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then { printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")" - printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" + if [ -n "${GATEWAY_HOST_ALIAS_IP}" ]; then + printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" + fi printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")" printf 'image_pull_policy = %s\n' "$(toml_string "${SANDBOX_IMAGE_PULL_POLICY}")" printf 'guest_tls_ca = %s\n' "$(toml_string "${PKI_DIR}/ca.crt")" diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 6382428946..20447b1391 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -287,9 +287,19 @@ Common findings: - Sandbox never registers: check gateway logs and the supervisor's gateway endpoint. - Calls to an external tool server fail while the sandbox is Ready: inspect `Tool server connections` in `openshell sandbox get `. For configured MCP-over-HTTP endpoints, JSON output exposes each address together with `last_result` and `last_reported_at` in `endpoint_statuses`. Select the endpoint by host, path, and ports, then check the reported failure boundary. `last_reported_at` records gateway acceptance time and can advance when retained evidence is accepted after a reset. Results do not expire or prove current availability; `HttpResponseReceived` can still contain a tool error. If several paths share a host and port, a failure before the path is known remains in logs. Verify the actual operation when current tool availability matters. - On Docker Desktop, repeated `Policy fetch failed after 5 attempts` messages - can mean host networking is disabled. Enable host networking in Docker - Desktop, ensure Enhanced Container Isolation is disabled, and verify the - gateway's primary endpoint is reachable from a host-networked container. + can mean the supervisor's selected network mode cannot reach the gateway. + Check the gateway's `Selected Docker supervisor networking` log. The default + `supervisor_network_mode = "auto"` selects bridge for inspected `sysbox-runc` + workloads (ECI), and host otherwise. Host mode requires Desktop host + networking enabled and ECI disabled; set `bridge` explicitly when neither + feature is enabled. Bridge mode uses `host.docker.internal` for the Desktop + host and requires a bridge-reachable listener on native Linux. Explicit + `grpc_endpoint` values must be reachable in the selected mode, including any + remapped published port, with matching TLS certificate names. Mode selection + does not bypass Landlock/seccomp qualification. A gateway container under ECI + also needs an administrator-approved Docker socket exception. See the + published [Docker configuration](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration.md) + for the current networking options. - Sandbox runtime image exits before printing `openshell-sandbox --version`: verify the configured image contains a static executable at `/openshell-sandbox`. - A sandbox with explicit `protocol: tcp` endpoints fails before workload readiness: confirm the selected isolation backend advertises TCP mediation, then inspect the sandbox and supervisor logs for protected-channel setup or listener failures. A driver that cannot supply the required outer egress fence and authenticated runtime channel must reject the policy before starting the agent. - Supervisor runtime validation fails: verify `supervisor_image` contains an `/openshell-supervisor` executable from the same release as the sandbox runtime, and that the dynamic loader and shared libraries it links against are available inside that image. `docker run --rm --network none --entrypoint /openshell-supervisor --version` should print that release; a `no such file or directory` error for a binary that exists means the loader or a library is missing. The supervisor runs from its own image and does not need to be static; only `/openshell-sandbox` must be.