diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 9caa8b3..51458a3 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -10,6 +10,7 @@ - [Building Topologies](guide/topology.md) - [NAT and Firewalls](guide/nat-and-firewalls.md) - [Running Code in Namespaces](guide/running-code.md) +- [Running Containers](guide/containers.md) - [Testing](guide/testing.md) - [Running in a VM](guide/vm.md) diff --git a/docs/guide/containers.md b/docs/guide/containers.md new file mode 100644 index 0000000..9bb9f89 --- /dev/null +++ b/docs/guide/containers.md @@ -0,0 +1,116 @@ +# Running Containers + +Some integration tests need auxiliary services that are easier to run as a +container than to embed: an ACME test server like [Pebble], a database, a +message broker, a real DNS server. patchbay can run these as ordinary lab +nodes. + +A **container is a device**. It gets its own network namespace, a veth uplink +to a router, and a lab IP, exactly like a device created with +[`add_device`](https://docs.rs/patchbay/latest/patchbay/struct.Lab.html#method.add_device). +The difference is only what runs inside it: an OCI image instead of a Rust +closure. Other lab devices reach the service at the container's lab IP. + +## Example + +```rust,ignore +let net = lab.add_router("net").build().await?; + +// Start Pebble as a lab device on `net`, and wait until it is listening. +let pebble = lab + .add_container("pebble", "ghcr.io/letsencrypt/pebble:latest") + .uplink(net.id()) + .env("PEBBLE_VA_NOSLEEP", "1") + .ready_tcp(14000) + .build() + .await?; + +// The directory URL other lab devices use. +let directory = format!("https://{}:14000/dir", pebble.ip().unwrap()); + +// A client device on the same network can now reach it. +let client = lab.add_device("client").uplink(net.id()).build().await?; +``` + +The builder mirrors the device builder for wiring (`uplink`, `iface`, +`default_via`, `mtu`) and adds container configuration: + +| Method | Purpose | +|--------|---------| +| `env(k, v)` | Set an environment variable in the container. | +| `arg(a)` / `args([..])` | Append to the container's command. | +| `volume(host, container)` | Bind-mount a host path read-write. | +| `volume_ro(host, container)` | Bind-mount a host path read-only. | +| `tmpfs(container)` | Mount a fresh tmpfs at a path. | +| `run_arg(a)` | Pass an extra flag to `podman run` (escape hatch). | +| `ready_tcp(port)` | Block `build` until `port` accepts a connection. | +| `runtime(name)` | Override the runtime binary (default `podman`). | + +## Volumes and config + +Hand a service its config or data with a bind mount. Host paths should be +absolute: + +```rust,ignore +let pebble = lab + .add_container("pebble", "ghcr.io/letsencrypt/pebble:latest") + .uplink(net.id()) + .volume_ro(config_dir.join("pebble.json"), "/test/config/pebble.json") + .args(["-config", "/test/config/pebble.json"]) + .ready_tcp(14000) + .build() + .await?; +``` + +On SELinux hosts a bind mount may need a `:z`/`:Z` label; add it with +`run_arg`, e.g. `run_arg("--security-opt=label=disable")`. + +The returned [`Container`] dereferences to [`Device`], so `ip()`, `run_sync()`, +and the other device accessors work directly. It also has: + +- `exec(cmd)` — run a command inside the container. +- `logs()` — fetch the container's logs. +- `write_file(path, bytes)` / `read_file(path)` — write or read a file inside + the container without a bind mount. +- `copy_to(host, container)` / `copy_from(container, host)` — copy files or + directories in or out. +- `stop()` — remove it early (also happens automatically on drop). + +## Runtime: podman, not docker + +Containers run with **podman**, which must be on `PATH`. This is not +incidental. patchbay is rootless: the whole process lives in an unprivileged +user namespace, and each node's network namespace is entered by a worker thread, +not persisted to a path. podman is daemonless, so the `podman run` process, +forked from inside the device's worker thread, is already in the device's +network namespace; `--network=host` then means "use the namespace I am already +in", and the container lands on the device's lab IP with no extra plumbing. + +Docker cannot do this rootlessly. Its client talks to `dockerd`, which runs in +its own namespace, so a docker container joins the daemon's namespace, not the +device's. Making docker work would require manipulating the container's +namespace from outside its owning user namespace, which rootless patchbay +cannot do. + +## Requirements + +Running a container needs a full subuid/subgid range inside patchbay's user +namespace, which is the standard rootless-podman setup: + +- `podman`, `newuidmap`, and `newgidmap` on `PATH`. +- An `/etc/subuid` and `/etc/subgid` entry for your user (for example + `you:100000:65536`). + +patchbay maps that range with `newuidmap`/`newgidmap` when it bootstraps its +user namespace. Without it patchbay falls back to a single-uid namespace, which +cannot unpack image layers; `build()` then fails with a storage error. Router +and device networking work regardless. + +The image is pulled on the host before the container starts, because the device +namespace it runs in has no route to a registry. If a specific image needs a +runtime tweak, pass it through with `run_arg`, for example +`run_arg("--cgroup-manager=cgroupfs")`. + +[Pebble]: https://github.com/letsencrypt/pebble +[`Container`]: https://docs.rs/patchbay/latest/patchbay/struct.Container.html +[`Device`]: https://docs.rs/patchbay/latest/patchbay/struct.Device.html diff --git a/patchbay/src/container.rs b/patchbay/src/container.rs new file mode 100644 index 0000000..7544237 --- /dev/null +++ b/patchbay/src/container.rs @@ -0,0 +1,812 @@ +//! Run OCI containers as lab devices. +//! +//! A [`Container`] is a [`Device`] whose workload is a container instead of a +//! Rust closure. It is wired into the lab exactly like a device (its own +//! network namespace, a veth uplink to a router, a lab IP) and the container +//! runs joined to that namespace, so it is reachable from other lab devices at +//! its lab IP. This lets an integration test stand up the auxiliary services it +//! needs, such as an ACME test server, a database, or a DNS server, next to the +//! code under test. +//! +//! # Runtime +//! +//! Containers run with `podman`, which must be on `PATH`. Podman is daemonless: +//! the `podman run` process is forked from inside the device's network +//! namespace, so `--network=host` makes the container join that namespace +//! directly. Docker does not work here because its daemon runs in a different +//! namespace, so a docker container would not land in the device's namespace. +//! +//! # Requirements +//! +//! Running a container inside patchbay's rootless user namespace needs a full +//! subuid/subgid range, so the host must have `newuidmap`/`newgidmap` installed +//! and an `/etc/subuid` and `/etc/subgid` entry for the user (the standard +//! rootless-podman setup). Without them patchbay falls back to a single-uid +//! namespace, in which image layers cannot be unpacked; [`build`] then fails +//! with a storage error. Device and router networking work either way. +//! +//! [`build`]: ContainerBuilder::build +//! +//! # Example +//! +//! ```no_run +//! # use patchbay::Lab; +//! # async fn f(lab: Lab, net_id: patchbay::NodeId) -> anyhow::Result<()> { +//! let pebble = lab +//! .add_container("pebble", "ghcr.io/letsencrypt/pebble:latest") +//! .uplink(net_id) +//! .env("PEBBLE_VA_NOSLEEP", "1") +//! .ready_tcp(14000) +//! .build() +//! .await?; +//! +//! // Reachable from any lab device on `pebble.ip()`. +//! let directory = format!( +//! "https://{}:14000/dir", +//! pebble.ip().expect("pebble has an ip") +//! ); +//! # let _ = directory; +//! # Ok(()) +//! # } +//! ``` + +use std::{ + net::{IpAddr, Ipv4Addr, SocketAddr}, + ops::Deref, + path::{Path, PathBuf}, + process::Command, + sync::{ + atomic::{AtomicU64, Ordering}, + Arc, + }, + time::{Duration, Instant}, +}; + +use anyhow::{bail, Context, Result}; +use tracing::{debug, warn}; + +use crate::{device::DeviceBuilder, Device}; + +/// Default container runtime binary. +const DEFAULT_RUNTIME: &str = "podman"; + +/// How long [`ContainerBuilder::ready_tcp`] waits for the service to accept a +/// connection before giving up. +const DEFAULT_READY_TIMEOUT: Duration = Duration::from_secs(30); + +/// Readiness gate applied after the container starts. +#[derive(Clone, Debug)] +enum Ready { + /// No readiness check; `build` returns as soon as the container starts. + None, + /// Wait until a TCP connection to this port on the container's IP succeeds. + Tcp { port: u16, timeout: Duration }, +} + +/// A mount to add to the container. +#[derive(Clone, Debug)] +enum Mount { + /// Bind-mount a host path at `container` path, read-only when `read_only`. + Bind { + host: PathBuf, + container: String, + read_only: bool, + }, + /// Mount a fresh tmpfs at `container` path. + Tmpfs { container: String }, +} + +impl Mount { + /// Renders this mount as `podman run` arguments. + fn to_args(&self) -> Vec { + match self { + Mount::Bind { + host, + container, + read_only, + } => { + let mut spec = format!("{}:{container}", host.display()); + if *read_only { + spec.push_str(":ro"); + } + vec!["--volume".to_string(), spec] + } + Mount::Tmpfs { container } => vec!["--tmpfs".to_string(), container.clone()], + } + } +} + +/// Builder for a [`Container`]; returned by [`Lab::add_container`](crate::Lab::add_container). +/// +/// The uplink and interface methods mirror [`Lab::add_device`](crate::Lab::add_device): +/// a container is a device, so it is wired into the topology the same way. +pub struct ContainerBuilder { + device: DeviceBuilder, + image: String, + args: Vec, + env: Vec<(String, String)>, + mounts: Vec, + run_args: Vec, + runtime: String, + ready: Ready, +} + +impl ContainerBuilder { + /// Wraps a device builder with container configuration. + pub(crate) fn new(device: DeviceBuilder, image: impl Into) -> Self { + Self { + device, + image: image.into(), + args: Vec::new(), + env: Vec::new(), + mounts: Vec::new(), + run_args: Vec::new(), + runtime: DEFAULT_RUNTIME.to_string(), + ready: Ready::None, + } + } + + /// Adds an auto-named interface (eth0, eth1, ...) with the given config. + /// + /// Accepts anything that converts to [`IfaceConfig`](crate::IfaceConfig), + /// including a bare [`NodeId`](crate::NodeId) for a simple routed uplink. + pub fn uplink(mut self, config: impl Into) -> Self { + self.device = self.device.uplink(config); + self + } + + /// Adds a named interface with the given configuration. + pub fn iface(mut self, ifname: &str, config: impl Into) -> Self { + self.device = self.device.iface(ifname, config); + self + } + + /// Overrides which interface carries the default route. + pub fn default_via(mut self, ifname: &str) -> Self { + self.device = self.device.default_via(ifname); + self + } + + /// Sets the MTU on all interfaces of the container device. + pub fn mtu(mut self, mtu: u32) -> Self { + self.device = self.device.mtu(mtu); + self + } + + /// Sets an environment variable inside the container. + pub fn env(mut self, key: impl Into, value: impl Into) -> Self { + self.env.push((key.into(), value.into())); + self + } + + /// Appends an argument to the container's command (after the image name). + pub fn arg(mut self, arg: impl Into) -> Self { + self.args.push(arg.into()); + self + } + + /// Appends several arguments to the container's command. + pub fn args(mut self, args: I) -> Self + where + I: IntoIterator, + S: Into, + { + self.args.extend(args.into_iter().map(Into::into)); + self + } + + /// Bind-mounts a host path into the container, read-write. + /// + /// Use this to hand the service its data or config, for example a Pebble + /// config file or a seeded database directory. The host path should be + /// absolute. For read-only, use [`volume_ro`](Self::volume_ro). + pub fn volume(mut self, host: impl Into, container: impl Into) -> Self { + self.mounts.push(Mount::Bind { + host: host.into(), + container: container.into(), + read_only: false, + }); + self + } + + /// Bind-mounts a host path into the container, read-only. + pub fn volume_ro(mut self, host: impl Into, container: impl Into) -> Self { + self.mounts.push(Mount::Bind { + host: host.into(), + container: container.into(), + read_only: true, + }); + self + } + + /// Mounts a fresh tmpfs at the given path inside the container. + /// + /// Useful for a writable scratch directory on an otherwise read-only image. + pub fn tmpfs(mut self, container: impl Into) -> Self { + self.mounts.push(Mount::Tmpfs { + container: container.into(), + }); + self + } + + /// Passes an extra flag to `podman run` (before the image name). + /// + /// An escape hatch for flags the builder does not model, such as + /// `--cgroup-manager=cgroupfs` or a `:z` SELinux volume label. + pub fn run_arg(mut self, arg: impl Into) -> Self { + self.run_args.push(arg.into()); + self + } + + /// Overrides the container runtime binary (default `podman`). + /// + /// Only daemonless runtimes that inherit the caller's network namespace + /// work; see the [module docs](self). + pub fn runtime(mut self, runtime: impl Into) -> Self { + self.runtime = runtime.into(); + self + } + + /// Waits, after start, until a TCP connection to `port` succeeds. + /// + /// The check connects to the container's own lab IP from inside its + /// namespace, so it reflects what other lab devices will see. Uses a + /// 30-second timeout; see [`ready_tcp_timeout`](Self::ready_tcp_timeout) to + /// change it. + pub fn ready_tcp(mut self, port: u16) -> Self { + self.ready = Ready::Tcp { + port, + timeout: DEFAULT_READY_TIMEOUT, + }; + self + } + + /// Like [`ready_tcp`](Self::ready_tcp) with an explicit timeout. + pub fn ready_tcp_timeout(mut self, port: u16, timeout: Duration) -> Self { + self.ready = Ready::Tcp { port, timeout }; + self + } + + /// Wires the container device, starts the container, and waits for + /// readiness. + /// + /// # Errors + /// + /// Returns an error if the device fails to wire, if the runtime binary is + /// missing or `run` fails (the error includes the runtime's stderr), or if + /// a [`ready_tcp`](Self::ready_tcp) gate times out. + pub async fn build(self) -> Result { + let name = container_name(&self.device); + + // Pull on the host before wiring the (registry-unreachable) device + // namespace. + ensure_image(&self.runtime, &self.image).await?; + let device = self.device.build().await?; + + let run_args = build_run_args( + &name, + &self.image, + &self.env, + &self.mounts, + &self.args, + &self.run_args, + ); + debug!(container = %name, runtime = %self.runtime, image = %self.image, "starting container"); + + // Fork the runtime from inside the device namespace so the container, + // run with `--network=host`, joins it. + let runtime = self.runtime.clone(); + let spawn_args = run_args.clone(); + let output = device + .run_sync(move || { + runtime_command(&runtime) + .args(&spawn_args) + .output() + .with_context(|| format!("spawn `{runtime} run` (is it installed?)")) + }) + .with_context(|| format!("start container '{name}'"))?; + if !output.status.success() { + bail!( + "`{} run` for container '{}' failed: {}", + self.runtime, + name, + String::from_utf8_lossy(&output.stderr).trim() + ); + } + + let container = Container { + device, + inner: Arc::new(ContainerInner { + name, + runtime: self.runtime, + }), + }; + + container.wait_ready(&self.ready).await?; + Ok(container) + } +} + +/// Returns a fresh host path in the temp directory for staging a `podman cp`. +fn unique_temp_path() -> PathBuf { + static COUNTER: AtomicU64 = AtomicU64::new(0); + let n = COUNTER.fetch_add(1, Ordering::Relaxed); + std::env::temp_dir().join(format!("patchbay-cp-{}-{n}", std::process::id())) +} + +/// Builds a lab-unique podman container name from the device's namespace. +/// +/// The namespace name (e.g. `lab3-d7`) is unique within a process, and the lab +/// prefix carries a per-process tag, so the resulting name does not collide +/// across parallel test processes on the same host. +fn container_name(device: &DeviceBuilder) -> String { + let inner = device.inner.core.lock().expect("poisoned"); + let prefix = inner.cfg.prefix.clone(); + let dev = inner + .device(device.id) + .map(|d| d.ns.to_string()) + .unwrap_or_else(|| format!("dev{}", device.id)); + sanitize_name(&format!("{prefix}-{dev}")) +} + +/// Replaces characters podman does not accept in a container name with `-`. +fn sanitize_name(raw: &str) -> String { + raw.chars() + .map(|c| { + if c.is_ascii_alphanumeric() || c == '_' || c == '.' || c == '-' { + c + } else { + '-' + } + }) + .collect() +} + +/// Builds the argument vector for ` run`. +/// +/// Detached (`-d`), joined to the caller's network namespace (`--network=host`), +/// and replacing any leftover container of the same name (`--replace`) so a +/// re-run after a crash does not collide. +fn build_run_args( + name: &str, + image: &str, + env: &[(String, String)], + mounts: &[Mount], + cmd_args: &[String], + extra_run_args: &[String], +) -> Vec { + let mut args = vec![ + "run".to_string(), + "-d".to_string(), + "--replace".to_string(), + "--network=host".to_string(), + // The image is pre-pulled on the host; the device namespace has no + // route to a registry. + "--pull=never".to_string(), + "--name".to_string(), + name.to_string(), + ]; + for (k, v) in env { + args.push("--env".to_string()); + args.push(format!("{k}={v}")); + } + for mount in mounts { + args.extend(mount.to_args()); + } + args.extend(extra_run_args.iter().cloned()); + args.push(image.to_string()); + args.extend(cmd_args.iter().cloned()); + args +} + +/// Builds a runtime command carrying the rootless environment podman needs +/// inside patchbay's user namespace. +/// +/// patchbay maps the invoking user to uid 0, so podman would otherwise treat +/// itself as rootful and use unwritable system paths. These variables tell it +/// it is the already-configured rootless namespace, so it uses the user's +/// rootless storage and reuses already-pulled images. +/// +/// These variables are the only way to get this behavior. podman keys "am I +/// rootless" off euid, there is no `containers.conf` override, and +/// `--userns=ns:` is broken for rootless (making it a supported flag is +/// the open request containers/podman#7774). They are internal but stable, and +/// exactly what podman sets on its own rootless re-exec. +/// +/// Two roads not taken: +/// - Running patchbay as a non-zero uid so podman detects rootless on its own. +/// The rootful-vs-rootless storage choice is keyed on euid, not the +/// namespace, so this only trades one variable for a non-root uid that breaks +/// patchbay's own uid-0 assumptions. +/// - Letting podman create its own nested user namespace for the container. Its +/// automatic (subuid-driven) nesting fails, since the inner user has no +/// `/etc/subuid` delegation; an explicit `--uidmap 0:0:65536` does work with +/// `--network=host`, but it is redundant, because patchbay's namespace +/// already maps the full range the container needs. +fn runtime_command(runtime: &str) -> Command { + let (uid, gid) = host_ids(); + let mut cmd = Command::new(runtime); + cmd.env("_CONTAINERS_USERNS_CONFIGURED", "done") + .env("_CONTAINERS_ROOTLESS_UID", uid.to_string()) + .env("_CONTAINERS_ROOTLESS_GID", gid.to_string()); + cmd +} + +/// Returns the outer (host) uid and gid that the current user namespace maps +/// inner id 0 to. These key the user's rootless container storage. +fn host_ids() -> (u32, u32) { + ( + outer_id("/proc/self/uid_map"), + outer_id("/proc/self/gid_map"), + ) +} + +/// Parses the outer id that a `/proc/self/{uid,gid}_map` maps inner id 0 to. +fn outer_id(path: &str) -> u32 { + std::fs::read_to_string(path) + .ok() + .and_then(|contents| { + contents.lines().find_map(|line| { + let mut fields = line.split_whitespace(); + if fields.next()? != "0" { + return None; + } + fields.next()?.parse::().ok() + }) + }) + .unwrap_or(0) +} + +/// Runs a podman subcommand and returns its output. +/// +/// Executes on a dedicated thread with its own mount namespace, in the host +/// network namespace. podman's storage setup makes its mounts private, which +/// needs a mount namespace owned by patchbay's user namespace; the plain +/// blocking-pool threads share the host mount namespace and cannot. The +/// container `run` does not use this path: it is forked from the device's +/// namespace worker, which already has a private mount namespace (and the +/// device network namespace). +async fn podman(runtime: &str, args: Vec) -> Result { + let runtime = runtime.to_string(); + tokio::task::spawn_blocking(move || podman_blocking(&runtime, &args)) + .await + .context("container runtime task panicked")? +} + +/// Blocking core of [`podman`], usable from `Drop`. +fn podman_blocking(runtime: &str, args: &[String]) -> Result { + let runtime = runtime.to_string(); + let args = args.to_vec(); + std::thread::spawn(move || -> std::io::Result { + // Private mount namespace so podman can make its storage mounts private. + let _ = nix::sched::unshare(nix::sched::CloneFlags::CLONE_NEWNS); + runtime_command(&runtime).args(&args).output() + }) + .join() + .map_err(|_| anyhow::anyhow!("container runtime thread panicked"))? + .context("spawn container runtime") +} + +/// Ensures the image is present locally, pulling it on the host (which has +/// registry access) if needed. +/// +/// The container itself runs in a device namespace with no route to a registry, +/// so the image must be present before that run. +async fn ensure_image(runtime: &str, image: &str) -> Result<()> { + if image_present(runtime, image).await { + return Ok(()); + } + let output = podman(runtime, vec!["pull".into(), image.into()]).await?; + if !output.status.success() { + bail!( + "pulling image '{image}' failed: {}", + String::from_utf8_lossy(&output.stderr).trim() + ); + } + Ok(()) +} + +/// Returns whether the image already exists in local storage. +async fn image_present(runtime: &str, image: &str) -> bool { + podman(runtime, vec!["image".into(), "exists".into(), image.into()]) + .await + .map(|o| o.status.success()) + .unwrap_or(false) +} + +/// Shared interior of a [`Container`], removed when the last handle drops. +struct ContainerInner { + name: String, + runtime: String, +} + +impl Drop for ContainerInner { + fn drop(&mut self) { + // Best-effort removal. Runs from within the patchbay process (and its + // user namespace), which is the same podman user context that created + // the container, so removal by name works regardless of the device + // namespace still being alive. + let args = ["rm", "-f", "-t", "1", &self.name].map(String::from); + match podman_blocking(&self.runtime, &args) { + Ok(out) if out.status.success() => {} + Ok(out) => warn!( + container = %self.name, + stderr = %String::from_utf8_lossy(&out.stderr).trim(), + "container removal reported an error" + ), + Err(err) => warn!(container = %self.name, %err, "could not run container removal"), + } + } +} + +/// A container running as a lab device. +/// +/// Cloneable handle. Dereferences to the underlying [`Device`], so device +/// accessors ([`ip`](Device::ip), [`spawn`](Device::spawn), ...) work directly. +/// The container is removed when the last handle is dropped, or explicitly via +/// [`stop`](Self::stop). +#[derive(Clone)] +pub struct Container { + device: Device, + inner: Arc, +} + +impl std::fmt::Debug for Container { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Container") + .field("name", &self.inner.name) + .field("ip", &self.device.ip()) + .finish() + } +} + +impl Deref for Container { + type Target = Device; + fn deref(&self) -> &Device { + &self.device + } +} + +impl Container { + /// Returns the underlying [`Device`] handle. + pub fn device(&self) -> &Device { + &self.device + } + + /// Returns the runtime container name (lab-unique). + pub fn container_name(&self) -> &str { + &self.inner.name + } + + /// Runs a command inside the running container and returns its output. + /// + /// # Errors + /// + /// Returns an error if the runtime cannot be spawned. A non-zero exit from + /// the command itself is reported in the returned [`std::process::Output`], + /// not as an error. + pub async fn exec(&self, cmd: I) -> Result + where + I: IntoIterator, + S: Into, + { + let mut args = vec!["exec".to_string(), self.inner.name.clone()]; + args.extend(cmd.into_iter().map(Into::into)); + podman(&self.inner.runtime, args).await + } + + /// Returns the container's logs (stdout and stderr, combined). + pub async fn logs(&self) -> Result { + let output = podman( + &self.inner.runtime, + vec!["logs".into(), self.inner.name.clone()], + ) + .await?; + let mut logs = String::from_utf8_lossy(&output.stdout).into_owned(); + logs.push_str(&String::from_utf8_lossy(&output.stderr)); + Ok(logs) + } + + /// Stops and removes the container. + /// + /// Called automatically when the last handle drops; use this to tear it + /// down early or to surface removal errors. + pub async fn stop(&self) -> Result<()> { + let args = vec![ + "rm".into(), + "-f".into(), + "-t".into(), + "1".into(), + self.inner.name.clone(), + ]; + let output = podman(&self.inner.runtime, args).await?; + if !output.status.success() { + bail!( + "removing container '{}' failed: {}", + self.inner.name, + String::from_utf8_lossy(&output.stderr).trim() + ); + } + Ok(()) + } + + /// Copies a host file or directory into the container. + /// + /// The destination's parent directory must already exist in the container. + pub async fn copy_to(&self, host_src: impl AsRef, container_dest: &str) -> Result<()> { + let src = host_src.as_ref().to_string_lossy().into_owned(); + let dest = format!("{}:{}", self.inner.name, container_dest); + let output = podman(&self.inner.runtime, vec!["cp".into(), src, dest]).await?; + if !output.status.success() { + bail!( + "copy into container '{}' failed: {}", + self.inner.name, + String::from_utf8_lossy(&output.stderr).trim() + ); + } + Ok(()) + } + + /// Copies a file or directory out of the container to a host path. + pub async fn copy_from(&self, container_src: &str, host_dest: impl AsRef) -> Result<()> { + let src = format!("{}:{}", self.inner.name, container_src); + let dest = host_dest.as_ref().to_string_lossy().into_owned(); + let output = podman(&self.inner.runtime, vec!["cp".into(), src, dest]).await?; + if !output.status.success() { + bail!( + "copy out of container '{}' failed: {}", + self.inner.name, + String::from_utf8_lossy(&output.stderr).trim() + ); + } + Ok(()) + } + + /// Writes bytes to a file inside the container, creating or replacing it. + /// + /// Accepts anything byte-like (`&str`, `String`, `&[u8]`, ...). The parent + /// directory must already exist in the container. + pub async fn write_file(&self, container_path: &str, contents: impl AsRef<[u8]>) -> Result<()> { + let tmp = unique_temp_path(); + tokio::fs::write(&tmp, contents.as_ref()) + .await + .context("write staging file")?; + let result = self.copy_to(&tmp, container_path).await; + let _ = tokio::fs::remove_file(&tmp).await; + result + } + + /// Reads a file from inside the container. + pub async fn read_file(&self, container_path: &str) -> Result> { + let tmp = unique_temp_path(); + let result = async { + self.copy_from(container_path, &tmp).await?; + tokio::fs::read(&tmp).await.context("read staged file") + } + .await; + let _ = tokio::fs::remove_file(&tmp).await; + result + } + + /// Polls the readiness gate until it passes or times out. + async fn wait_ready(&self, ready: &Ready) -> Result<()> { + let Ready::Tcp { port, timeout } = ready else { + return Ok(()); + }; + let ip: IpAddr = self + .device + .ip() + .map(IpAddr::V4) + .unwrap_or(IpAddr::V4(Ipv4Addr::LOCALHOST)); + let target = SocketAddr::new(ip, *port); + let deadline = Instant::now() + *timeout; + loop { + // Connect from inside the container's namespace, so this measures + // what other lab devices will see. + let connected = self.device.run_sync(move || { + Ok( + std::net::TcpStream::connect_timeout(&target, Duration::from_millis(500)) + .is_ok(), + ) + })?; + if connected { + debug!(container = %self.inner.name, %target, "container ready"); + return Ok(()); + } + if Instant::now() >= deadline { + let logs = self.logs().await.unwrap_or_default(); + bail!( + "container '{}' did not accept a connection on {} within {:?}\n--- logs ---\n{}", + self.inner.name, + target, + timeout, + logs.trim() + ); + } + tokio::time::sleep(Duration::from_millis(200)).await; + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn run_args_have_the_required_flags() { + let args = build_run_args( + "lab-p1-c0", + "docker.io/library/alpine", + &[("A".into(), "1".into()), ("B".into(), "2".into())], + &[], + &["sleep".into(), "infinity".into()], + &["--cgroup-manager=cgroupfs".into()], + ); + // Detached, name, host network, replace. + assert_eq!(&args[0], "run"); + assert!(args.contains(&"-d".to_string())); + assert!(args.contains(&"--network=host".to_string())); + assert!(args.contains(&"--replace".to_string())); + let name_idx = args.iter().position(|a| a == "--name").unwrap(); + assert_eq!(args[name_idx + 1], "lab-p1-c0"); + // Env pairs. + assert!(args.windows(2).any(|w| w[0] == "--env" && w[1] == "A=1")); + assert!(args.windows(2).any(|w| w[0] == "--env" && w[1] == "B=2")); + // Extra run arg comes before the image. + let image_idx = args + .iter() + .position(|a| a == "docker.io/library/alpine") + .unwrap(); + let cgroup_idx = args + .iter() + .position(|a| a == "--cgroup-manager=cgroupfs") + .unwrap(); + assert!(cgroup_idx < image_idx); + // Command args come after the image, in order. + assert_eq!(&args[image_idx + 1], "sleep"); + assert_eq!(&args[image_idx + 2], "infinity"); + } + + #[test] + fn mounts_render_before_the_image() { + let mounts = vec![ + Mount::Bind { + host: "/srv/data".into(), + container: "/data".into(), + read_only: false, + }, + Mount::Bind { + host: "/etc/conf".into(), + container: "/conf".into(), + read_only: true, + }, + Mount::Tmpfs { + container: "/scratch".into(), + }, + ]; + let args = build_run_args("c", "img", &[], &mounts, &[], &[]); + let image_idx = args.iter().position(|a| a == "img").unwrap(); + assert!(args + .windows(2) + .any(|w| w[0] == "--volume" && w[1] == "/srv/data:/data")); + assert!(args + .windows(2) + .any(|w| w[0] == "--volume" && w[1] == "/etc/conf:/conf:ro")); + assert!(args + .windows(2) + .any(|w| w[0] == "--tmpfs" && w[1] == "/scratch")); + // All mount args precede the image. + let last_mount = args + .iter() + .rposition(|a| a == "--volume" || a == "--tmpfs") + .unwrap(); + assert!(last_mount < image_idx); + } + + #[test] + fn sanitize_name_replaces_invalid_chars() { + assert_eq!(sanitize_name("lab-p1/c0"), "lab-p1-c0"); + assert_eq!(sanitize_name("ok_name.1-2"), "ok_name.1-2"); + assert_eq!(sanitize_name("a b:c"), "a-b-c"); + } +} diff --git a/patchbay/src/lab.rs b/patchbay/src/lab.rs index f4dfca0..6573d82 100644 --- a/patchbay/src/lab.rs +++ b/patchbay/src/lab.rs @@ -1058,6 +1058,34 @@ impl Lab { } } + /// Returns a builder for a container node. + /// + /// A container is a device whose workload is an OCI image run with podman, + /// joined to the device's network namespace and reachable from other lab + /// devices at its lab IP. Wire it into the topology like a device, then + /// call [`build`](crate::ContainerBuilder::build): + /// + /// ```no_run + /// # use patchbay::Lab; + /// # async fn f(lab: Lab, net_id: patchbay::NodeId) -> anyhow::Result<()> { + /// let db = lab + /// .add_container("db", "docker.io/library/postgres:16") + /// .uplink(net_id) + /// .env("POSTGRES_PASSWORD", "test") + /// .ready_tcp(5432) + /// .build() + /// .await?; + /// # let _ = db; + /// # Ok(()) + /// # } + /// ``` + /// + /// Requires `podman` on `PATH`; see [`Container`](crate::Container) for why + /// docker does not work. + pub fn add_container(&self, name: &str, image: impl Into) -> crate::ContainerBuilder { + crate::ContainerBuilder::new(self.add_device(name), image) + } + // ── removal ────────────────────────────────────────────────────────── /// Removes a device from the lab, destroying its namespace and all interfaces. diff --git a/patchbay/src/lib.rs b/patchbay/src/lib.rs index 74a4743..ca075ef 100644 --- a/patchbay/src/lib.rs +++ b/patchbay/src/lib.rs @@ -201,6 +201,8 @@ use anyhow::{anyhow, bail, Context, Result}; pub mod config; /// Shared filename constants for the run output directory. pub mod consts; +/// Run OCI containers as lab devices. +pub(crate) mod container; pub(crate) mod core; /// Device handle and builder. pub(crate) mod device; @@ -235,6 +237,7 @@ pub mod util; pub(crate) mod wiring; pub(crate) mod writer; +pub use container::{Container, ContainerBuilder}; pub use device::{Device, DeviceBuilder}; pub use firewall::PortPolicy; pub use iface::{Iface, IfaceConfig}; diff --git a/patchbay/src/tests/container.rs b/patchbay/src/tests/container.rs new file mode 100644 index 0000000..8439e85 --- /dev/null +++ b/patchbay/src/tests/container.rs @@ -0,0 +1,140 @@ +//! Container-as-device integration: a containerized service is reachable from a +//! lab device at the container's lab IP. +//! +//! Gated on `podman` being available and pulls a small public image, so it +//! skips cleanly where podman is absent (for example in a minimal CI image). + +use std::io::{Read, Write}; + +use super::*; + +/// Returns `true` if a working `podman` is on `PATH`. +fn podman_available() -> bool { + std::process::Command::new("podman") + .arg("--version") + .output() + .map(|o| o.status.success()) + .unwrap_or(false) +} + +/// A container attached to a router is reachable from another device on it. +/// +/// Runs `nginx:alpine`, waits for port 80 to accept, then fetches `/` from a +/// separate client device and checks for an nginx response. +#[tokio::test(flavor = "current_thread")] +#[traced_test] +async fn container_service_reachable_from_device() -> Result<()> { + check_caps()?; + if !podman_available() { + eprintln!("skipping container_service_reachable_from_device: podman not on PATH"); + return Ok(()); + } + + let lab = Lab::new().await?; + let net = lab.add_router("net").build().await?; + let web = lab + .add_container("web", "docker.io/library/nginx:alpine") + .uplink(net.id()) + .ready_tcp(80) + .build() + .await?; + let client = lab.add_device("client").uplink(net.id()).build().await?; + + let web_ip = web.ip().context("container has no ip")?; + let addr = SocketAddr::new(IpAddr::V4(web_ip), 80); + + // Fetch `/` from inside the client namespace, over the lab network. + let response = client.run_sync(move || { + let mut stream = std::net::TcpStream::connect_timeout(&addr, Duration::from_secs(5)) + .context("connect to container")?; + stream.write_all(b"GET / HTTP/1.0\r\nHost: web\r\n\r\n")?; + let mut buf = String::new(); + stream.read_to_string(&mut buf)?; + Ok(buf) + })?; + + let status = response.lines().next().unwrap_or_default(); + assert!( + status.contains("200") || response.to_lowercase().contains("nginx"), + "expected an nginx response from the container, got status line: {status:?}" + ); + Ok(()) +} + +/// A read-only bind mount is visible inside the container. +#[tokio::test(flavor = "current_thread")] +#[traced_test] +async fn container_bind_mount_visible() -> Result<()> { + check_caps()?; + if !podman_available() { + eprintln!("skipping container_bind_mount_visible: podman not on PATH"); + return Ok(()); + } + + let dir = std::env::temp_dir().join(format!("patchbay-vol-{}", std::process::id())); + std::fs::create_dir_all(&dir).context("create mount dir")?; + std::fs::write(dir.join("hello.txt"), "patchbay-volume").context("write mount file")?; + + let lab = Lab::new().await?; + let net = lab.add_router("net").build().await?; + let box_ = lab + .add_container("box", "docker.io/library/alpine:latest") + .uplink(net.id()) + .volume_ro(&dir, "/data") + .args(["sleep", "infinity"]) + .build() + .await?; + + let out = box_.exec(["cat", "/data/hello.txt"]).await?; + let _ = std::fs::remove_dir_all(&dir); + assert!( + out.status.success(), + "cat failed: {}", + String::from_utf8_lossy(&out.stderr).trim() + ); + assert_eq!( + String::from_utf8_lossy(&out.stdout).trim(), + "patchbay-volume" + ); + Ok(()) +} + +/// write_file/read_file round-trip and copy_to/copy_from work against a +/// running container. +#[tokio::test(flavor = "current_thread")] +#[traced_test] +async fn container_file_transfer() -> Result<()> { + check_caps()?; + if !podman_available() { + eprintln!("skipping container_file_transfer: podman not on PATH"); + return Ok(()); + } + + let dir = std::env::temp_dir().join(format!("patchbay-xfer-{}", std::process::id())); + std::fs::create_dir_all(&dir).context("create work dir")?; + + let lab = Lab::new().await?; + let net = lab.add_router("net").build().await?; + let box_ = lab + .add_container("box", "docker.io/library/alpine:latest") + .uplink(net.id()) + .args(["sleep", "infinity"]) + .build() + .await?; + + // write_file then read_file round-trips. + box_.write_file("/tmp/greeting", "hello-from-host").await?; + assert_eq!(box_.read_file("/tmp/greeting").await?, b"hello-from-host"); + + // copy_from a container file to the host. + box_.copy_from("/tmp/greeting", dir.join("out.txt")).await?; + assert_eq!(std::fs::read(dir.join("out.txt"))?, b"hello-from-host"); + + // copy_to a host file into the container. + std::fs::write(dir.join("in.txt"), "from-copy-to")?; + box_.copy_to(dir.join("in.txt"), "/tmp/in.txt").await?; + assert_eq!(box_.read_file("/tmp/in.txt").await?, b"from-copy-to"); + + let _ = std::fs::remove_dir_all(&dir); + Ok(()) +} diff --git a/patchbay/src/tests/mod.rs b/patchbay/src/tests/mod.rs index 945d984..655ec8c 100644 --- a/patchbay/src/tests/mod.rs +++ b/patchbay/src/tests/mod.rs @@ -34,6 +34,7 @@ use super::*; use crate::{check_caps, config}; mod alloc; +mod container; mod devtools; mod dns; mod firewall; diff --git a/patchbay/src/userns.rs b/patchbay/src/userns.rs index ad31d21..ee377ee 100644 --- a/patchbay/src/userns.rs +++ b/patchbay/src/userns.rs @@ -36,13 +36,19 @@ pub unsafe fn init_userns_for_ctor() { #[cfg(target_os = "linux")] fn do_bootstrap() -> anyhow::Result<()> { use anyhow::Context; - use nix::sched::{unshare, CloneFlags}; let uid = nix::unistd::Uid::current().as_raw(); let gid = nix::unistd::Gid::current().as_raw(); - unshare(CloneFlags::CLONE_NEWUSER).context("unshare(CLONE_NEWUSER) failed")?; - std::fs::write("/proc/self/setgroups", "deny\n").context("write setgroups")?; - std::fs::write("/proc/self/uid_map", format!("0 {uid} 1\n")).context("write uid_map")?; - std::fs::write("/proc/self/gid_map", format!("0 {gid} 1\n")).context("write gid_map")?; + + // Prefer a full subuid/subgid range map when the host is set up for it + // (newuidmap/newgidmap on PATH plus /etc/subuid,/etc/subgid entries). The + // extra ids let nested rootless container runtimes chown image files and + // mount overlay; a single-uid map cannot. Fall back to the single-uid map, + // which needs no host configuration, when the range is unavailable. + match RangeParams::detect(uid, gid) { + Some(range) => range_bootstrap(uid, gid, &range).context("range userns bootstrap")?, + None => single_uid_bootstrap(uid, gid).context("single-uid userns bootstrap")?, + } + if nix::unistd::Uid::effective().is_root() { Ok(()) } else { @@ -50,6 +56,167 @@ fn do_bootstrap() -> anyhow::Result<()> { } } +/// Single-entry map (inner 0 to the current uid/gid). Needs no host setup. +#[cfg(target_os = "linux")] +fn single_uid_bootstrap(uid: u32, gid: u32) -> anyhow::Result<()> { + use anyhow::Context; + use nix::sched::{unshare, CloneFlags}; + unshare(CloneFlags::CLONE_NEWUSER).context("unshare(CLONE_NEWUSER) failed")?; + std::fs::write("/proc/self/setgroups", "deny\n").context("write setgroups")?; + std::fs::write("/proc/self/uid_map", format!("0 {uid} 1\n")).context("write uid_map")?; + std::fs::write("/proc/self/gid_map", format!("0 {gid} 1\n")).context("write gid_map")?; + Ok(()) +} + +/// Sub-id ranges and helper paths for a full range map. +#[cfg(target_os = "linux")] +struct RangeParams { + newuidmap: std::path::PathBuf, + newgidmap: std::path::PathBuf, + sub_uid_start: u32, + sub_uid_count: u32, + sub_gid_start: u32, + sub_gid_count: u32, +} + +#[cfg(target_os = "linux")] +impl RangeParams { + /// Detects whether a range map is possible for the current user. + /// + /// Requires `newuidmap` and `newgidmap` on `PATH` and an `/etc/subuid` and + /// `/etc/subgid` entry for the user. Returns `None` (use the single-uid + /// fallback) if anything is missing. + fn detect(uid: u32, gid: u32) -> Option { + let newuidmap = find_in_path("newuidmap")?; + let newgidmap = find_in_path("newgidmap")?; + let user = nix::unistd::User::from_uid(nix::unistd::Uid::from_raw(uid)).ok()??; + let (sub_uid_start, sub_uid_count) = parse_subid("/etc/subuid", &user.name, uid)?; + let (sub_gid_start, sub_gid_count) = parse_subid("/etc/subgid", &user.name, gid)?; + Some(Self { + newuidmap, + newgidmap, + sub_uid_start, + sub_uid_count, + sub_gid_start, + sub_gid_count, + }) + } +} + +/// Maps inner 0 to the current uid plus the sub-id range to inner 1.., using the +/// setuid `newuidmap`/`newgidmap` helpers. +/// +/// The current process enters the new user namespace, then a short-lived child +/// (still in the parent namespace, where the setuid helpers have authority) +/// writes the range maps against the parent's pid. +#[cfg(target_os = "linux")] +fn range_bootstrap(uid: u32, gid: u32, range: &RangeParams) -> anyhow::Result<()> { + use std::os::fd::AsFd; + + use anyhow::{bail, Context}; + use nix::{ + sched::{unshare, CloneFlags}, + sys::wait::waitpid, + unistd::{fork, getpid, pipe, read, write, ForkResult}, + }; + + // p2c signals the child once the parent has unshared (and carries the + // parent pid); c2p carries the child's success byte back. + let (p2c_r, p2c_w) = pipe().context("pipe")?; + let (c2p_r, c2p_w) = pipe().context("pipe")?; + + // SAFETY: called single-threaded during bootstrap (before Tokio starts). + // The child only reads a pid, execs the setuid helpers, and `_exit`s. + match unsafe { fork() }.context("fork")? { + ForkResult::Child => { + drop(p2c_w); + drop(c2p_r); + let mut pid_buf = [0u8; 4]; + let ok = read(p2c_r.as_fd(), &mut pid_buf).is_ok_and(|n| n == 4) && { + let target = u32::from_ne_bytes(pid_buf).to_string(); + run_idmap( + &range.newuidmap, + &target, + uid, + range.sub_uid_start, + range.sub_uid_count, + ) && run_idmap( + &range.newgidmap, + &target, + gid, + range.sub_gid_start, + range.sub_gid_count, + ) + }; + let _ = write(c2p_w.as_fd(), &[u8::from(ok)]); + // SAFETY: exit without running atexit handlers in the forked child. + unsafe { libc::_exit(0) }; + } + ForkResult::Parent { child } => { + drop(p2c_r); + drop(c2p_w); + unshare(CloneFlags::CLONE_NEWUSER).context("unshare(CLONE_NEWUSER) failed")?; + let pid = getpid().as_raw() as u32; + write(p2c_w.as_fd(), &pid.to_ne_bytes()).context("signal child")?; + let mut status = [0u8; 1]; + let got = read(c2p_r.as_fd(), &mut status).context("await child")?; + let _ = waitpid(child, None); + drop(p2c_w); + if got != 1 || status[0] != 1 { + bail!("newuidmap/newgidmap failed to write the range map"); + } + Ok(()) + } + } +} + +/// Runs `helper 0 1 1 `, mapping inner 0 to +/// `id` and the sub-id range to inner 1. +#[cfg(target_os = "linux")] +fn run_idmap(helper: &std::path::Path, pid: &str, id: u32, sub_start: u32, sub_count: u32) -> bool { + std::process::Command::new(helper) + .args([ + pid, + "0", + &id.to_string(), + "1", + "1", + &sub_start.to_string(), + &sub_count.to_string(), + ]) + .status() + .map(|s| s.success()) + .unwrap_or(false) +} + +/// Finds an executable by name on `PATH`. +#[cfg(target_os = "linux")] +fn find_in_path(name: &str) -> Option { + let path = std::env::var_os("PATH")?; + std::env::split_paths(&path) + .map(|dir| dir.join(name)) + .find(|candidate| candidate.is_file()) +} + +/// Parses an `/etc/subuid`-style file for the `login:start:count` entry that +/// matches the user by name or numeric id. +#[cfg(target_os = "linux")] +fn parse_subid(path: &str, name: &str, id: u32) -> Option<(u32, u32)> { + let contents = std::fs::read_to_string(path).ok()?; + let id_str = id.to_string(); + for line in contents.lines() { + let mut fields = line.split(':'); + let login = fields.next()?; + if login != name && login != id_str { + continue; + } + let start = fields.next()?.parse().ok()?; + let count = fields.next()?.parse().ok()?; + return Some((start, count)); + } + None +} + #[cfg(not(target_os = "linux"))] fn do_bootstrap() -> anyhow::Result<()> { Ok(())