diff --git a/TESTING.md b/TESTING.md index 084535630a..988e745b3c 100644 --- a/TESTING.md +++ b/TESTING.md @@ -22,6 +22,11 @@ tests/tmachine/ # QEMU test machines, Ansible playbooks, and the Session tests/runner/ # openshell-test-runner: runs a test family in a tmachine environment ``` +## Performance benchmarks + +Run `mise run perf:seccomp` for the native/filtered seccomp broker +microbenchmark. + ## Rust Tests Unit tests live inline with `#[cfg(test)] mod tests` blocks. Integration tests diff --git a/crates/openshell-sandbox/README.md b/crates/openshell-sandbox/README.md index e365721b8e..7e5dc6db3a 100644 --- a/crates/openshell-sandbox/README.md +++ b/crates/openshell-sandbox/README.md @@ -25,6 +25,57 @@ the last workload reference to a listener removes it without waiting for registr collection. Broker source descriptors always use close-on-exec independently of the workload's requested descriptor flags. +A blocking loopback `connect` waits on a small pool of reusable worker threads +rather than a thread per call. A job never queues behind a busy worker: it goes +to a parked worker or starts a new one, and a pool with neither refuses with +`EAGAIN`, so a slow or full listener cannot delay another connect or the +dispatcher. Three bounds hold together. Each job carries an admission slot +shared with external opens (256 in all), which its worker releases only after +the pool records that worker parked or retired. Live workers are capped at that +same number under the pool's own lock. At most four workers stay parked, for up +to 30 seconds; a worker that finishes beyond that exits at once. + +`setsockopt` calls for `TCP_NODELAY`, `SO_REUSEADDR`, and `SO_KEEPALIVE` run in +the kernel without a notification. The broker decides `setsockopt` from its two +scalar arguments and would continue these pairs. Every other pair, and any +level or option that does not fit a non-negative C `int`, still notifies and +keeps the broker's `EPERM` and `EINVAL` results. The active qualification +probe exercises both sides. + +## Provider files + +Provider files appear to the workload beneath `/run/openshell/providers`. +Every workload `open`, `openat`, and `openat2` notifies the broker, which +serves a provider path by injecting a read-only descriptor for a sealed +in-memory file and continues every other open. + +Each published file is an immutable revision backed by one in-memory file. The +runtime creates that file when the revision is published, not when the +workload opens it: it writes the content, seals the file against writes, +growing, shrinking, and further sealing, and keeps only a read-only descriptor +for it. Each workload open reopens that descriptor through the runtime's own +`/proc/self/fd`, so every open gets its own read-only open file description +and file offset, positioned at the start. + +An update prepares every new revision before it replaces the published set in +one step, so a failed update leaves the previous set in place. A file whose +bytes did not change keeps its revision. Opens after an update see the new +revisions, descriptors opened earlier keep the bytes they opened, and a +detached file is `ENOENT` for later opens. + +All opens of one revision refer to one file, which the workload owns: it can +change that file's mode or reopen a descriptor for writing. Neither reaches +the content, which the seals protect, and the runtime restores the mode if it +denies an open. The runtime holds one descriptor per published file, at most +64. + +The broker classifies each open from an exact copy of the first six bytes of +its path. Six bytes identify a `/proc/` path and the start of a provider path; +only those candidates have their whole path read and decided. An open whose +first six bytes cannot be read, or match neither, continues, and unrelated +`openat2` calls do not require a copy of their `open_how`. Every continued +open remains subject to the kernel and Landlock. + A boundary configured with `RuntimeInitiatedTlsTcp` binds no listener. It dials only the driver-provided supervisor addresses, which must be concrete, non-loopback, and share one port, and serves TLS and gRPC over that outbound diff --git a/crates/openshell-sandbox/src/bin/seccomp-perf.rs b/crates/openshell-sandbox/src/bin/seccomp-perf.rs index 09bda3449f..4acea88e76 100644 --- a/crates/openshell-sandbox/src/bin/seccomp-perf.rs +++ b/crates/openshell-sandbox/src/bin/seccomp-perf.rs @@ -15,21 +15,51 @@ use clap::builder::{PossibleValue, PossibleValuesParser}; #[cfg(target_os = "linux")] use clap::{Parser, Subcommand, ValueEnum as _}; #[cfg(target_os = "linux")] -use openshell_sandbox::perf::{BenchmarkOptions, Layer, Protocol}; +use openshell_sandbox::perf::{BenchmarkOptions, Layer, Protocol, WorkerRun}; #[cfg(target_os = "linux")] #[derive(Debug, Parser)] #[command( - about = "Measure native and seccomp-filtered socket performance", - long_about = "Measure native and seccomp-filtered TCP socket performance." + about = "Measure native and seccomp-filtered socket and file-open performance", + long_about = "Measure native and seccomp-filtered TCP socket and file-open performance.\n\n\ + Layer `notify-only` and `--landlock` attribute the cost of a workload file open; \ + see the perf module documentation." )] struct Cli { - /// Benchmark layer: native, filtered, or all. - #[arg(long, default_value = "all", value_parser = ["native", "filtered", "all"])] + /// Benchmark layer. `all` is native and filtered; `attribution` adds the + /// harness-only notify-only layer. + #[arg( + long, + default_value = "all", + value_parser = ["native", "filtered", "notify-only", "all", "attribution"] + )] layer: String, - /// Implemented protocol to benchmark, or all. + /// Protocol to benchmark. `all` is the TCP pair; `file` is the file-open + /// attribution set; `mixed` runs file opens and TCP operations at the + /// same time, on `concurrency` threads each, and reports each class. #[arg(long, default_value = "all", value_parser = protocol_value_parser())] protocol: String, + /// Confine each worker with a Landlock domain before it measures. + #[arg(long)] + landlock: bool, + /// File the file protocols open. + #[arg(long, default_value = "/etc/passwd")] + file_path: std::path::PathBuf, + /// Publish one synthetic provider file and make the file protocols open + /// it, so the broker serves each open by injection. Requires + /// `--layer filtered`; no other layer has virtual provider files. + #[arg(long, conflicts_with = "file_path")] + provider_file: bool, + /// Size of the synthetic provider file. + #[arg(long, default_value_t = 4_096, requires = "provider_file")] + provider_file_bytes: usize, + /// Experiment: request the kernel's synchronous-wakeup hint on the + /// listener of each filtered layer. The production runtime never does. + #[arg(long)] + sync_wakeup: bool, + /// Experiment: threads answering notifications in the notify-only layer. + #[arg(long, default_value_t = 1)] + notify_receivers: usize, #[arg(long, default_value_t = 10_000)] iterations: u64, #[arg(long, default_value_t = 1_000)] @@ -44,7 +74,11 @@ struct Cli { #[cfg(target_os = "linux")] fn protocol_value_parser() -> PossibleValuesParser { - let mut values = vec![PossibleValue::new("all")]; + let mut values = vec![ + PossibleValue::new("all"), + PossibleValue::new("file"), + PossibleValue::new("mixed"), + ]; values.extend( Protocol::value_variants() .iter() @@ -70,6 +104,12 @@ enum Command { concurrency: usize, #[arg(long)] payload_bytes: usize, + #[arg(long)] + landlock: bool, + #[arg(long, default_value = "/etc/passwd")] + file_path: std::path::PathBuf, + #[arg(long)] + provider_file_bytes: Option, }, } @@ -83,17 +123,25 @@ fn main() -> anyhow::Result<()> { warmup, concurrency, payload_bytes, + landlock, + file_path, + provider_file_bytes, }) = cli.command { - let report = openshell_sandbox::perf::run_worker( + let reports = openshell_sandbox::perf::run_worker(&WorkerRun { protocol, target, iterations, warmup, concurrency, payload_bytes, - )?; - println!("{}", serde_json::to_string(&report)?); + landlock, + file_path, + provider_file_bytes, + })?; + for report in reports { + println!("{}", serde_json::to_string(&report)?); + } return Ok(()); } @@ -104,6 +152,11 @@ fn main() -> anyhow::Result<()> { warmup: cli.warmup, concurrency: cli.concurrency, payload_bytes: cli.payload_bytes, + landlock: cli.landlock, + file_path: cli.file_path, + sync_wakeup: cli.sync_wakeup, + notify_receivers: cli.notify_receivers, + provider_file_bytes: cli.provider_file.then_some(cli.provider_file_bytes), }; for report in openshell_sandbox::perf::run(options)? { println!("{}", serde_json::to_string(&report)?); diff --git a/crates/openshell-sandbox/src/linux/proc_fd.rs b/crates/openshell-sandbox/src/linux/proc_fd.rs index 729b60e03f..227b5bbdb9 100644 --- a/crates/openshell-sandbox/src/linux/proc_fd.rs +++ b/crates/openshell-sandbox/src/linux/proc_fd.rs @@ -52,6 +52,80 @@ pub fn installed_socket_inodes_excluding( Ok(inodes) } +/// Count this process's open descriptors, as one enumeration of +/// `/proc/self/fd` reports them. +/// +/// An enumeration opens the directory it lists, so its result is the number +/// of descriptors open before the call plus one. Callers size their headroom +/// against that value. Linux 6.2 and later report the open-descriptor count as +/// the size of `/proc/self/fd`, which costs one `stat` instead of an open, +/// one directory read per batch of entries, and a close. Support is +/// established once by an active probe, never from the kernel version, and +/// the fast count adds one to stay equal to an enumeration. Any kernel, +/// profile, or procfs that does not pass the probe keeps enumerating. +pub fn open_descriptor_count() -> io::Result { + static STAT_COUNT_SUPPORTED: std::sync::OnceLock = std::sync::OnceLock::new(); + let supported = *STAT_COUNT_SUPPORTED.get_or_init(|| { + stat_count_tracks_enumeration( + stat_descriptor_count, + || enumerate_descriptor_count().ok(), + || rustix::event::eventfd(0, rustix::event::EventfdFlags::CLOEXEC).ok(), + ) + }); + if supported && let Some(open) = stat_descriptor_count() { + return Ok(open.saturating_add(1)); + } + enumerate_descriptor_count() +} + +fn enumerate_descriptor_count() -> io::Result { + fs::read_dir("/proc/self/fd").map(Iterator::count) +} + +/// Open-descriptor count from the directory size, or `None` where the kernel +/// does not report one (older kernels report a size of zero). +fn stat_descriptor_count() -> Option { + let size = fs::metadata("/proc/self/fd").ok()?.len(); + usize::try_from(size).ok().filter(|count| *count > 0) +} + +/// Decide whether `stat` reports a live descriptor count. +/// +/// A sample qualifies only when two `stat` counts bracketing an enumeration +/// agree, the enumeration reports exactly one more (its own directory +/// descriptor), and holding one extra descriptor raises the `stat` count by +/// exactly one. Other threads open and close descriptors concurrently, so a +/// disturbed sample is retried a few times; persistent disagreement selects +/// enumeration. +fn stat_count_tracks_enumeration( + mut stat: impl FnMut() -> Option, + mut enumerate: impl FnMut() -> Option, + mut hold_descriptor: impl FnMut() -> Option, +) -> bool { + const ATTEMPTS: usize = 8; + for _ in 0..ATTEMPTS { + let Some(before) = stat() else { + return false; + }; + let Some(enumerated) = enumerate() else { + return false; + }; + let Some(held) = hold_descriptor() else { + return false; + }; + let with_held = stat(); + drop(held); + let after = stat(); + if after == Some(before) + && with_held == before.checked_add(1) + && Some(enumerated) == before.checked_add(1) + { + return true; + } + } + false +} + /// Return the socket inode currently installed at `fd` in `tid`'s descriptor /// table. /// @@ -184,4 +258,65 @@ mod tests { .contains(&inode) ); } + + #[test] + fn stat_count_is_used_only_when_it_tracks_descriptors() { + let hold = || Some(()); + // A kernel that reports no size never qualifies. + assert!(!stat_count_tracks_enumeration(|| None, || Some(8), hold)); + // A size that ignores a newly held descriptor is not a count. + assert!(!stat_count_tracks_enumeration(|| Some(7), || Some(8), hold)); + // An enumeration that does not include its own descriptor disagrees. + let mut counts = [7, 8, 7].into_iter().cycle(); + assert!(!stat_count_tracks_enumeration( + || counts.next(), + || Some(7), + hold + )); + // A consistent sample qualifies: before, with one held, after. + let mut counts = [7, 8, 7].into_iter().cycle(); + assert!(stat_count_tracks_enumeration( + || counts.next(), + || Some(8), + hold + )); + // A sample disturbed by concurrent descriptor churn is retried. + let mut counts = [7, 9, 8, 8, 9, 8].into_iter(); + assert!(stat_count_tracks_enumeration( + || counts.next(), + || Some(9), + hold + )); + // Failing to enumerate or to hold a descriptor never qualifies. + assert!(!stat_count_tracks_enumeration(|| Some(7), || None, hold)); + assert!(!stat_count_tracks_enumeration( + || Some(7), + || Some(8), + || None::<()> + )); + } + + #[test] + fn descriptor_count_follows_open_descriptors() { + // Other tests open and close descriptors concurrently, so compare + // against a large deliberate change rather than an exact value. + const HELD: usize = 64; + let before = open_descriptor_count().unwrap(); + assert!(before > 3, "count includes the standard descriptors"); + let held = (0..HELD) + .map(|_| File::open("/dev/null").expect("open descriptor")) + .collect::>(); + let during = open_descriptor_count().unwrap(); + drop(held); + let after = open_descriptor_count().unwrap(); + assert!(during >= before + HELD / 2, "{before} -> {during}"); + assert!(after + HELD / 2 <= during, "{during} -> {after}"); + // The fast count and an enumeration agree to within concurrent churn. + let enumerated = enumerate_descriptor_count().unwrap(); + let counted = open_descriptor_count().unwrap(); + assert!( + enumerated.abs_diff(counted) < HELD / 2, + "{enumerated} vs {counted}" + ); + } } diff --git a/crates/openshell-sandbox/src/linux/seccomp_notify.rs b/crates/openshell-sandbox/src/linux/seccomp_notify.rs index 665cadcefe..cf513f0ee2 100644 --- a/crates/openshell-sandbox/src/linux/seccomp_notify.rs +++ b/crates/openshell-sandbox/src/linux/seccomp_notify.rs @@ -27,7 +27,7 @@ const SECCOMP_RET_ALLOW: u32 = 0x7fff_0000; const BPF_LD_W_ABS: u16 = 0x20; const BPF_JMP_JEQ_K: u16 = 0x15; -#[cfg(target_arch = "x86_64")] +#[cfg(any(test, target_arch = "x86_64"))] const BPF_JMP_JSET_K: u16 = 0x45; const BPF_ALU_AND_K: u16 = 0x54; const BPF_RET_K: u16 = 0x06; @@ -46,6 +46,22 @@ const CONNECTED_SEND_FLAGS: u32 = as u32; const PROBE_NOTIFICATION_TIMEOUT: Duration = Duration::from_secs(5); +/// Scalar `(level, option)` pairs `setsockopt` may set without a broker round +/// trip. +/// +/// The broker decides `setsockopt` from these two scalar arguments alone and +/// continues every pair it does not deny, so allowing an exact pair in the +/// filter returns the same result without waking the broker. The list is a +/// narrow whitelist rather than the complement of the broker's denied set: +/// every other pair, and every call whose level or option does not fit a +/// non-negative C `int`, still notifies and keeps the broker's `EPERM` and +/// `EINVAL` results. +pub(crate) const SETSOCKOPT_FAST_PATH: &[(i32, i32)] = &[ + (libc::IPPROTO_TCP, libc::TCP_NODELAY), + (libc::SOL_SOCKET, libc::SO_REUSEADDR), + (libc::SOL_SOCKET, libc::SO_KEEPALIVE), +]; + const IOC_NRBITS: u32 = 8; const IOC_TYPEBITS: u32 = 8; const IOC_SIZEBITS: u32 = 14; @@ -122,6 +138,10 @@ const SECCOMP_IOCTL_NOTIF_RECV: libc::c_ulong = iowr::(0); const SECCOMP_IOCTL_NOTIF_SEND: libc::c_ulong = iowr::(1); const SECCOMP_IOCTL_NOTIF_ID_VALID: libc::c_ulong = iow::(2); const SECCOMP_IOCTL_NOTIF_ADDFD: libc::c_ulong = iow::(3); +#[cfg(feature = "perf-harness")] +const SECCOMP_IOCTL_NOTIF_SET_FLAGS: libc::c_ulong = iow::(4); +#[cfg(feature = "perf-harness")] +const SECCOMP_USER_NOTIF_FD_SYNC_WAKE_UP: libc::c_ulong = 1; /// One validated seccomp user-notification request. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -165,6 +185,13 @@ impl NotificationProbeReport { pub fn connected_send_fast_path(self) -> bool { self.features.0 & 4 != 0 } + + /// Whether whitelisted scalar `setsockopt` pairs bypassed notification + /// while other pairs and unrepresentable arguments remained mediated. + #[must_use] + pub fn scalar_setsockopt_fast_path(self) -> bool { + self.features.0 & 8 != 0 + } } /// Owned listener returned by `SECCOMP_FILTER_FLAG_NEW_LISTENER`. @@ -185,6 +212,39 @@ impl NotificationListener { self.fd.as_raw_fd() } + /// Ask the kernel to wake this listener synchronously, on the notifying + /// task's CPU (Linux 6.6 and newer). + /// + /// Only the opt-in performance harness can request this. Production + /// listeners keep the default wakeup: the hint was measured and not + /// adopted. + #[cfg(feature = "perf-harness")] + pub fn enable_sync_wakeup(&self) -> io::Result<()> { + let request = SECCOMP_IOCTL_NOTIF_SET_FLAGS; + // musl models the request as `c_int`; the value fits either type. + #[cfg(target_env = "musl")] + let request = libc::c_int::from_ne_bytes( + u32::try_from(request) + .map_err(|_| { + io::Error::new(io::ErrorKind::InvalidInput, "ioctl request exceeds 32 bits") + })? + .to_ne_bytes(), + ); + // SAFETY: this ioctl takes its flags by value, not through a pointer. + let result = unsafe { + libc::ioctl( + self.fd.as_raw_fd(), + request, + SECCOMP_USER_NOTIF_FD_SYNC_WAKE_UP, + ) + }; + if result < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(()) + } + } + /// Receive the next kernel notification. pub fn receive(&self) -> io::Result { let mut raw = RawNotification::default(); @@ -316,10 +376,14 @@ pub fn install_listener(syscalls: &[i64]) -> io::Result { "at least one notified syscall is required", )); } - verify_notification_sizes()?; - set_no_new_privileges()?; - - install_notification_filter(syscalls) + let mut syscalls = syscalls.to_vec(); + syscalls.sort_unstable(); + syscalls.dedup(); + let rules = syscalls + .into_iter() + .map(FilterRule::Notify) + .collect::>(); + install_rules(&rules) } /// Install the capability-free workload networking listener on the calling @@ -329,8 +393,23 @@ pub fn install_listener(syscalls: &[i64]) -> io::Result { /// reconfigure an INET endpoint. Connected `send()`/null-destination /// `sendto()` retains the audited cBPF fast path. `accept`/`accept4` run /// natively: the broker binds every INET socket to loopback before injecting -/// it, and accepted sockets inherit their listener's binding. +/// it, and accepted sockets inherit their listener's binding. `setsockopt` +/// allows the [`SETSOCKOPT_FAST_PATH`] pairs in the filter. Every `open`, +/// `openat`, and `openat2` notifies: the broker serves provider files by +/// descriptor injection. pub fn install_workload_listener() -> io::Result { + install_rules(&workload_filter_rules()) +} + +/// Qualify the notification ABI, set `no_new_privs` on the calling thread, +/// and install `rules` there. +fn install_rules(rules: &[FilterRule]) -> io::Result { + verify_notification_sizes()?; + set_no_new_privileges()?; + install_notification_filter(rules) +} + +fn workload_filter_rules() -> Vec { #[allow(unused_mut)] // SYS_open is unavailable on some architectures. let mut syscalls = vec![ libc::SYS_socket, @@ -351,7 +430,34 @@ pub fn install_workload_listener() -> io::Result { ]; #[cfg(target_arch = "x86_64")] syscalls.push(libc::SYS_open); - install_listener(&syscalls) + syscalls + .into_iter() + .map(|syscall| { + if syscall == libc::SYS_setsockopt { + return FilterRule::ScalarSetsockopt; + } + FilterRule::Notify(syscall) + }) + .collect() +} + +/// One syscall's treatment in a listener filter. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum FilterRule { + /// Notify with the syscall's standing treatment: connected `sendto` and + /// non-`SIGCONT` thread signals keep their audited fast paths. + Notify(i64), + /// `setsockopt`, allowing only [`SETSOCKOPT_FAST_PATH`] pairs natively. + ScalarSetsockopt, +} + +impl FilterRule { + const fn syscall(self) -> i64 { + match self { + Self::Notify(syscall) => syscall, + Self::ScalarSetsockopt => libc::SYS_setsockopt, + } + } } /// Run a no-capability conformance probe. @@ -363,8 +469,9 @@ pub fn probe_notification_api() -> io::Result { probe_scalar_round_trip()?; probe_addfd_send()?; probe_connected_sendto_fast_path()?; + probe_scalar_setsockopt_fast_path()?; Ok(NotificationProbeReport { - features: NotificationProbeFeatures(1 | 2 | 4), + features: NotificationProbeFeatures(1 | 2 | 4 | 8), }) } @@ -586,6 +693,129 @@ fn probe_connected_sendto_fast_path() -> io::Result<()> { Ok(()) } +/// Raw `setsockopt` whose level and option keep all 64 argument bits, so the +/// probe can present values a C `int` prototype would truncate. +fn raw_setsockopt(fd: RawFd, level: u64, option: u64) -> io::Result<()> { + let value: libc::c_int = 1; + // SAFETY: the value pointer and length describe one live C int. + let result = unsafe { + libc::syscall( + libc::SYS_setsockopt, + fd, + level, + option, + std::ptr::addr_of!(value), + size_of::(), + ) + }; + if result < 0 { + Err(io::Error::last_os_error()) + } else { + Ok(()) + } +} + +fn expect_errno(result: io::Result<()>, errno: i32, what: &str) -> io::Result<()> { + match result { + Err(error) if error.raw_os_error() == Some(errno) => Ok(()), + Ok(()) => Err(io::Error::other(format!("{what} unexpectedly succeeded"))), + Err(error) => Err(io::Error::other(format!( + "{what} returned {error} instead of errno {errno}" + ))), + } +} + +const fn unsigned_argument(value: libc::c_int) -> u64 { + value.unsigned_abs() as u64 +} + +fn probe_scalar_setsockopt_fast_path() -> io::Result<()> { + const HIGH_WORD: u64 = 1 << 32; + const ABOVE_INT_MAX: u64 = 0x8000_0000; + let (socket, _peer) = std::os::unix::net::UnixStream::pair()?; + let level_tcp = unsigned_argument(libc::IPPROTO_TCP); + let level_socket = unsigned_argument(libc::SOL_SOCKET); + let nodelay = unsigned_argument(libc::TCP_NODELAY); + let reuse = unsigned_argument(libc::SO_REUSEADDR); + let bind_device = unsigned_argument(libc::SO_BINDTODEVICE); + + let (sender, receiver) = mpsc::sync_channel(1); + let launcher = thread::spawn(move || -> io::Result<()> { + let listener = install_rules(&[FilterRule::ScalarSetsockopt])?; + sender + .send(listener) + .map_err(|_| io::Error::other("setsockopt broker disappeared"))?; + let fd = socket.as_raw_fd(); + // Whitelisted pairs run in the kernel. SO_REUSEADDR succeeds on any + // socket; TCP_NODELAY on this UNIX socket returns the kernel's own + // EOPNOTSUPP, which no broker response in this probe produces. + raw_setsockopt(fd, level_socket, reuse)?; + expect_errno( + raw_setsockopt(fd, level_tcp, nodelay), + libc::EOPNOTSUPP, + "native TCP_NODELAY on a UNIX socket", + )?; + // Everything else notifies, in this order. + expect_errno( + raw_setsockopt(fd, level_socket, bind_device), + libc::EPERM, + "denied socket option", + )?; + expect_errno( + raw_setsockopt(fd, level_tcp | HIGH_WORD, nodelay), + libc::EINVAL, + "socket option level with a nonzero upper word", + )?; + expect_errno( + raw_setsockopt(fd, level_tcp, nodelay | HIGH_WORD), + libc::EINVAL, + "socket option name with a nonzero upper word", + )?; + expect_errno( + raw_setsockopt(fd, level_tcp, nodelay | ABOVE_INT_MAX), + libc::EINVAL, + "socket option name above INT_MAX", + ) + }); + + let Ok(listener) = receiver.recv() else { + return Err(launcher_failure(launcher, "setsockopt")); + }; + for (level, option, errno) in [ + (level_socket, bind_device, libc::EPERM), + (level_tcp | HIGH_WORD, nodelay, libc::EINVAL), + (level_tcp, nodelay | HIGH_WORD, libc::EINVAL), + (level_tcp, nodelay | ABOVE_INT_MAX, libc::EINVAL), + ] { + let notification = match receive_probe_notification(&listener) { + Ok(notification) => notification, + Err(error) => { + // A launcher that failed on its own explains the missing + // notification better than the timeout does. + return Err(match launcher.join() { + Ok(Err(launcher_error)) => launcher_error, + _ => error, + }); + } + }; + if i64::from(notification.syscall) != libc::SYS_setsockopt + || notification.args[1] != level + || notification.args[2] != option + { + let _ = listener.respond_errno(notification.id, libc::EIO); + let _ = launcher.join(); + return Err(io::Error::other( + "scalar setsockopt fast path notified an unexpected call", + )); + } + listener.respond_errno(notification.id, errno)?; + } + launcher + .join() + .map_err(|_| io::Error::other("setsockopt launcher panicked"))??; + Ok(()) +} + /// Recover the real error from a probe launcher that exited before handing its /// listener to the broker. /// @@ -633,8 +863,8 @@ fn receive_probe_notification(listener: &NotificationListener) -> io::Result io::Result { - let mut program = build_filter(syscalls)?; +fn install_notification_filter(rules: &[FilterRule]) -> io::Result { + let mut program = build_filter(rules)?; let length = u16::try_from(program.len()) .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "seccomp filter is too large"))?; let mut fprog = libc::sock_fprog { @@ -662,7 +892,7 @@ fn install_notification_filter(syscalls: &[i64]) -> io::Result io::Result> { +fn build_filter(rules: &[FilterRule]) -> io::Result> { let mut program = vec![ stmt(BPF_LD_W_ABS, SECCOMP_DATA_ARCH_OFFSET), jump(BPF_JMP_JEQ_K, native_audit_arch(), 1, 0), @@ -676,10 +906,27 @@ fn build_filter(syscalls: &[i64]) -> io::Result> { stmt(BPF_RET_K, SECCOMP_RET_KILL_PROCESS), ]); - let mut syscalls = syscalls.to_vec(); - syscalls.sort_unstable(); - syscalls.dedup(); - for syscall in syscalls { + // One rule per syscall, in ascending syscall order. A syscall named twice + // is rejected rather than letting the first block silently win. + let mut rules = rules.to_vec(); + rules.sort_unstable_by_key(|rule| rule.syscall()); + if rules + .windows(2) + .any(|pair| pair[0].syscall() == pair[1].syscall()) + { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "seccomp filter names a syscall more than once", + )); + } + for rule in rules { + let syscall = match rule { + FilterRule::Notify(syscall) => syscall, + FilterRule::ScalarSetsockopt => { + append_setsockopt_filter(&mut program)?; + continue; + } + }; if syscall == libc::SYS_sendto { append_sendto_filter(&mut program)?; continue; @@ -728,6 +975,49 @@ fn append_sendto_filter(program: &mut Vec) -> io::Result<()> Ok(()) } +/// Allow exactly the [`SETSOCKOPT_FAST_PATH`] pairs; notify everything else. +/// +/// The kernel reads `level` and `optname` as C `int`s, ignoring the upper +/// argument words. The broker instead refuses any value that does not fit a +/// non-negative `i32` with `EINVAL`, so a nonzero upper word must notify +/// rather than be compared by its low word. Whitelisted values are small +/// positive constants, so an exact low-word match with a zero upper word is +/// exactly a value the broker would accept and continue. +fn append_setsockopt_filter(program: &mut Vec) -> io::Result<()> { + const HIGH_WORD_CHECKS: usize = 6; + const PAIR_LENGTH: usize = 5; + let syscall = u32::try_from(libc::SYS_setsockopt) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "negative setsockopt syscall"))?; + let body = HIGH_WORD_CHECKS + PAIR_LENGTH * SETSOCKOPT_FAST_PATH.len() + 1; + let skip = u8::try_from(body) + .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "setsockopt filter too long"))?; + program.push(jump(BPF_JMP_JEQ_K, syscall, 0, skip)); + for argument in [1, 2] { + program.extend([ + stmt(BPF_LD_W_ABS, argument_word_offset(argument, 1)), + jump(BPF_JMP_JEQ_K, 0, 1, 0), + stmt(BPF_RET_K, SECCOMP_RET_USER_NOTIF), + ]); + } + for (level, option) in SETSOCKOPT_FAST_PATH { + let level = u32::try_from(*level).map_err(|_| { + io::Error::new(io::ErrorKind::InvalidInput, "negative socket option level") + })?; + let option = u32::try_from(*option).map_err(|_| { + io::Error::new(io::ErrorKind::InvalidInput, "negative socket option name") + })?; + program.extend([ + stmt(BPF_LD_W_ABS, argument_word_offset(1, 0)), + jump(BPF_JMP_JEQ_K, level, 0, 3), + stmt(BPF_LD_W_ABS, argument_word_offset(2, 0)), + jump(BPF_JMP_JEQ_K, option, 0, 1), + stmt(BPF_RET_K, SECCOMP_RET_ALLOW), + ]); + } + program.push(stmt(BPF_RET_K, SECCOMP_RET_USER_NOTIF)); + Ok(()) +} + /// Notify a thread-group signal only when it sends `SIGCONT`, which the broker /// refuses while the workload is frozen. Every other signal (for example Go's /// preemption signal) stays in the kernel. @@ -864,9 +1154,230 @@ mod tests { assert_eq!(error.kind(), io::ErrorKind::InvalidInput); } + /// Evaluate a classic-BPF seccomp program over one syscall, supporting + /// exactly the instructions this module emits. + fn evaluate(program: &[libc::sock_filter], syscall: i64, args: [u64; 6]) -> u32 { + let data = SeccompData { + nr: i32::try_from(syscall).unwrap(), + arch: native_audit_arch(), + instruction_pointer: 0, + args, + }; + let word = |offset: u32| -> u32 { + let offset = offset as usize; + if offset == 0 { + return data.nr.cast_unsigned(); + } + if offset == 4 { + return data.arch; + } + let index = (offset - 16) / 8; + let value = data.args[index]; + if (offset - 16).is_multiple_of(8) { + (value & 0xffff_ffff) as u32 + } else { + (value >> 32) as u32 + } + }; + let mut accumulator = 0_u32; + let mut pc = 0_usize; + loop { + let instruction = program[pc]; + pc += 1; + match instruction.code { + BPF_LD_W_ABS => accumulator = word(instruction.k), + BPF_ALU_AND_K => accumulator &= instruction.k, + BPF_RET_K => return instruction.k, + BPF_JMP_JEQ_K | BPF_JMP_JSET_K => { + let taken = if instruction.code == BPF_JMP_JEQ_K { + accumulator == instruction.k + } else { + accumulator & instruction.k != 0 + }; + pc += usize::from(if taken { + instruction.jt + } else { + instruction.jf + }); + } + other => panic!("unsupported BPF opcode {other:#x}"), + } + } + } + + fn arg(value: i32) -> u64 { + u64::from(value.cast_unsigned()) + } + + #[test] + fn workload_filter_allows_only_whitelisted_scalar_setsockopt() { + const HIGH: u64 = 1 << 32; + { + let program = build_filter(&workload_filter_rules()).unwrap(); + let decide = |level: u64, option: u64| { + evaluate( + &program, + libc::SYS_setsockopt, + [3, level, option, 0x7000_0000_1000, 4, 0], + ) + }; + for (level, option) in SETSOCKOPT_FAST_PATH { + let (level, option) = (arg(*level), arg(*option)); + assert_eq!(decide(level, option), SECCOMP_RET_ALLOW); + // The broker refuses values outside a non-negative i32 with + // EINVAL; none of them may take the fast path. + for (level, option) in [ + (level | HIGH, option), + (level, option | HIGH), + (level | 0x8000_0000, option), + (level, option | 0x8000_0000), + (level | (u64::MAX << 32), option), + (option, level), + ] { + if SETSOCKOPT_FAST_PATH + .iter() + .any(|pair| (arg(pair.0), arg(pair.1)) == (level, option)) + { + continue; + } + assert_eq!( + decide(level, option), + SECCOMP_RET_USER_NOTIF, + "{level:#x}/{option:#x}" + ); + } + } + // Every pair outside the whitelist notifies, including each + // denied option and other benign ones. + for (level, option) in [ + (libc::SOL_SOCKET, libc::SO_BINDTODEVICE), + (libc::SOL_SOCKET, libc::SO_BINDTOIFINDEX), + (libc::IPPROTO_IP, libc::IP_UNICAST_IF), + (libc::IPPROTO_IP, libc::IP_MULTICAST_IF), + (libc::IPPROTO_IPV6, libc::IPV6_UNICAST_IF), + (libc::IPPROTO_IPV6, libc::IPV6_MULTICAST_IF), + (libc::IPPROTO_IPV6, libc::IPV6_ADDRFORM), + (libc::IPPROTO_TCP, libc::TCP_FASTOPEN_CONNECT), + (libc::IPPROTO_TCP, libc::TCP_FASTOPEN), + (libc::SOL_SOCKET, libc::SO_RCVBUF), + (libc::IPPROTO_IPV6, libc::IPV6_V6ONLY), + (libc::IPPROTO_IP, libc::TCP_NODELAY), + (libc::IPPROTO_TCP, libc::SO_REUSEADDR), + ] { + assert_eq!( + decide(arg(level), arg(option)), + SECCOMP_RET_USER_NOTIF, + "{level}/{option}" + ); + } + } + } + + #[test] + fn workload_filter_keeps_precedence_around_the_new_fast_paths() { + let unsupported_arch = |program: &[libc::sock_filter]| { + // A foreign architecture is killed before any syscall rule. + let mut pc = 0; + assert_eq!(program[pc].code, BPF_LD_W_ABS); + assert_eq!(program[pc].k, SECCOMP_DATA_ARCH_OFFSET); + pc += 1; + assert_eq!(program[pc].k, native_audit_arch()); + assert_eq!(program[pc + 1].k, SECCOMP_RET_KILL_PROCESS); + }; + { + let program = build_filter(&workload_filter_rules()).unwrap(); + unsupported_arch(&program); + let decide = |syscall: i64, args: [u64; 6]| evaluate(&program, syscall, args); + // Mediated syscalls on either side of setsockopt and openat in + // number order still notify; unrelated syscalls are allowed. + for syscall in [ + libc::SYS_socket, + libc::SYS_connect, + libc::SYS_bind, + libc::SYS_listen, + libc::SYS_sendmsg, + libc::SYS_sendmmsg, + libc::SYS_kill, + libc::SYS_tkill, + libc::SYS_rt_sigqueueinfo, + libc::SYS_openat2, + ] { + assert_eq!(decide(syscall, [0; 6]), SECCOMP_RET_USER_NOTIF, "{syscall}"); + } + for syscall in [libc::SYS_getpid, libc::SYS_read, libc::SYS_getsockopt] { + assert_eq!(decide(syscall, [0; 6]), SECCOMP_RET_ALLOW, "{syscall}"); + } + // The audited sendto and resume-signal paths are unchanged. + assert_eq!( + decide(libc::SYS_sendto, [3, 1, 1, 0, 0, 0]), + SECCOMP_RET_ALLOW + ); + assert_eq!( + decide(libc::SYS_sendto, [3, 1, 1, 0, 0x1000, 16]), + SECCOMP_RET_USER_NOTIF + ); + assert_eq!( + decide(libc::SYS_tgkill, [1, 1, arg(libc::SIGCONT), 0, 0, 0]), + SECCOMP_RET_USER_NOTIF + ); + assert_eq!( + decide(libc::SYS_tgkill, [1, 1, arg(libc::SIGURG), 0, 0, 0]), + SECCOMP_RET_ALLOW + ); + #[cfg(target_arch = "x86_64")] + assert_eq!( + evaluate( + &program, + libc::SYS_setsockopt | i64::from(X32_SYSCALL_BIT), + [3, arg(libc::IPPROTO_TCP), arg(libc::TCP_NODELAY), 0, 4, 0] + ), + SECCOMP_RET_KILL_PROCESS + ); + } + } + + #[test] + fn workload_filter_notifies_every_open() { + let program = build_filter(&workload_filter_rules()).unwrap(); + for flags in [ + libc::O_RDONLY, + libc::O_RDONLY | libc::O_CLOEXEC, + libc::O_RDONLY | libc::O_DIRECTORY, + libc::O_WRONLY, + libc::O_RDWR | libc::O_CREAT | libc::O_TRUNC, + ] { + assert_eq!( + evaluate(&program, libc::SYS_openat, [0, 0x1000, arg(flags), 0, 0, 0]), + SECCOMP_RET_USER_NOTIF, + "{flags:#o}" + ); + #[cfg(target_arch = "x86_64")] + assert_eq!( + evaluate(&program, libc::SYS_open, [0x1000, arg(flags), 0, 0, 0, 0]), + SECCOMP_RET_USER_NOTIF, + "{flags:#o}" + ); + } + assert_eq!( + evaluate(&program, libc::SYS_openat2, [0, 0x1000, 0x2000, 24, 0, 0]), + SECCOMP_RET_USER_NOTIF + ); + } + + #[test] + fn filter_rejects_a_syscall_named_twice() { + let error = build_filter(&[ + FilterRule::Notify(libc::SYS_setsockopt), + FilterRule::ScalarSetsockopt, + ]) + .expect_err("duplicate syscall must fail"); + assert_eq!(error.kind(), io::ErrorKind::InvalidInput); + } + #[test] fn active_notification_probe_passes() { let report = probe_notification_api().expect("active notification probe"); + assert!(report.scalar_setsockopt_fast_path()); assert!(report.notification_round_trip()); assert!(report.addfd_send()); assert!(report.connected_send_fast_path()); diff --git a/crates/openshell-sandbox/src/linux/task_memory.rs b/crates/openshell-sandbox/src/linux/task_memory.rs index df2afa9925..63c8f0fc9e 100644 --- a/crates/openshell-sandbox/src/linux/task_memory.rs +++ b/crates/openshell-sandbox/src/linux/task_memory.rs @@ -55,8 +55,14 @@ pub fn read_exact(tid: u32, address: u64, destination: &mut [u8]) -> io::Result< ) }); match copied { - Ok(copied) => require_exact(copied, destination.len(), "task-memory read"), + Ok(copied) => { + #[cfg(feature = "perf-harness")] + PROCESS_VM_READS.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + require_exact(copied, destination.len(), "task-memory read") + } Err(error) if syscall_profile_denied(&error) => { + #[cfg(feature = "perf-harness")] + PROC_MEM_FALLBACKS.fetch_add(1, std::sync::atomic::Ordering::Relaxed); read_exact_from_proc_mem(tid, address, destination) .map_err(|fallback| fallback_error("read", &error, fallback)) } @@ -64,6 +70,24 @@ pub fn read_exact(tid: u32, address: u64, destination: &mut [u8]) -> io::Result< } } +#[cfg(feature = "perf-harness")] +static PROCESS_VM_READS: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); +#[cfg(feature = "perf-harness")] +static PROC_MEM_FALLBACKS: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + +/// Task-memory reads by backend: `process_vm_readv`, then `/proc//mem`. +/// +/// Counted only in the opt-in performance harness, to establish which +/// backend a container posture uses. +#[cfg(feature = "perf-harness")] +#[must_use] +pub fn read_backend_counts() -> (u64, u64) { + ( + PROCESS_VM_READS.load(std::sync::atomic::Ordering::Relaxed), + PROC_MEM_FALLBACKS.load(std::sync::atomic::Ordering::Relaxed), + ) +} + fn syscall_profile_denied(error: &io::Error) -> bool { matches!( error.raw_os_error(), diff --git a/crates/openshell-sandbox/src/network_broker.rs b/crates/openshell-sandbox/src/network_broker.rs index 323f1ac715..1947dabcbe 100644 --- a/crates/openshell-sandbox/src/network_broker.rs +++ b/crates/openshell-sandbox/src/network_broker.rs @@ -5,14 +5,14 @@ #![allow(unsafe_code)] -use std::collections::HashMap; +use std::collections::{HashMap, VecDeque}; use std::io; use std::mem::size_of; use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr, TcpListener, TcpStream, UdpSocket}; use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd, RawFd}; use std::sync::atomic::AtomicUsize; use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::{Arc, Mutex}; +use std::sync::{Arc, Condvar, Mutex}; use std::time::{Duration, Instant}; use crate::accept_errors::{ @@ -43,6 +43,12 @@ const DNS_RELAY_ADDRESS: SocketAddr = SocketAddr::V4(std::net::SocketAddrV4::new )); const RELAY_CONNECT_TIMEOUT: Duration = Duration::from_secs(5); const NETWORK_DECISION_TIMEOUT: Duration = Duration::from_secs(30); +/// How long a blocking-connect worker stays parked for reuse before exiting. +const CONNECT_WORKER_IDLE_TIMEOUT: Duration = Duration::from_secs(30); +/// Parked blocking-connect workers kept for reuse. A worker that finishes +/// while this many are already parked exits instead, so a burst of blocking +/// connects does not leave a thread per connect behind. +const CONNECT_WORKER_IDLE_LIMIT: usize = 4; fn retry_notification_receive(error: &io::Error) -> bool { error.kind() == io::ErrorKind::Interrupted || error.raw_os_error() == Some(libc::ENOENT) @@ -215,6 +221,7 @@ struct NotificationQueues { pending: mpsc::Sender, dns_relay: DnsRelay, active_opens: Arc, + connect_workers: Arc, descriptor_soft_limit: usize, decision_timeout: Duration, } @@ -230,14 +237,24 @@ pub struct NetworkBroker { descriptor_soft_limit: usize, dns_address: SocketAddr, healthy: Arc, + #[cfg(test)] + active_opens: Arc, + #[cfg(test)] + connect_workers: Arc, } impl NetworkBroker { + /// Start the broker for `listener`. pub(crate) fn start( listener: NotificationListener, protected_control_port: Option, ) -> io::Result { - Self::start_with_dns_address(listener, DNS_RELAY_ADDRESS, protected_control_port) + Self::start_configured( + listener, + DNS_RELAY_ADDRESS, + protected_control_port, + NETWORK_DECISION_TIMEOUT, + ) } #[cfg(any(test, feature = "perf-harness"))] @@ -249,6 +266,7 @@ impl NetworkBroker { ) } + #[cfg(any(test, feature = "perf-harness"))] fn start_with_dns_address( listener: NotificationListener, dns_address: SocketAddr, @@ -262,12 +280,28 @@ impl NetworkBroker { ) } + #[cfg(any(test, feature = "perf-harness"))] fn start_with_decision_timeout( listener: NotificationListener, dns_address: SocketAddr, protected_control_port: Option, decision_timeout: Duration, ) -> io::Result { + Self::start_configured( + listener, + dns_address, + protected_control_port, + decision_timeout, + ) + } + + fn start_configured( + listener: NotificationListener, + dns_address: SocketAddr, + protected_control_port: Option, + decision_timeout: Duration, + ) -> io::Result { + let provider_files = crate::provider_files::ProviderFiles::default(); let listener = Arc::new(listener); let (pending_tx, pending_rx) = mpsc::channel(OPEN_QUEUE_CAPACITY); let (pending_dns_tx, pending_dns_rx) = mpsc::channel(DNS_QUEUE_CAPACITY); @@ -276,7 +310,6 @@ impl NetworkBroker { let dns_address = dns_relay.address; let descriptor_soft_limit = descriptor_soft_limit()?; let registry = Arc::new(Mutex::new(SocketRegistry::new(1, SOCKET_CAPACITY)?)); - let provider_files = crate::provider_files::ProviderFiles::default(); let workload_frozen = Arc::new(AtomicBool::new(false)); let queues = NotificationQueues { provider_files: provider_files.clone(), @@ -286,9 +319,19 @@ impl NetworkBroker { pending: pending_tx, dns_relay, active_opens, + connect_workers: ConnectWorkers::new(ConnectWorkerLimits { + max_workers: OPEN_QUEUE_CAPACITY, + max_idle: CONNECT_WORKER_IDLE_LIMIT, + idle_timeout: CONNECT_WORKER_IDLE_TIMEOUT, + }), descriptor_soft_limit, decision_timeout, }; + #[cfg(test)] + let (test_active_opens, test_connect_workers) = ( + Arc::clone(&queues.active_opens), + Arc::clone(&queues.connect_workers), + ); let healthy = Arc::new(AtomicBool::new(true)); let broker_healthy = healthy.clone(); let broker_registry = Arc::clone(®istry); @@ -316,10 +359,10 @@ impl NetworkBroker { // broker keeps mediating the rest. let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { dispatch_notification( - Arc::clone(&broker_registry), - Arc::clone(&listener), + &broker_registry, + &listener, notification, - queues.clone(), + &queues, ) })); match outcome { @@ -362,6 +405,10 @@ impl NetworkBroker { descriptor_soft_limit, dns_address, healthy, + #[cfg(test)] + active_opens: test_active_opens, + #[cfg(test)] + connect_workers: test_connect_workers, }) } @@ -687,22 +734,22 @@ fn serve_dns_tcp(mut stream: TcpStream, pending: mpsc::Sender) } fn dispatch_notification( - registry: Arc>, - listener: Arc, + registry: &Arc>, + listener: &Arc, notification: Notification, - queues: NotificationQueues, + queues: &NotificationQueues, ) -> io::Result<()> { let syscall = i64::from(notification.syscall); if syscall == libc::SYS_openat || syscall == libc::SYS_openat2 { - return queues.provider_files.handle_open(&listener, notification); + return queues.provider_files.handle_open(listener, notification); } #[cfg(target_arch = "x86_64")] if syscall == libc::SYS_open { - return queues.provider_files.handle_open(&listener, notification); + return queues.provider_files.handle_open(listener, notification); } if matches!(syscall, libc::SYS_kill | libc::SYS_rt_sigqueueinfo) { return crate::linux::process_signal::mediate_process_signal( - &listener, + listener, notification, std::process::id(), &queues.workload_frozen, @@ -713,7 +760,7 @@ fn dispatch_notification( libc::SYS_tkill | libc::SYS_tgkill | libc::SYS_rt_tgsigqueueinfo ) { return crate::linux::process_signal::mediate_thread_signal( - &listener, + listener, notification, std::process::id(), &queues.workload_frozen, @@ -721,8 +768,8 @@ fn dispatch_notification( } if syscall == libc::SYS_socket { return create_socket( - ®istry, - &listener, + registry, + listener, notification, queues.descriptor_soft_limit, ); @@ -731,16 +778,16 @@ fn dispatch_notification( return connect_socket(registry, listener, notification, queues); } if syscall == libc::SYS_bind { - return bind_socket(®istry, &listener, notification); + return bind_socket(registry, listener, notification); } if syscall == libc::SYS_listen { - return listen_socket(®istry, &listener, notification); + return listen_socket(registry, listener, notification); } if matches!( syscall, libc::SYS_sendto | libc::SYS_sendmsg | libc::SYS_sendmmsg ) { - return classify_send(®istry, &listener, notification, &queues.dns_relay); + return classify_send(registry, listener, notification, &queues.dns_relay); } if syscall == libc::SYS_setsockopt { let level = i32::try_from(notification.args[1]) @@ -868,7 +915,7 @@ fn descriptor_soft_limit() -> io::Result { } fn open_descriptor_count() -> io::Result { - std::fs::read_dir("/proc/self/fd").map(Iterator::count) + crate::linux::proc_fd::open_descriptor_count() } fn descriptor_headroom_exhausted( @@ -896,11 +943,16 @@ fn prepare_registry_for_socket_with_count( ) -> io::Result<()> { let required = SOCKET_FD_HEADROOM.saturating_add(1); let mut registry = lock(registry); - if registry.is_full() - || descriptor_headroom_exhausted(descriptor_soft_limit, open_descriptors()?, required) + // One count decides the ordinary path. Descriptor use is recounted only + // after a collection, which is the one step here that changes it; a + // second count without one would merely resample concurrent ambient use + // a few microseconds later, which the next socket request does anyway. + if !registry.is_full() + && !descriptor_headroom_exhausted(descriptor_soft_limit, open_descriptors()?, required) { - collect_closed_socket_entries_locked(&mut registry)?; + return Ok(()); } + collect_closed_socket_entries_locked(&mut registry)?; if registry.is_full() || descriptor_headroom_exhausted(descriptor_soft_limit, open_descriptors()?, required) { @@ -929,9 +981,11 @@ fn ensure_descriptor_headroom_with_count( mut open_descriptors: impl FnMut() -> io::Result, ) -> io::Result<()> { let mut registry = lock(registry); - if descriptor_headroom_exhausted(descriptor_soft_limit, open_descriptors()?, required) { - collect_closed_socket_entries_locked(&mut registry)?; + if !descriptor_headroom_exhausted(descriptor_soft_limit, open_descriptors()?, required) { + return Ok(()); } + // Recount only after the collection that may have released descriptors. + collect_closed_socket_entries_locked(&mut registry)?; if descriptor_headroom_exhausted(descriptor_soft_limit, open_descriptors()?, required) { return Err(io::Error::from_raw_os_error(libc::EMFILE)); } @@ -952,27 +1006,29 @@ fn reject_protected_control_destination( } fn connect_socket( - registry: Arc>, - listener: Arc, + registry: &Arc>, + listener: &Arc, notification: Notification, - queues: NotificationQueues, + queues: &NotificationQueues, ) -> io::Result<()> { let NotificationQueues { pending, dns_relay, active_opens, + connect_workers, identity_resolver, decision_timeout, protected_control_port, .. } = queues; + let decision_timeout = *decision_timeout; let notification_started = Instant::now(); let fd = raw_fd(notification.args[0])?; - let address_family = - read_socket_family(notification.tid, notification.args[1], notification.args[2])?; + let (address_family, copied_destination) = + read_socket_family_and_addr(notification.tid, notification.args[1], notification.args[2])?; if !matches!(address_family, libc::AF_INET | libc::AF_INET6) { if address_family == libc::AF_UNSPEC { - let mut registry = lock(®istry); + let mut registry = lock(registry); if let Ok(entry) = registry.resolve_mut(notification.tid, fd) && entry.metadata().kind == InetKind::DnsUdp && matches!( @@ -987,7 +1043,7 @@ fn connect_socket( return listener.respond_value(notification.id, 0); } } - if lock(®istry) + if lock(registry) .resolve_for_native_fallback(notification.tid, fd)? .is_some() { @@ -998,17 +1054,35 @@ fn connect_socket( // Native non-INET descriptors remain kernel-driven. return listener.respond_continue(notification.id); } - let destination = - read_socket_addr(notification.tid, notification.args[1], notification.args[2])?; - reject_protected_control_destination(destination, protected_control_port)?; - let (kind, socket_identity, nonblocking, repeated) = { - let registry = lock(®istry); + let destination = copied_destination?; + reject_protected_control_destination(destination, *protected_control_port)?; + // The metadata service lives in the supervisor, even though SDKs address + // it through loopback, and the DNS relay has its own admission path. + let ordinary_loopback = destination.ip().is_loopback() + && !openshell_core::google_cloud::is_metadata_destination(destination) + && destination != dns_relay.address; + let (kind, socket_identity, nonblocking, repeated, local_connector) = { + let registry = lock(registry); let entry = registry.resolve(notification.tid, fd)?; + let kind = entry.metadata().kind; + let repeated = repeated_connect_outcome(entry.state(), kind, destination); + // A loopback TCP connect waits on a duplicate of the retained socket. + // Take it in the scope that captured the socket identity, so the two + // name the same socket and the descriptor is resolved once. + let local_connector = if ordinary_loopback && kind == InetKind::Tcp && repeated.is_none() { + Some(rustix::io::fcntl_dupfd_cloexec( + entry.retained_preconnect()?, + 3, + )?) + } else { + None + }; ( - entry.metadata().kind, + kind, entry.identity(), entry.metadata().nonblocking, - repeated_connect_outcome(entry.state(), entry.metadata().kind, destination), + repeated, + local_connector, ) }; match repeated { @@ -1017,7 +1091,7 @@ fn connect_socket( None => {} } if kind == InetKind::DnsUdp && destination.port() == 0 { - let mut registry = lock(®istry); + let mut registry = lock(registry); let entry = registry.resolve_mut(notification.tid, fd)?; if !matches!( entry.state(), @@ -1046,7 +1120,7 @@ fn connect_socket( return listener.respond_value(notification.id, 0); } if destination == dns_relay.address { - let mut registry = lock(®istry); + let mut registry = lock(registry); let entry = registry.resolve_mut(notification.tid, fd)?; if !matches!( entry.state(), @@ -1079,18 +1153,25 @@ fn connect_socket( && !openshell_core::google_cloud::is_metadata_destination(destination) { if kind == InetKind::Tcp { + let connector = local_connector.ok_or_else(|| { + io::Error::other("loopback TCP connect has no retained socket duplicate") + })?; return connect_local_tcp( - ®istry, - &listener, + registry, + listener, notification, - fd, - socket_identity, - destination, - &active_opens, + LocalConnectTarget { + fd, + socket_identity, + destination, + }, + connector, + active_opens, + connect_workers, ); } // A UDP connect completes immediately. - let mut registry = lock(®istry); + let mut registry = lock(registry); let entry = registry.resolve_mut(notification.tid, fd)?; listener.validate_id(notification.id)?; connect_exact(entry.retained_preconnect()?.as_raw_fd(), destination)?; @@ -1105,7 +1186,7 @@ fn connect_socket( let identity = identity_resolver.resolve(notification.tid); let (decision_tx, decision_rx) = std::sync::mpsc::sync_channel(1); let (relay_tx, relay_rx) = oneshot::channel(); - let slot = acquire_pending_open_slot(&active_opens)?; + let slot = acquire_pending_open_slot(active_opens)?; pending .try_send(PendingTcpOpen { destination, @@ -1126,7 +1207,8 @@ fn connect_socket( io::Error::new(io::ErrorKind::BrokenPipe, "network-open queue closed") } })?; - let worker_listener = Arc::clone(&listener); + let worker_listener = Arc::clone(listener); + let registry = Arc::clone(registry); std::thread::Builder::new() .name("openshell-network-open".to_string()) .spawn(move || { @@ -1192,16 +1274,16 @@ fn connect_local_tcp( registry: &Arc>, listener: &Arc, notification: Notification, - fd: RawFd, - socket_identity: SocketIdentity, - destination: SocketAddr, + target: LocalConnectTarget, + connector: OwnedFd, active_opens: &Arc, + connect_workers: &Arc, ) -> io::Result<()> { - let connector = { - let registry = lock(registry); - let entry = registry.resolve(notification.tid, fd)?; - rustix::io::fcntl_dupfd_cloexec(entry.retained_preconnect()?, 3)? - }; + let LocalConnectTarget { + fd, + socket_identity, + destination, + } = target; listener.validate_id(notification.id)?; // SAFETY: F_GETFL reads the flags of the live shared open file. let flags = unsafe { libc::fcntl(connector.as_raw_fd(), libc::F_GETFL) }; @@ -1229,42 +1311,288 @@ fn connect_local_tcp( Err(error) => Err(error), }; } + // The admission bound precedes the handoff: at most OPEN_QUEUE_CAPACITY + // blocking connects and external opens wait at once. The pool holds the + // slot until its worker has parked or retired, and separately caps its + // live and parked threads. let slot = acquire_pending_open_slot(active_opens)?; - let registry = Arc::clone(registry); - let worker_listener = Arc::clone(listener); - std::thread::Builder::new() - .name("openshell-local-connect".to_string()) - .spawn(move || { - let _slot = slot; - // The duplicate shares the workload's blocking open file; wait - // natively without changing its flags. - let result = with_sockaddr(destination, |pointer, length| { - // SAFETY: pointer/length describe a live sockaddr and the - // connector is a live duplicate of the workload's socket. - if unsafe { libc::connect(connector.as_raw_fd(), pointer, length) } == 0 { - Ok(()) - } else { - Err(io::Error::last_os_error()) - } - }); - if result.is_ok() { - commit_local_connect( - ®istry, - notification.tid, - fd, - socket_identity, - destination, - ); + connect_workers.submit( + LocalConnectJob { + connector, + registry: Arc::clone(registry), + listener: Arc::clone(listener), + notification, + target, + }, + slot, + ) +} + +/// The workload socket a loopback connect acts on. +#[derive(Clone, Copy)] +struct LocalConnectTarget { + fd: RawFd, + socket_identity: SocketIdentity, + destination: SocketAddr, +} + +/// One admitted blocking loopback connect, owned by the worker that waits. +struct LocalConnectJob { + connector: OwnedFd, + registry: Arc>, + listener: Arc, + notification: Notification, + target: LocalConnectTarget, +} + +impl PoolJob for LocalConnectJob { + /// Run the job, answering the blocked syscall even if the job panics, so + /// a faulty connect cannot take a reusable worker down with it. + fn run_contained(self) { + let listener = Arc::clone(&self.listener); + let id = self.notification.id; + if std::panic::catch_unwind(std::panic::AssertUnwindSafe(move || self.run())).is_err() { + tracing::error!("sandbox local-connect worker job panicked"); + let _ = listener.respond_errno(id, libc::EIO); + } + } +} + +impl LocalConnectJob { + fn run(self) { + let LocalConnectTarget { + fd, + socket_identity, + destination, + } = self.target; + // The duplicate shares the workload's blocking open file; wait + // natively without changing its flags. + let result = with_sockaddr(destination, |pointer, length| { + // SAFETY: pointer/length describe a live sockaddr and the + // connector is a live duplicate of the workload's socket. + if unsafe { libc::connect(self.connector.as_raw_fd(), pointer, length) } == 0 { + Ok(()) + } else { + Err(io::Error::last_os_error()) + } + }); + if result.is_ok() { + commit_local_connect( + &self.registry, + self.notification.tid, + fd, + socket_identity, + destination, + ); + } + let _ = match result { + Ok(()) => self.listener.respond_value(self.notification.id, 0), + Err(error) => self + .listener + .respond_errno(self.notification.id, error_to_errno(&error)), + }; + } +} + +/// Work a [`ConnectWorkers`] thread runs to completion. +trait PoolJob: Send + 'static { + /// Run the job. Implementations contain their own panics and always + /// answer whatever waits on the job. + fn run_contained(self); +} + +/// Thread bounds of a [`ConnectWorkers`] pool. +#[derive(Clone, Copy, Debug)] +struct ConnectWorkerLimits { + /// Live worker threads, busy or parked. A submission that would exceed + /// it is refused. + max_workers: usize, + /// Parked workers kept for reuse; a worker finishing beyond it exits. + max_idle: usize, + /// How long a parked worker waits for a job before exiting. + idle_timeout: Duration, +} + +/// Reusable threads that wait in blocking loopback connects. +/// +/// A blocking connect must wait off the dispatcher, and a slow or full +/// listener can hold its waiter indefinitely, so a job is never queued behind +/// a busy worker: it goes to a parked worker reserved for it, and otherwise +/// starts a new thread. Submission never blocks. +/// +/// Three bounds hold together: +/// +/// - Each job arrives with an admission permit (a [`PendingOpenSlot`]). The +/// worker releases it only after the pool state shows that worker parked or +/// retired, so every busy worker and every queued job holds a permit and +/// busy workers never outnumber admitted opens. +/// - `max_workers` caps live threads, busy and parked. It is checked under +/// the state lock on every spawn and does not rely on the permit quota. +/// - `max_idle` caps parked threads. A worker that finishes while that many +/// are parked exits at once; parked workers exit after `idle_timeout`. +/// +/// With the external-open workers that share the permit quota, the broker +/// therefore runs at most `OPEN_QUEUE_CAPACITY + max_idle` such threads. +struct ConnectWorkers { + state: Mutex>, + wake: Condvar, + limits: ConnectWorkerLimits, + /// Threads started over the pool's lifetime. + spawned: AtomicUsize, +} + +struct ConnectWorkerState { + /// Jobs handed to parked workers. Each entry reserved one parked worker + /// when it was queued, so `queue.len()` never exceeds the parked count. + queue: VecDeque<(Job, Permit)>, + /// Parked workers not reserved for a queued job. + idle: usize, + /// Live worker threads: busy, parked, or between the two. + workers: usize, +} + +/// Accounts for a worker thread lost to a panic that escaped its job. A +/// worker that retires in order removes itself under the state lock, in the +/// same step that stops counting it as parked, and disarms this guard. +struct WorkerExit<'a, Job, Permit>(&'a ConnectWorkers); + +impl WorkerExit<'_, Job, Permit> { + /// Retire the calling worker in the critical section `state` guards. + fn retire(self, state: &mut ConnectWorkerState) { + state.workers -= 1; + std::mem::forget(self); + } +} + +impl Drop for WorkerExit<'_, Job, Permit> { + fn drop(&mut self) { + lock(&self.0.state).workers -= 1; + } +} + +impl ConnectWorkers { + fn new(limits: ConnectWorkerLimits) -> Arc { + Arc::new(Self { + state: Mutex::new(ConnectWorkerState { + queue: VecDeque::new(), + idle: 0, + workers: 0, + }), + wake: Condvar::new(), + limits, + spawned: AtomicUsize::new(0), + }) + } + + /// Hand `job` to a parked worker, or start a worker for it, without + /// blocking. + /// + /// On failure the job and its permit are dropped, which releases the + /// admission slot and connector; the caller reports the error to the + /// blocked syscall. A pool at `max_workers` with no parked worker refuses + /// with `EAGAIN`, the result an exhausted admission quota gives. + fn submit(self: &Arc, job: Job, permit: Permit) -> io::Result<()> { + { + let mut state = lock(&self.state); + if state.idle > 0 { + // Reserve one parked worker for this job so a concurrent + // submission cannot count on the same one. + state.idle -= 1; + state.queue.push_back((job, permit)); + drop(state); + self.wake.notify_one(); + return Ok(()); + } + if state.workers >= self.limits.max_workers { + return Err(io::Error::from_raw_os_error(libc::EAGAIN)); + } + state.workers += 1; + } + let pool = Arc::clone(self); + let started = std::thread::Builder::new() + .name("openshell-local-connect".to_string()) + .spawn(move || pool.work(job, permit)); + match started { + Ok(_) => { + self.spawned.fetch_add(1, Ordering::Relaxed); + Ok(()) + } + Err(error) => { + lock(&self.state).workers -= 1; + Err(io::Error::other(format!( + "start local-connect worker: {error}" + ))) + } + } + } + + fn work(self: Arc, first: Job, permit: Permit) { + // Declared before the exit guard, so on every way out of this + // function, an escaped job panic included, the live-worker count + // drops before the permit is released. A submission admitted by the + // freed permit therefore never sees this thread still counted, and + // the live-worker cap cannot refuse it spuriously. + let mut held = Some(permit); + let exit = WorkerExit(&*self); + let mut job = first; + loop { + job.run_contained(); + let mut state = lock(&self.state); + if let Some((next, next_permit)) = state.queue.pop_front() { + // Take the queued job directly. The parked worker reserved + // for it stays parked and becomes available again. + state.idle += 1; + drop(state); + job = next; + // Releases the finished job's permit. + drop(held.replace(next_permit)); + continue; } - let _ = match result { - Ok(()) => worker_listener.respond_value(notification.id, 0), - Err(error) => { - worker_listener.respond_errno(notification.id, error_to_errno(&error)) + if state.idle >= self.limits.max_idle { + // Enough workers are parked. The live count drops here, + // under the lock; the permit is released on return. + exit.retire(&mut state); + return; + } + state.idle += 1; + // This worker is now recorded as parked; release the finished + // job's permit. + drop(held.take()); + let deadline = Instant::now() + self.limits.idle_timeout; + job = loop { + // A queued job always has a reserved parked worker, so take + // it even at the deadline. Its reservation already removed + // one worker from `idle`. + if let Some((next, next_permit)) = state.queue.pop_front() { + drop(held.replace(next_permit)); + break next; + } + let now = Instant::now(); + if now >= deadline { + // The queue is empty, so this parked worker is one of + // the unreserved ones `idle` counts. + state.idle -= 1; + exit.retire(&mut state); + return; } + state = self + .wake + .wait_timeout(state, deadline.saturating_duration_since(now)) + .unwrap_or_else(std::sync::PoisonError::into_inner) + .0; }; - }) - .map_err(|error| io::Error::other(format!("start local-connect worker: {error}")))?; - Ok(()) + } + } + + #[cfg(test)] + fn spawned(&self) -> usize { + self.spawned.load(Ordering::Relaxed) + } + + #[cfg(test)] + fn workers(&self) -> usize { + lock(&self.state).workers + } } /// Record a completed or in-progress loopback TCP connect, unless the @@ -1406,7 +1734,9 @@ fn bind_socket( notification: Notification, ) -> io::Result<()> { let fd = raw_fd(notification.args[0])?; - if !socket_address_is_inet(notification.tid, notification.args[1], notification.args[2])? { + let (address_family, copied_local) = + read_socket_family_and_addr(notification.tid, notification.args[1], notification.args[2])?; + if !matches!(address_family, libc::AF_INET | libc::AF_INET6) { if lock(registry) .resolve_for_native_fallback(notification.tid, fd)? .is_some() @@ -1415,7 +1745,7 @@ fn bind_socket( } return listener.respond_continue(notification.id); } - let local = read_socket_addr(notification.tid, notification.args[1], notification.args[2])?; + let local = copied_local?; if !local.ip().is_loopback() && !local.ip().is_unspecified() { return Err(io::Error::from_raw_os_error(libc::EACCES)); } @@ -1797,11 +2127,43 @@ fn read_socket_addr(tid: u32, address: u64, length: u64) -> io::Result io::Result { - Ok(matches!( - read_socket_family(tid, address, length)?, - libc::AF_INET | libc::AF_INET6 - )) +/// Copy a workload socket address once and return its family together with +/// the decoded INET address. +/// +/// `connect` and `bind` first need the family, to leave non-INET addresses to +/// the kernel, and then the address. Reading both from one copy halves the +/// task-memory reads and decides both from the same bytes. The outer result +/// is exactly [`read_socket_family`]'s and the inner one exactly +/// [`read_socket_addr`]'s: a length the address read refuses, or a copy of the +/// full length that fails (for example a tail in an unmapped page), falls +/// back to those two reads so every error is the one they report. +fn read_socket_family_and_addr( + tid: u32, + address: u64, + length: u64, +) -> io::Result<(i32, io::Result)> { + let Ok(byte_length) = usize::try_from(length) else { + return Err(io::Error::from_raw_os_error(libc::EINVAL)); + }; + if address == 0 || byte_length < size_of::() { + return Err(io::Error::from_raw_os_error(libc::EFAULT)); + } + if byte_length <= size_of::() { + let mut storage = std::mem::MaybeUninit::::zeroed(); + // SAFETY: the zeroed storage is valid for byte access and + // `byte_length` was bounded by its size. + let bytes = unsafe { + std::slice::from_raw_parts_mut(storage.as_mut_ptr().cast::(), byte_length) + }; + if task_memory::read_exact(tid, address, bytes).is_ok() { + // SAFETY: every byte of the zeroed storage is initialized. + let storage = unsafe { storage.assume_init() }; + let family = i32::from(storage.ss_family); + return Ok((family, decode_sockaddr(storage, byte_length))); + } + } + let family = read_socket_family(tid, address, length)?; + Ok((family, read_socket_addr(tid, address, length))) } fn read_socket_family(tid: u32, address: u64, length: u64) -> io::Result { @@ -2135,6 +2497,295 @@ mod tests { let detached = launcher.execute(move || std::fs::read(path)).unwrap(); assert_eq!(detached.unwrap_err().raw_os_error(), Some(libc::ENOENT)); } + + /// `openat2` with the given `open_how` fields. + fn raw_openat2( + path: &std::ffi::CStr, + flags: i32, + mode: u64, + resolve: u64, + size: usize, + ) -> io::Result { + let how = [u64::try_from(flags).expect("open flags"), mode, resolve]; + // SAFETY: path is NUL-terminated and how spans at least size bytes. + let fd = unsafe { + libc::syscall( + libc::SYS_openat2, + libc::AT_FDCWD, + path.as_ptr(), + how.as_ptr(), + size, + ) + }; + if fd < 0 { + return Err(io::Error::last_os_error()); + } + // SAFETY: a successful openat2 returns one newly owned descriptor. + Ok(unsafe { std::fs::File::from_raw_fd(i32::try_from(fd).expect("open fd fits")) }) + } + + fn raw_openat(path: *const libc::c_char, flags: i32) -> io::Result { + // SAFETY: the caller passes a NUL-terminated path. + let fd = unsafe { libc::openat(libc::AT_FDCWD, path, flags, 0o600) }; + if fd < 0 { + return Err(io::Error::last_os_error()); + } + // SAFETY: a successful openat returns one newly owned descriptor. + Ok(unsafe { std::fs::File::from_raw_fd(fd) }) + } + + fn errno_of(result: io::Result) -> Option { + result.err().and_then(|error| error.raw_os_error()) + } + + /// A copy of `path` that ends at the last byte before an unmapped page, + /// so reading past its terminator faults. + /// + /// The following page is unmapped rather than `PROT_NONE`: a forced read + /// through `/proc//mem` succeeds on an inaccessible private mapping, + /// and only fails where nothing is mapped. + struct PathBeforeUnmappedPage { + /// The one page that stays mapped and holds the path. + mapping: *mut libc::c_void, + page_size: usize, + path: *const libc::c_char, + } + + impl PathBeforeUnmappedPage { + fn new(path: &std::ffi::CStr) -> Self { + let bytes = path.to_bytes_with_nul(); + // The base page size is a property of the running kernel (4K, + // 16K, or 64K on ARM64), not a constant. + // SAFETY: sysconf has no preconditions. + let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) }; + let page_size = usize::try_from(page_size).expect("base page size"); + assert!(page_size.is_power_of_two()); + assert!(bytes.len() <= page_size, "path must fit in one page"); + // SAFETY: a fresh private mapping of two base pages, which mmap + // returns page aligned. The second page is unmapped again and the + // copy stays inside the first. + unsafe { + let mapping = libc::mmap( + std::ptr::null_mut(), + 2 * page_size, + libc::PROT_READ | libc::PROT_WRITE, + libc::MAP_PRIVATE | libc::MAP_ANONYMOUS, + -1, + 0, + ); + assert_ne!(mapping, libc::MAP_FAILED); + // Nothing keeps the hole reserved: an unrelated mapping made + // later by this process may land in it, which would only make + // the read past the terminator succeed instead of fault. + let hole = mapping.cast::().add(page_size); + assert_eq!(libc::munmap(hole.cast(), page_size), 0); + let start = hole.sub(bytes.len()); + std::ptr::copy_nonoverlapping(bytes.as_ptr(), start, bytes.len()); + Self { + mapping, + page_size, + path: start.cast(), + } + } + } + } + + impl Drop for PathBeforeUnmappedPage { + fn drop(&mut self) { + // SAFETY: only the page still owned from `new`; the hole after it + // was released there and may belong to another mapping by now. + unsafe { libc::munmap(self.mapping, self.page_size) }; + } + } + + #[test] + fn injected_provider_descriptors_and_open_classification_keep_syscall_results() { + use std::os::unix::fs::{FileExt as _, MetadataExt as _}; + + let (launcher, listener) = + crate::linux::workload_launcher::start().expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start broker"); + let provider = "/run/openshell/providers/acme/client.toml"; + broker + .provider_files() + .replace(HashMap::from([(provider.to_string(), "0123456789".into())])) + .unwrap(); + let scratch = tempfile::tempdir().unwrap(); + let ordinary = scratch.path().join("ordinary"); + std::fs::write(&ordinary, "ordinary").unwrap(); + let created = scratch.path().join("created"); + + let outcome = launcher + .execute(move || -> Result<(), String> { + let check = |condition: bool, what: &str| { + if condition { + Ok(()) + } else { + Err(format!("{what}: {}", io::Error::last_os_error())) + } + }; + let provider_path = std::ffi::CString::new(provider).unwrap(); + let missing = c"/run/openshell/providers/acme/missing"; + let read_only = libc::O_RDONLY | libc::O_CLOEXEC; + + // Two injected descriptors: one file, two file offsets. + let mut first = std::fs::File::open(provider).map_err(|e| e.to_string())?; + let mut second = raw_openat2(&provider_path, read_only, 0, 0, 24) + .map_err(|e| format!("openat2 provider file: {e}"))?; + let mut head = [0_u8; 4]; + first.read_exact(&mut head).map_err(|e| e.to_string())?; + let mut whole = String::new(); + second + .read_to_string(&mut whole) + .map_err(|e| e.to_string())?; + let mut rest = String::new(); + first.read_to_string(&mut rest).map_err(|e| e.to_string())?; + check( + &head == b"0123" && whole == "0123456789" && rest == "456789", + "injected descriptors share a file offset", + )?; + let identity = |file: &std::fs::File| { + let metadata = file.metadata().expect("injected descriptor metadata"); + (metadata.dev(), metadata.ino(), metadata.mode() & 0o7777) + }; + check( + identity(&first) == identity(&second) && identity(&first).2 == 0o600, + "opens of one revision are one private file", + )?; + check( + first.write_at(b"changed", 0).is_err() && first.set_len(0).is_err(), + "injected descriptor is writable", + )?; + + // Close-on-exec follows the request, for both entry points. + let close_on_exec = |file: &std::fs::File| { + // SAFETY: F_GETFD on a live descriptor. + let flags = unsafe { libc::fcntl(file.as_raw_fd(), libc::F_GETFD) }; + flags & libc::FD_CLOEXEC != 0 + }; + let inherited = raw_openat(provider_path.as_ptr(), libc::O_RDONLY) + .map_err(|e| format!("openat provider file: {e}"))?; + check( + close_on_exec(&second) && !close_on_exec(&inherited), + "close-on-exec does not follow the open flags", + )?; + let inherited = raw_openat2(&provider_path, libc::O_RDONLY, 0, 0, 24) + .map_err(|e| format!("openat2 provider file: {e}"))?; + check(!close_on_exec(&inherited), "openat2 forced close-on-exec")?; + + // Managed flags keep their errno through openat and openat2. + for (flags, errno) in [ + (libc::O_WRONLY, libc::EACCES), + (libc::O_RDWR, libc::EACCES), + (libc::O_RDONLY | libc::O_CREAT, libc::EACCES), + (libc::O_RDONLY | libc::O_TRUNC, libc::EACCES), + (libc::O_RDONLY | libc::O_APPEND, libc::EACCES), + // O_TMPFILE includes the O_DIRECTORY bit, so a directory + // open is refused as a creating one. + (libc::O_RDONLY | libc::O_DIRECTORY, libc::EACCES), + (libc::O_RDONLY | libc::O_DIRECT, libc::EINVAL), + (libc::O_RDONLY | libc::O_PATH, libc::EINVAL), + ] { + check( + errno_of(raw_openat(provider_path.as_ptr(), flags)) == Some(errno), + &format!("openat provider file with flags {flags:#o}"), + )?; + check( + errno_of(raw_openat2(&provider_path, flags, 0, 0, 24)) == Some(errno), + &format!("openat2 provider file with flags {flags:#o}"), + )?; + } + // An openat2 request the broker cannot serve exactly. + for (mode, resolve, size) in [(0, 0x08, 24), (0o600, 0, 24), (0, 0, 16)] { + check( + errno_of(raw_openat2(&provider_path, read_only, mode, resolve, size)) + == Some(libc::EINVAL), + "openat2 provider file with unsupported open_how", + )?; + } + // A file that is not published is absent whatever the flags. + check( + errno_of(raw_openat(missing.as_ptr(), libc::O_WRONLY)) == Some(libc::ENOENT) + && errno_of(raw_openat2(missing, read_only, 0, 0x08, 24)) + == Some(libc::ENOENT), + "unpublished provider file", + )?; + + // Ordinary opens are the kernel's, with every open_how field. + let ordinary = std::ffi::CString::new(ordinary.to_str().unwrap()).unwrap(); + let created = std::ffi::CString::new(created.to_str().unwrap()).unwrap(); + // RESOLVE_NO_MAGICLINKS: refused for a provider file above. + let mut content = String::new(); + raw_openat2(&ordinary, read_only, 0, 0x02, 24) + .and_then(|mut file| file.read_to_string(&mut content)) + .map_err(|e| format!("openat2 ordinary file: {e}"))?; + check(content == "ordinary", "ordinary file content")?; + raw_openat2( + &created, + libc::O_WRONLY | libc::O_CREAT | libc::O_EXCL, + 0o600, + 0, + 24, + ) + .and_then(|mut file| file.write_all(b"created")) + .map_err(|e| format!("openat2 creating an ordinary file: {e}"))?; + check( + errno_of(raw_openat2(&ordinary, read_only, 0, 0, 16)) == Some(libc::EINVAL) + && errno_of(raw_openat(std::ptr::null(), libc::O_RDONLY)) + == Some(libc::EFAULT), + "the kernel's own errors for an ordinary open", + )?; + // A short path that ends right before an unmapped page. + let short = PathBeforeUnmappedPage::new(c"/etc/passwd"); + raw_openat(short.path, libc::O_RDONLY) + .map_err(|e| format!("ordinary path before an unmapped page: {e}"))?; + // Shorter than the six classification bytes: that copy + // fails, and the kernel still resolves the path. + let shorter = PathBeforeUnmappedPage::new(c"/etc"); + raw_openat(shorter.path, libc::O_RDONLY) + .map_err(|e| format!("five-byte path before an unmapped page: {e}"))?; + + // The caller's thread name file is served by the broker, not + // continued: the kernel would ignore O_EXCL without O_CREAT, + // the broker reports EEXIST. + let comm = c"/proc/thread-self/comm"; + let comm_at_page_end = PathBeforeUnmappedPage::new(comm); + let exclusive = libc::O_WRONLY | libc::O_EXCL; + check( + errno_of(raw_openat(comm.as_ptr(), exclusive)) == Some(libc::EEXIST) + && errno_of(raw_openat2(comm, exclusive, 0, 0, 24)) == Some(libc::EEXIST) + && errno_of(raw_openat(comm_at_page_end.path, exclusive)) + == Some(libc::EEXIST), + "thread name open was not mediated", + )?; + raw_openat2(comm, libc::O_WRONLY | libc::O_CLOEXEC, 0, 0, 24) + .and_then(|mut file| file.write_all(b"renamed")) + .map_err(|e| format!("rename thread through openat2: {e}"))?; + let name = std::fs::read_to_string("/proc/thread-self/comm") + .map_err(|e| format!("read thread name: {e}"))?; + check(name == "renamed\n", "thread was not renamed")?; + raw_openat(comm_at_page_end.path, libc::O_WRONLY | libc::O_CLOEXEC) + .and_then(|mut file| file.write_all(b"again")) + .map_err(|e| format!("rename thread through openat: {e}"))?; + let name = std::fs::read_to_string("/proc/thread-self/comm") + .map_err(|e| format!("read thread name: {e}"))?; + check(name == "again\n", "thread was not renamed again")?; + // Other /proc paths, and a directory open, stay the kernel's. + std::fs::read_to_string("/proc/self/status") + .map_err(|e| format!("read /proc/self/status: {e}"))?; + raw_openat(c"/proc/self".as_ptr(), libc::O_RDONLY | libc::O_DIRECTORY) + .map_err(|e| format!("open /proc/self: {e}"))?; + Ok(()) + }) + .unwrap(); + outcome.unwrap(); + assert_eq!( + std::fs::read_to_string(scratch.path().join("created")).unwrap(), + "created" + ); + broker.confirm_healthy().expect("broker stays healthy"); + } + use std::io::{Read as _, Write as _}; use std::os::unix::net::{UnixListener, UnixStream}; @@ -2173,7 +2824,7 @@ mod tests { assert_eq!(registry.retained_preconnect_count(), 1); let registry = Mutex::new(registry); - let mut observed = std::collections::VecDeque::from([64, 63]); + let mut observed = VecDeque::from([64, 63]); prepare_registry_for_socket_with_count(®istry, 128, || { observed .pop_front() @@ -2209,7 +2860,7 @@ mod tests { registry.commit(tentative).unwrap(); let registry = Mutex::new(registry); - let mut observed = std::collections::VecDeque::from([120, 112]); + let mut observed = VecDeque::from([120, 112]); ensure_descriptor_headroom_with_count(®istry, 128, 16, || { observed .pop_front() @@ -3211,6 +3862,836 @@ mod tests { assert_eq!(nonblocking_result, Some(libc::EINPROGRESS)); } + /// A pool job that reports its worker thread and optionally waits. + struct TestJob { + release: Option>, + done: std::sync::mpsc::Sender, + panic: bool, + } + + impl PoolJob for TestJob { + fn run_contained(self) { + if let Some(release) = &self.release { + let _ = release.recv(); + } + let _ = self.done.send(std::thread::current().id()); + assert!(!self.panic, "test job escapes its containment"); + } + } + + /// An admission permit that counts itself, like `PendingOpenSlot`. + struct TestPermit(Arc); + + impl TestPermit { + fn acquire(held: &Arc) -> Self { + held.fetch_add(1, Ordering::AcqRel); + Self(Arc::clone(held)) + } + + /// Admit like `acquire_pending_open_slot`: refuse at `quota`. + fn try_acquire(held: &Arc, quota: usize) -> Option { + held.fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| { + (current < quota).then_some(current + 1) + }) + .ok() + .map(|_| Self(Arc::clone(held))) + } + } + + impl Drop for TestPermit { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::AcqRel); + } + } + + type TestPool = Arc>; + + fn test_pool(max_workers: usize, max_idle: usize, idle_timeout: Duration) -> TestPool { + ConnectWorkers::new(ConnectWorkerLimits { + max_workers, + max_idle, + idle_timeout, + }) + } + + fn quick_job(done: &std::sync::mpsc::Sender) -> TestJob { + TestJob { + release: None, + done: done.clone(), + panic: false, + } + } + + fn blocked_job( + done: &std::sync::mpsc::Sender, + ) -> (TestJob, std::sync::mpsc::Sender<()>) { + let (release, wait) = std::sync::mpsc::channel(); + ( + TestJob { + release: Some(wait), + done: done.clone(), + panic: false, + }, + release, + ) + } + + fn wait_for(mut condition: impl FnMut() -> bool, what: &str) { + let deadline = Instant::now() + Duration::from_secs(10); + while !condition() { + assert!(Instant::now() < deadline, "timed out waiting for {what}"); + std::thread::sleep(Duration::from_millis(2)); + } + } + + /// The pool's structural invariants, checked under its lock. + fn assert_pool_invariants(pool: &TestPool) { + let state = lock(&pool.state); + assert!(state.workers <= pool.limits.max_workers, "live worker cap"); + assert!(state.idle <= pool.limits.max_idle, "parked worker cap"); + assert!( + state.idle + state.queue.len() <= state.workers, + "parked and reserved workers are live workers" + ); + } + + #[test] + fn connect_workers_reuse_a_parked_thread() { + let held = Arc::new(AtomicUsize::new(0)); + let pool = test_pool(8, 4, Duration::from_secs(30)); + let (done, finished) = std::sync::mpsc::channel(); + let mut threads = std::collections::HashSet::new(); + for _ in 0..16 { + pool.submit(quick_job(&done), TestPermit::acquire(&held)) + .unwrap(); + threads.insert(finished.recv_timeout(Duration::from_secs(10)).unwrap()); + // The worker parks before the next submission reserves it. + wait_for(|| lock(&pool.state).idle == 1, "worker to park"); + // A parked worker holds no admission permit. + wait_for(|| held.load(Ordering::Acquire) == 0, "permit release"); + } + assert_eq!(threads.len(), 1, "sequential jobs share one worker"); + assert_eq!(pool.spawned(), 1); + assert_eq!(pool.workers(), 1); + } + + #[test] + fn connect_workers_never_queue_behind_a_busy_worker() { + // A blocked job models a connect to a full listener. Later jobs must + // start at once on other threads, and the pool must not leak the + // reservation when the blocked ones finish. + let held = Arc::new(AtomicUsize::new(0)); + let pool = test_pool(8, 4, Duration::from_secs(30)); + let (done, finished) = std::sync::mpsc::channel(); + let mut releases = Vec::new(); + for _ in 0..3 { + let (job, release) = blocked_job(&done); + releases.push(release); + pool.submit(job, TestPermit::acquire(&held)).unwrap(); + } + assert_eq!(pool.workers(), 3); + assert_eq!(lock(&pool.state).idle, 0); + let started = Instant::now(); + pool.submit(quick_job(&done), TestPermit::acquire(&held)) + .unwrap(); + let quick = finished.recv_timeout(Duration::from_secs(10)).unwrap(); + assert!(started.elapsed() < Duration::from_secs(2)); + assert_eq!(pool.spawned(), 4); + let mut blocked = std::collections::HashSet::new(); + for release in releases { + release.send(()).unwrap(); + blocked.insert(finished.recv_timeout(Duration::from_secs(10)).unwrap()); + } + assert_eq!(blocked.len(), 3); + assert!(!blocked.contains(&quick)); + wait_for(|| lock(&pool.state).idle == 4, "all workers to park"); + // Parked workers absorb a burst without new threads, and every + // queued job is taken. + for _ in 0..4 { + pool.submit(quick_job(&done), TestPermit::acquire(&held)) + .unwrap(); + } + for _ in 0..4 { + finished.recv_timeout(Duration::from_secs(10)).unwrap(); + } + assert_eq!(pool.spawned(), 4); + wait_for(|| held.load(Ordering::Acquire) == 0, "permit release"); + assert!(lock(&pool.state).queue.is_empty()); + assert_pool_invariants(&pool); + } + + #[test] + fn connect_workers_exit_after_the_idle_timeout() { + let held = Arc::new(AtomicUsize::new(0)); + let pool = test_pool(8, 4, Duration::from_millis(50)); + let (done, finished) = std::sync::mpsc::channel(); + for _ in 0..2 { + pool.submit(quick_job(&done), TestPermit::acquire(&held)) + .unwrap(); + } + for _ in 0..2 { + finished.recv_timeout(Duration::from_secs(10)).unwrap(); + } + wait_for(|| pool.workers() == 0, "idle workers to exit"); + assert_eq!(lock(&pool.state).idle, 0); + assert_eq!(held.load(Ordering::Acquire), 0); + // The pool starts a fresh worker afterwards. + let before = pool.spawned(); + pool.submit(quick_job(&done), TestPermit::acquire(&held)) + .unwrap(); + finished.recv_timeout(Duration::from_secs(10)).unwrap(); + assert_eq!(pool.spawned(), before + 1); + } + + #[test] + fn connect_workers_account_for_a_worker_lost_to_a_panic() { + let held = Arc::new(AtomicUsize::new(0)); + let pool = test_pool(8, 4, Duration::from_secs(30)); + let (done, finished) = std::sync::mpsc::channel(); + pool.submit( + TestJob { + release: None, + done: done.clone(), + panic: true, + }, + TestPermit::acquire(&held), + ) + .unwrap(); + finished.recv_timeout(Duration::from_secs(10)).unwrap(); + // The lost worker gives back both its live registration and the + // admission permit of the job that took it down. + wait_for(|| pool.workers() == 0, "panicked worker to be accounted"); + wait_for(|| held.load(Ordering::Acquire) == 0, "permit release"); + assert_eq!(lock(&pool.state).idle, 0); + pool.submit(quick_job(&done), TestPermit::acquire(&held)) + .unwrap(); + finished.recv_timeout(Duration::from_secs(10)).unwrap(); + assert_eq!(pool.spawned(), 2); + } + + #[test] + fn connect_workers_refuse_beyond_the_live_worker_cap() { + // The live-thread cap is the pool's own: it holds even when the + // caller's admission quota would allow more. + let held = Arc::new(AtomicUsize::new(0)); + let pool = test_pool(3, 2, Duration::from_secs(30)); + let (done, finished) = std::sync::mpsc::channel(); + let mut releases = Vec::new(); + for _ in 0..3 { + let (job, release) = blocked_job(&done); + releases.push(release); + pool.submit(job, TestPermit::acquire(&held)).unwrap(); + } + assert_eq!(pool.workers(), 3); + let started = Instant::now(); + let error = pool + .submit(quick_job(&done), TestPermit::acquire(&held)) + .expect_err("a fourth live worker must be refused"); + assert_eq!(error.raw_os_error(), Some(libc::EAGAIN)); + assert!( + started.elapsed() < Duration::from_secs(2), + "a refused submission must not wait for a worker" + ); + // The refused job gave its permit back and started nothing. + assert_eq!(held.load(Ordering::Acquire), 3); + assert_eq!(pool.spawned(), 3); + assert_eq!(pool.workers(), 3); + assert!(lock(&pool.state).queue.is_empty()); + for release in releases { + release.send(()).unwrap(); + finished.recv_timeout(Duration::from_secs(10)).unwrap(); + } + // Two park, the third retires at once rather than after the timeout. + wait_for(|| pool.workers() == 2, "surplus worker to retire"); + wait_for(|| held.load(Ordering::Acquire) == 0, "permit release"); + assert_eq!(lock(&pool.state).idle, 2); + // Capacity is available again. + pool.submit(quick_job(&done), TestPermit::acquire(&held)) + .unwrap(); + finished.recv_timeout(Duration::from_secs(10)).unwrap(); + assert_eq!(pool.spawned(), 3); + } + + #[test] + fn connect_worker_permit_outlives_its_busy_worker() { + // The permit is released only once the pool state records the worker + // parked or retired. Sampled under the pool lock, busy workers and + // queued jobs therefore never outnumber held permits, so a + // submission admitted by a freed permit cannot find the freeing + // thread still counted as busy. With the admission quota equal to + // the live-worker cap, as in the broker, the cap must therefore + // never refuse a job the quota admitted. + const QUOTA: usize = 6; + let held = Arc::new(AtomicUsize::new(0)); + let pool = test_pool(QUOTA, 2, Duration::from_millis(5)); + let (done, finished) = std::sync::mpsc::channel(); + let stop = Arc::new(AtomicBool::new(false)); + let sampler = { + let (pool, held, stop) = (Arc::clone(&pool), Arc::clone(&held), Arc::clone(&stop)); + std::thread::spawn(move || { + let mut samples = 0_usize; + while !stop.load(Ordering::Acquire) { + { + let state = lock(&pool.state); + // Read the permits while holding the lock: a worker + // changes its state under the lock before releasing. + let permits = held.load(Ordering::Acquire); + let busy = state.workers - state.idle - state.queue.len(); + assert!( + busy + state.queue.len() <= permits, + "{busy} busy and {} queued exceed {permits} permits", + state.queue.len() + ); + assert!(state.workers <= pool.limits.max_workers); + assert!(state.idle <= pool.limits.max_idle); + } + samples += 1; + std::thread::yield_now(); + } + samples + }) + }; + let submitters = (0..8) + .map(|_| { + let (pool, held, done) = (Arc::clone(&pool), Arc::clone(&held), done.clone()); + std::thread::spawn(move || { + for round in 0..400 { + // Wait for admission, as a blocked workload retries. + let permit = loop { + if let Some(permit) = TestPermit::try_acquire(&held, QUOTA) { + break permit; + } + std::thread::yield_now(); + }; + pool.submit(quick_job(&done), permit) + .expect("the worker cap refused an admitted job"); + if round % 16 == 0 { + // Let parked workers reach their idle timeout so + // retirement races with submission too. + std::thread::sleep(Duration::from_millis(6)); + } + } + }) + }) + .collect::>(); + for submitter in submitters { + submitter.join().unwrap(); + } + // Every job ran: none was stranded in the queue by a worker that + // retired or timed out. + for _ in 0..3_200 { + finished.recv_timeout(Duration::from_secs(10)).unwrap(); + } + wait_for(|| held.load(Ordering::Acquire) == 0, "permit release"); + wait_for(|| pool.workers() == 0, "workers to time out"); + stop.store(true, Ordering::Release); + assert!(sampler.join().expect("invariant sampler") > 0); + let state = lock(&pool.state); + assert_eq!((state.idle, state.queue.len()), (0, 0)); + } + + #[test] + fn connect_workers_do_not_starve_queued_jobs_at_the_idle_cap() { + // With one parked slot, jobs queued to the parked worker while + // others finish and retire must still all run, and promptly. + let held = Arc::new(AtomicUsize::new(0)); + let pool = test_pool(16, 1, Duration::from_secs(30)); + let (done, finished) = std::sync::mpsc::channel(); + for _ in 0..200 { + let mut releases = Vec::new(); + for _ in 0..3 { + let (job, release) = blocked_job(&done); + releases.push(release); + pool.submit(job, TestPermit::acquire(&held)).unwrap(); + } + // Finish the blocked jobs while submitting quick ones, so + // reservation, direct takeover, and retirement interleave. + for release in releases { + release.send(()).unwrap(); + pool.submit(quick_job(&done), TestPermit::acquire(&held)) + .unwrap(); + } + for _ in 0..6 { + finished + .recv_timeout(Duration::from_secs(10)) + .expect("a queued job was starved"); + } + assert_pool_invariants(&pool); + } + wait_for(|| held.load(Ordering::Acquire) == 0, "permit release"); + wait_for(|| pool.workers() == 1, "surplus workers to retire"); + let state = lock(&pool.state); + assert_eq!((state.idle, state.queue.len()), (1, 0)); + } + + /// Accept and hold every connection so blocking connects complete. + fn accepting_service() -> SocketAddr { + let service = TcpListener::bind("127.0.0.1:0").unwrap(); + let address = service.local_addr().unwrap(); + std::thread::spawn(move || { + // Leak each accepted connection so it stays open for the test. + for stream in service.incoming() { + let Ok(stream) = stream else { break }; + std::mem::forget(stream); + } + }); + address + } + + #[test] + fn blocking_loopback_connects_reuse_one_worker_and_release_their_slots() { + let address = accepting_service(); + let (launcher, listener) = + crate::linux::workload_launcher::start().expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let peers = launcher + .execute(move || -> io::Result> { + (0..24) + .map(|_| { + let socket = socket2::Socket::new( + socket2::Domain::IPV4, + socket2::Type::STREAM, + None, + )?; + // A blocking connect: the worker path, not the + // dispatcher's nonblocking one. + socket.connect(&address.into())?; + Ok(socket.peer_addr()?.as_socket().unwrap()) + }) + .collect() + }) + .expect("launcher result") + .expect("blocking connects"); + assert!(peers.iter().all(|peer| *peer == address)); + assert_eq!( + broker.connect_workers.spawned(), + 1, + "sequential blocking connects must not start a thread each" + ); + // Each job returned its admission slot when it finished. + wait_for( + || broker.active_opens.load(Ordering::Acquire) == 0, + "admission slots to be released", + ); + } + + #[test] + fn blocked_loopback_connects_do_not_delay_another_blocking_connect() { + // Several blocking connects wait on a listener with a full backlog. + // A blocking connect to a healthy listener must not queue behind + // them, and unrelated mediation must stay prompt. + let saturated = + socket2::Socket::new(socket2::Domain::IPV4, socket2::Type::STREAM, None).unwrap(); + saturated + .bind(&"127.0.0.1:0".parse::().unwrap().into()) + .unwrap(); + saturated.listen(0).unwrap(); + let stalled = saturated.local_addr().unwrap().as_socket().unwrap(); + let healthy = accepting_service(); + let (launcher, listener) = + crate::linux::workload_launcher::start().expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let (connect_wait, socket_wait) = launcher + .execute(move || { + let filled = TcpStream::connect(stalled).expect("fill accept queue"); + let pending = (0..3) + .map(|_| std::thread::spawn(move || TcpStream::connect(stalled))) + .collect::>(); + std::thread::sleep(Duration::from_millis(500)); + let started = Instant::now(); + let socket = + socket2::Socket::new(socket2::Domain::IPV4, socket2::Type::STREAM, None) + .unwrap(); + socket.connect(&healthy.into()).expect("healthy connect"); + let connect_wait = started.elapsed(); + let started = Instant::now(); + drop(UdpSocket::bind("127.0.0.1:0")); + let socket_wait = started.elapsed(); + // Leave the stalled connects running; closing the listener + // when the test ends resets them. + drop(pending); + drop(filled); + (connect_wait, socket_wait) + }) + .expect("launcher result"); + assert!( + connect_wait < Duration::from_secs(2), + "blocking connect waited {connect_wait:?} behind stalled connects" + ); + assert!( + socket_wait < Duration::from_secs(2), + "socket creation waited {socket_wait:?} behind stalled connects" + ); + // Three workers are still waiting; a fourth served the healthy one. + assert!(broker.connect_workers.spawned() >= 4); + assert!(broker.active_opens.load(Ordering::Acquire) >= 3); + drop(saturated); + wait_for( + || broker.active_opens.load(Ordering::Acquire) == 0, + "stalled connects to release their slots", + ); + } + + #[test] + fn blocking_loopback_connect_admission_is_bounded() { + let address = accepting_service(); + let (launcher, listener) = + crate::linux::workload_launcher::start().expect("start workload launcher"); + let broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let connect = move || { + let socket = + socket2::Socket::new(socket2::Domain::IPV4, socket2::Type::STREAM, None).unwrap(); + let result = socket + .connect(&address.into()) + .err() + .and_then(|error| error.raw_os_error()); + // A refused admission leaves the socket unconnected and usable. + (result, socket.peer_addr().is_ok(), socket) + }; + // Occupy every admission slot, as that many waiting opens would. + broker + .active_opens + .store(OPEN_QUEUE_CAPACITY, Ordering::Release); + let (refused, connected, socket) = launcher.execute(connect).expect("launcher result"); + assert_eq!(refused, Some(libc::EAGAIN)); + assert!(!connected); + assert_eq!(broker.connect_workers.spawned(), 0); + assert_eq!( + broker.active_opens.load(Ordering::Acquire), + OPEN_QUEUE_CAPACITY + ); + // One slot frees; the same socket then connects. + broker + .active_opens + .store(OPEN_QUEUE_CAPACITY - 1, Ordering::Release); + let retried = launcher + .execute(move || { + socket + .connect(&address.into()) + .err() + .and_then(|error| error.raw_os_error()) + }) + .expect("launcher result"); + assert_eq!(retried, None); + wait_for( + || broker.active_opens.load(Ordering::Acquire) == OPEN_QUEUE_CAPACITY - 1, + "the admitted connect to release its slot", + ); + } + + #[test] + fn local_connect_commit_ignores_a_replaced_descriptor() { + let metadata = SocketMetadata { + family: InetFamily::V4, + kind: InetKind::Tcp, + close_on_exec: true, + nonblocking: false, + creator_generation: 1, + }; + let mut registry = SocketRegistry::new(1, 2).unwrap(); + let mut create = || { + let socket = OwnedFd::from( + socket2::Socket::new(socket2::Domain::IPV4, socket2::Type::STREAM, None).unwrap(), + ); + let installed = rustix::io::fcntl_dupfd_cloexec(&socket, 3).unwrap(); + let tentative = registry.stage(socket, metadata).unwrap(); + let identity = registry.commit(tentative).unwrap(); + (installed, identity) + }; + let (original, original_identity) = create(); + let (replacement, replacement_identity) = create(); + // The workload replaces the descriptor while a worker waits in the + // blocking connect of the original socket. + // SAFETY: both descriptors are live; replace only the test-owned FD. + assert_eq!( + unsafe { libc::dup2(replacement.as_raw_fd(), original.as_raw_fd()) }, + original.as_raw_fd() + ); + let registry = Mutex::new(registry); + let peer: SocketAddr = "127.0.0.1:8080".parse().unwrap(); + commit_local_connect( + ®istry, + std::process::id(), + original.as_raw_fd(), + original_identity, + peer, + ); + { + let registry = lock(®istry); + let entry = registry + .resolve(std::process::id(), original.as_raw_fd()) + .unwrap(); + assert_eq!(entry.identity(), replacement_identity); + assert_eq!(entry.state(), &SocketState::Created); + assert!(entry.retained_preconnect().is_ok()); + } + // The matching identity is recorded and releases the source. + commit_local_connect( + ®istry, + std::process::id(), + original.as_raw_fd(), + replacement_identity, + peer, + ); + let registry = lock(®istry); + let entry = registry + .resolve(std::process::id(), original.as_raw_fd()) + .unwrap(); + assert_eq!(entry.state(), &SocketState::Local { peer }); + assert!(entry.retained_preconnect().is_err()); + } + + #[test] + fn descriptor_headroom_counts_once_without_pressure() { + let registry = Mutex::new(SocketRegistry::new(1, 2).unwrap()); + let mut counts = 0; + prepare_registry_for_socket_with_count(®istry, 1_024, || { + counts += 1; + Ok(32) + }) + .unwrap(); + assert_eq!(counts, 1, "socket creation enumerates descriptors once"); + let mut counts = 0; + ensure_descriptor_headroom_with_count(®istry, 1_024, 16, || { + counts += 1; + Ok(32) + }) + .unwrap(); + assert_eq!(counts, 1, "exec headroom enumerates descriptors once"); + } + + #[test] + fn descriptor_headroom_recounts_after_collection_and_fails_closed() { + let registry = Mutex::new(SocketRegistry::new(1, 2).unwrap()); + // Ambient use grows past the reserve and collection frees nothing: + // the recount after collection must refuse the socket. + let mut observed = VecDeque::from([64, 64]); + let error = prepare_registry_for_socket_with_count(®istry, 128, || { + observed + .pop_front() + .ok_or_else(|| io::Error::other("unexpected descriptor recount")) + }) + .expect_err("exhausted headroom must refuse the socket"); + assert_eq!(error.raw_os_error(), Some(libc::EMFILE)); + assert!(observed.is_empty(), "collection is followed by one recount"); + // Use that grows only after the single count is seen by the next + // request, exactly as a second count taken moments later would be. + let mut observed = VecDeque::from([63, 64, 64]); + let mut count = || { + observed + .pop_front() + .ok_or_else(|| io::Error::other("unexpected descriptor recount")) + }; + prepare_registry_for_socket_with_count(®istry, 128, &mut count).unwrap(); + assert_eq!( + prepare_registry_for_socket_with_count(®istry, 128, &mut count) + .expect_err("grown ambient use must refuse the next socket") + .raw_os_error(), + Some(libc::EMFILE) + ); + let mut observed = VecDeque::from([120, 120]); + assert_eq!( + ensure_descriptor_headroom_with_count(®istry, 128, 16, || { + observed + .pop_front() + .ok_or_else(|| io::Error::other("unexpected descriptor recount")) + }) + .expect_err("exhausted exec headroom") + .raw_os_error(), + Some(libc::EMFILE) + ); + assert!(observed.is_empty()); + } + + #[test] + fn single_copy_socket_address_matches_the_separate_reads() { + let tid = std::process::id(); + let errno = |error: &io::Error| error.raw_os_error(); + let compare = |address: u64, length: u64| { + let family = read_socket_family(tid, address, length); + let combined = read_socket_family_and_addr(tid, address, length); + match (&family, &combined) { + (Ok(family), Ok((combined_family, combined_addr))) => { + assert_eq!(family, combined_family, "family at length {length}"); + let separate = read_socket_addr(tid, address, length); + match (&separate, combined_addr) { + (Ok(separate), Ok(combined)) => assert_eq!(separate, combined), + (Err(separate), Err(combined)) => { + assert_eq!(errno(separate), errno(combined), "length {length}"); + } + _ => panic!("address results differ at length {length}"), + } + } + (Err(family), Err(combined)) => { + assert_eq!(errno(family), errno(combined), "length {length}"); + } + _ => panic!("family results differ at length {length}"), + } + combined + }; + + let v4: SocketAddr = "127.0.0.1:8080".parse().unwrap(); + let v6: SocketAddr = "[::1]:9090".parse().unwrap(); + for expected in [v4, v6] { + with_sockaddr(expected, |pointer, length| { + let (family, address) = compare(pointer as u64, u64::from(length)).unwrap(); + assert!(matches!(family, libc::AF_INET | libc::AF_INET6)); + assert_eq!(address.unwrap(), expected); + // A truncated INET address keeps its family and is refused. + let (family, address) = compare(pointer as u64, 8).unwrap(); + assert!(matches!(family, libc::AF_INET | libc::AF_INET6)); + assert_eq!(errno(&address.unwrap_err()), Some(libc::EAFNOSUPPORT)); + // Lengths the kernel would fault or refuse. + assert_eq!( + errno(&compare(pointer as u64, 1).unwrap_err()), + Some(libc::EFAULT) + ); + assert_eq!(errno(&compare(0, 16).unwrap_err()), Some(libc::EFAULT)); + Ok(()) + }) + .unwrap(); + } + + // A non-INET family is reported without decoding an address. + let mut unix: libc::sockaddr_un = unsafe { std::mem::zeroed() }; + unix.sun_family = libc::sa_family_t::try_from(libc::AF_UNIX).unwrap(); + let (family, address) = compare( + std::ptr::addr_of!(unix) as u64, + size_of::() as u64, + ) + .unwrap(); + assert_eq!(family, libc::AF_UNIX); + assert_eq!(errno(&address.unwrap_err()), Some(libc::EAFNOSUPPORT)); + + // An address whose declared length runs into an unmapped page, or + // exceeds sockaddr_storage, still yields the family from its first + // bytes; only the address read fails, as before. + // SAFETY: map two private pages and make the second inaccessible. + let page = 4096_usize; + let mapping = unsafe { + libc::mmap( + std::ptr::null_mut(), + page * 2, + libc::PROT_READ | libc::PROT_WRITE, + libc::MAP_PRIVATE | libc::MAP_ANONYMOUS, + -1, + 0, + ) + }; + assert_ne!(mapping, libc::MAP_FAILED); + // SAFETY: the second page lies within the fresh mapping. + assert_eq!( + unsafe { libc::mprotect(mapping.cast::().add(page).cast(), page, libc::PROT_NONE) }, + 0 + ); + let tail = size_of::(); + // SAFETY: the last `tail` bytes of the first page are writable. + let location = unsafe { mapping.cast::().add(page - tail) }; + with_sockaddr(v4, |pointer, _| { + // SAFETY: source and destination are live and do not overlap. + unsafe { std::ptr::copy_nonoverlapping(pointer.cast::(), location, tail) }; + Ok(()) + }) + .unwrap(); + let exact = compare(location as u64, tail as u64).unwrap(); + assert_eq!(exact.1.unwrap(), v4); + for length in [tail as u64 + 1, 128, 129, 4096] { + let (family, address) = compare(location as u64, length).unwrap(); + assert_eq!(family, libc::AF_INET, "length {length}"); + assert!(address.is_err(), "length {length}"); + } + // SAFETY: unmap the mapping created above. + unsafe { libc::munmap(mapping, page * 2) }; + } + + #[test] + fn socket_option_fast_path_pairs_are_never_denied() { + use crate::linux::seccomp_notify::SETSOCKOPT_FAST_PATH; + assert!(!SETSOCKOPT_FAST_PATH.is_empty()); + for (level, option) in SETSOCKOPT_FAST_PATH { + // The filter allows these pairs without asking the broker, so + // each must be one the broker would continue, and must fit the + // non-negative i32 the broker requires. + assert!( + !socket_option_is_denied(*level, *option), + "{level}/{option} is denied but takes the filter fast path" + ); + assert!(*level >= 0 && *option >= 0); + } + assert!(SETSOCKOPT_FAST_PATH.contains(&(libc::IPPROTO_TCP, libc::TCP_NODELAY))); + } + + #[test] + fn workload_socket_options_keep_their_results_with_the_filter_fast_path() { + const HIGH_WORD: u64 = 1 << 32; + let address = accepting_service(); + let (launcher, listener) = + crate::linux::workload_launcher::start().expect("start workload launcher"); + let _broker = NetworkBroker::start_for_test(listener).expect("start network broker"); + let results = launcher + .execute(move || -> io::Result>> { + let socket = + socket2::Socket::new(socket2::Domain::IPV4, socket2::Type::STREAM, None)?; + socket.connect(&address.into())?; + let raw = |level: u64, option: u64| { + let value: libc::c_int = 1; + // SAFETY: the value pointer and length describe one int. + let result = unsafe { + libc::syscall( + libc::SYS_setsockopt, + socket.as_raw_fd(), + level, + option, + std::ptr::addr_of!(value), + size_of::(), + ) + }; + (result < 0).then(|| io::Error::last_os_error().raw_os_error().unwrap()) + }; + let arg = |value: i32| u64::from(value.cast_unsigned()); + let tcp = arg(libc::IPPROTO_TCP); + let sol = arg(libc::SOL_SOCKET); + let results = vec![ + // Filter fast path. + raw(tcp, arg(libc::TCP_NODELAY)), + raw(sol, arg(libc::SO_REUSEADDR)), + raw(sol, arg(libc::SO_KEEPALIVE)), + // Broker-continued. + raw(sol, arg(libc::SO_RCVBUF)), + // Broker-denied. + raw(sol, arg(libc::SO_BINDTOIFINDEX)), + raw(tcp, arg(libc::TCP_FASTOPEN_CONNECT)), + // Unrepresentable arguments keep the broker's EINVAL, + // including for an otherwise whitelisted pair. + raw(tcp | HIGH_WORD, arg(libc::TCP_NODELAY)), + raw(tcp, arg(libc::TCP_NODELAY) | HIGH_WORD), + raw(tcp, arg(libc::TCP_NODELAY) | 0x8000_0000), + raw(sol | 0x8000_0000, arg(libc::SO_REUSEADDR)), + ]; + // The fast path really set the option on the socket. + assert!(socket.tcp_nodelay()?); + assert!(socket.keepalive()?); + Ok(results) + }) + .expect("launcher result") + .expect("workload socket"); + assert_eq!( + results, + vec![ + None, + None, + None, + None, + Some(libc::EPERM), + Some(libc::EPERM), + Some(libc::EINVAL), + Some(libc::EINVAL), + Some(libc::EINVAL), + Some(libc::EINVAL), + ] + ); + } + #[test] fn workload_cannot_bind_a_non_loopback_source_address() { // Workload sockets can only present loopback source addresses. A diff --git a/crates/openshell-sandbox/src/perf.rs b/crates/openshell-sandbox/src/perf.rs index 408d6d69a1..558b50c094 100644 --- a/crates/openshell-sandbox/src/perf.rs +++ b/crates/openshell-sandbox/src/perf.rs @@ -2,11 +2,30 @@ // SPDX-License-Identifier: Apache-2.0 //! Opt-in microbenchmarks for the production seccomp network listener. +//! +//! Besides the production broker, one harness-only layer attributes the cost +//! of a workload file open. It exists only under this feature and adds no +//! switch to the production runtime: `notify-only` installs the production +//! filter and answers every notification with "continue" from a loop that +//! reads no workload memory. Against `filtered`, it separates the +//! notification round trip from the broker's path-prefix copy. +//! +//! `--landlock` additionally confines the worker with a Landlock domain, to +//! show what the filesystem hook adds to an open in any layer. +//! +//! `--provider-file` makes the file protocols open a managed provider file, +//! which the `filtered` layer's broker serves by injection, instead of an +//! ordinary file the kernel opens. The `mixed-*` protocols run file opens and +//! TCP operations at the same time and report each class separately. +use std::fs::File; use std::io::{self, Read as _, Write as _}; use std::net::{IpAddr, Ipv4Addr, SocketAddr, TcpListener, TcpStream}; +use std::os::unix::fs::FileExt as _; +use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; -use std::sync::mpsc; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::{Arc, mpsc}; use std::thread; use std::time::{Duration, Instant}; @@ -21,16 +40,26 @@ use crate::network_broker::NetworkBroker; #[derive(Clone, Copy, Debug, Deserialize, Serialize, ValueEnum)] #[serde(rename_all = "kebab-case")] pub enum Layer { + /// No seccomp filter. Native, + /// The production listener and broker; every file open notifies. Filtered, + /// Harness only: the production filter answered by a loop that continues + /// every notification without reading workload memory. + NotifyOnly, } impl Layer { + /// `all` is the original native/filtered pair, so existing invocations + /// keep measuring what they measured. `attribution` adds the layer that + /// separates the parts of a file open. pub fn selection(value: &str) -> anyhow::Result> { match value { "all" => Ok(vec![Self::Native, Self::Filtered]), + "attribution" => Ok(vec![Self::Native, Self::NotifyOnly, Self::Filtered]), "native" => Ok(vec![Self::Native]), "filtered" => Ok(vec![Self::Filtered]), + "notify-only" => Ok(vec![Self::NotifyOnly]), _ => bail!("unknown layer {value}"), } } @@ -41,12 +70,30 @@ impl Layer { pub enum Protocol { TcpConnect, TcpStream, + /// Open a file read-only and close it. + FileOpenClose, + /// Open a file read-only, read `payload-bytes` from it, and close it. + FileOpenRead, + /// Read `payload-bytes` from one descriptor opened before measurement. + FileRead, + /// `file-open-read` and `tcp-connect` workers running at the same time. + MixedFileTcpConnect, + /// `file-open-read` and `tcp-stream` workers running at the same time. + MixedFileTcpStream, } impl Protocol { + /// `all` is the original TCP pair; `file` is the file-open attribution + /// set; `mixed` is the two contention workloads. pub fn selection(value: &str) -> anyhow::Result> { match value { "all" => Ok(vec![Self::TcpConnect, Self::TcpStream]), + "mixed" => Ok(vec![Self::MixedFileTcpConnect, Self::MixedFileTcpStream]), + "file" => Ok(vec![ + Self::FileOpenClose, + Self::FileOpenRead, + Self::FileRead, + ]), _ => Ok(vec![ Self::from_str(value, true).map_err(|error| anyhow::anyhow!("{error}"))?, ]), @@ -62,6 +109,48 @@ pub struct BenchmarkOptions { pub warmup: u64, pub concurrency: usize, pub payload_bytes: usize, + /// Confine each worker with a Landlock domain before it measures. + pub landlock: bool, + /// File the file protocols open. + pub file_path: PathBuf, + /// Request the kernel's synchronous-wakeup hint on each filtered layer's + /// listener. Fails where the kernel does not support it. + pub sync_wakeup: bool, + /// Threads answering notifications in the notify-only layer. + pub notify_receivers: usize, + /// Open a managed provider file of this many bytes instead of + /// `file_path`. Only the filtered layer's virtual delivery serves one. + pub provider_file_bytes: Option, +} + +/// The managed provider file `--provider-file` publishes and opens. +pub const PROVIDER_FILE_PATH: &str = "/run/openshell/providers/perf/config.toml"; +/// The largest provider file the broker accepts. +const MAX_PROVIDER_FILE_BYTES: usize = 65_536; + +/// The content of the synthetic provider file: the same for every run of a +/// given size, so a worker can check what it reads without being told. +fn synthetic_provider_file(bytes: usize) -> String { + use std::fmt::Write as _; + let mut content = String::with_capacity(bytes + 64); + let mut line = 0_usize; + while content.len() < bytes { + let _ = writeln!(content, "setting_{line:06} = \"openshell-perf-{line:06}\""); + line += 1; + } + // ASCII only, so any byte offset is a character boundary. + content.truncate(bytes); + content +} + +/// Task-memory copies the broker made while one worker ran, by the backend +/// that served them. +#[derive(Clone, Copy, Debug, Default, Deserialize, Serialize)] +pub struct TaskMemoryReads { + /// Copies `process_vm_readv` completed. + pub process_vm: u64, + /// Copies that fell back to `/proc//mem`. + pub proc_mem_fallback: u64, } #[derive(Debug, Deserialize, Serialize)] @@ -78,28 +167,173 @@ pub struct BenchmarkReport { pub latency_ns_p50: u64, pub latency_ns_p95: u64, pub latency_ns_p99: u64, + #[serde(default)] + pub latency_ns_max: u64, + /// Whether the worker ran inside a Landlock domain. + #[serde(default)] + pub landlock: bool, + /// Broker task-memory copies during this run. Absent for layers without + /// the production broker. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub task_memory_reads: Option, + /// Whether the layer's listener used the synchronous-wakeup hint. + #[serde(default)] + pub sync_wakeup: bool, + /// Threads that answered notifications, for the notify-only layer. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub notify_receivers: Option, + /// The file a file operation opened. Absent for TCP operations. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub file_path: Option, + /// Size of the managed provider file the broker served by injection. + /// Absent when the file was an ordinary one the kernel opened. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub provider_file_bytes: Option, + /// For a mixed protocol: the class of operation this report covers. + /// Every other field then describes that class alone, measured while + /// the other class ran. `iterations` is what the class completed in the + /// shared window, not a requested count. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub class: Option, + /// For a mixed protocol: threads that ran this class. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub class_workers: Option, +} + +/// A launcher whose children inherit one layer's filter, with whatever +/// answers its notifications. +struct LayerRuntime { + launcher: workload_launcher::WorkloadLauncher, + /// Present when the production broker serves the layer. + broker: Option, +} + +impl LayerRuntime { + fn start( + layer: Layer, + sync_wakeup: bool, + notify_receivers: usize, + ) -> anyhow::Result> { + let prepare = |listener: &crate::linux::seccomp_notify::NotificationListener| { + if sync_wakeup { + listener + .enable_sync_wakeup() + .context("request synchronous listener wakeup")?; + } + anyhow::Ok(()) + }; + match layer { + Layer::Native => Ok(None), + Layer::Filtered => { + let (launcher, listener) = workload_launcher::start()?; + prepare(&listener)?; + let broker = NetworkBroker::start_for_test(listener)?; + broker.confirm_healthy()?; + Ok(Some(Self { + launcher, + broker: Some(broker), + })) + } + Layer::NotifyOnly => { + let (launcher, listener) = workload_launcher::start()?; + prepare(&listener)?; + let listener = Arc::new(listener); + // The kernel hands each notification to one receiver. + for _ in 0..notify_receivers { + let listener = Arc::clone(&listener); + thread::Builder::new() + .name("seccomp-perf-notify-only".into()) + .spawn(move || { + loop { + match listener.receive() { + Ok(notification) => { + // The notifying task may have died. + let _ = listener.respond_continue(notification.id); + } + Err(error) + if error.kind() == io::ErrorKind::Interrupted + || error.raw_os_error() == Some(libc::ENOENT) => {} + Err(_) => break, + } + } + }) + .context("start notify-only responder")?; + } + Ok(Some(Self { + launcher, + broker: None, + })) + } + } + } +} + +fn task_memory_reads() -> TaskMemoryReads { + let (process_vm, proc_mem_fallback) = crate::linux::task_memory::read_backend_counts(); + TaskMemoryReads { + process_vm, + proc_mem_fallback, + } } pub fn run(options: BenchmarkOptions) -> anyhow::Result> { - if options.iterations == 0 || options.concurrency == 0 { - bail!("iterations and concurrency must be greater than zero"); + if options.iterations == 0 || options.concurrency == 0 || options.notify_receivers == 0 { + bail!("iterations, concurrency, and notify-receivers must be greater than zero"); } if options.payload_bytes == 0 || options.payload_bytes > 65_507 { bail!("payload-bytes must be between 1 and 65507"); } + if let Some(bytes) = options.provider_file_bytes { + // Refuse rather than measure an ordinary file under another name: + // no other layer serves a provider path by injection. + if bytes == 0 || bytes > MAX_PROVIDER_FILE_BYTES { + bail!("provider-file-bytes must be between 1 and {MAX_PROVIDER_FILE_BYTES}"); + } + if !options + .layers + .iter() + .all(|layer| matches!(layer, Layer::Filtered)) + { + bail!( + "--provider-file measures virtual provider-file delivery, which only the filtered layer has; pass --layer filtered" + ); + } + if !options + .protocols + .iter() + .all(|protocol| protocol.opens_file()) + { + bail!("--provider-file applies to the file and mixed protocols only"); + } + } + let file_path = options + .provider_file_bytes + .map_or(options.file_path, |_| PathBuf::from(PROVIDER_FILE_PATH)); let executable = std::env::current_exe().context("resolve benchmark executable")?; let mut reports = Vec::new(); - let mut filtered_runtime = None; for layer in options.layers { - if matches!(layer, Layer::Filtered) && filtered_runtime.is_none() { - let (launcher, listener) = workload_launcher::start()?; - let broker = NetworkBroker::start_for_test(listener)?; - broker.confirm_healthy()?; - filtered_runtime = Some((launcher, broker)); + // Each layer has its own launcher thread and filter. + let runtime = LayerRuntime::start(layer, options.sync_wakeup, options.notify_receivers)?; + if let Some(bytes) = options.provider_file_bytes { + runtime + .as_ref() + .and_then(|runtime| runtime.broker.as_ref()) + .context("the layer has no broker to serve a provider file")? + .provider_files() + .replace(std::collections::HashMap::from([( + PROVIDER_FILE_PATH.to_string(), + synthetic_provider_file(bytes), + )])) + .context("publish the synthetic provider file")?; } for protocol in &options.protocols { - let fixture = Fixture::start(*protocol, filtered_runtime.as_ref().map(|(_, b)| b))?; + if matches!(layer, Layer::NotifyOnly) && !protocol.is_file() { + bail!( + "the notify-only layer continues every syscall and measures file protocols only" + ); + } + let fixture = Fixture::start(*protocol)?; let mut command = Command::new(&executable); command .arg("worker") @@ -120,24 +354,52 @@ pub fn run(options: BenchmarkOptions) -> anyhow::Result> { .arg(options.concurrency.to_string()) .arg("--payload-bytes") .arg(options.payload_bytes.to_string()) + .arg("--file-path") + .arg(&file_path) .stdout(Stdio::piped()) .stderr(Stdio::inherit()); - let output = match layer { - Layer::Native => command.output(), - Layer::Filtered => filtered_runtime - .as_ref() - .expect("filtered runtime") - .0 - .execute(move || command.output())?, + if let Some(bytes) = options.provider_file_bytes { + command.arg("--provider-file-bytes").arg(bytes.to_string()); + } + if options.landlock { + command.arg("--landlock"); + } + let before = task_memory_reads(); + let output = match &runtime { + None => command.output(), + Some(runtime) => runtime.launcher.execute(move || command.output())?, } .context("run benchmark worker")?; if !output.status.success() { bail!("benchmark worker exited with {}", output.status); } - let mut report: BenchmarkReport = - serde_json::from_slice(&output.stdout).context("decode worker report")?; - report.layer = layer; - reports.push(report); + let after = task_memory_reads(); + // One report, or one per class of a mixed protocol. The + // task-memory copies are those of the whole worker run. + let worker_reports = output + .stdout + .split(|byte| *byte == b'\n') + .filter(|line| !line.is_empty()) + .map(serde_json::from_slice::) + .collect::, _>>() + .context("decode worker report")?; + if worker_reports.is_empty() { + bail!("benchmark worker reported nothing"); + } + for mut report in worker_reports { + report.layer = layer; + report.sync_wakeup = options.sync_wakeup && runtime.is_some(); + report.notify_receivers = + matches!(layer, Layer::NotifyOnly).then_some(options.notify_receivers); + report.task_memory_reads = runtime + .as_ref() + .and_then(|runtime| runtime.broker.as_ref()) + .map(|_| TaskMemoryReads { + process_vm: after.process_vm - before.process_vm, + proc_mem_fallback: after.proc_mem_fallback - before.proc_mem_fallback, + }); + reports.push(report); + } } } Ok(reports) @@ -151,10 +413,14 @@ const OPERATION_TIMEOUT: Duration = Duration::from_secs(10); const START_TIMEOUT: Duration = Duration::from_secs(30); impl Fixture { - fn start(protocol: Protocol, _broker: Option<&NetworkBroker>) -> io::Result { + fn start(protocol: Protocol) -> io::Result { match protocol { - Protocol::TcpConnect => start_tcp_fixture(false), - Protocol::TcpStream => start_tcp_fixture(true), + Protocol::TcpConnect | Protocol::MixedFileTcpConnect => start_tcp_fixture(false), + Protocol::TcpStream | Protocol::MixedFileTcpStream => start_tcp_fixture(true), + // File protocols need no peer; the worker ignores the target. + Protocol::FileOpenClose | Protocol::FileOpenRead | Protocol::FileRead => Ok(Self { + target: SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0), + }), } } } @@ -189,43 +455,167 @@ fn start_tcp_fixture(echo: bool) -> io::Result { Ok(Fixture { target }) } -#[allow(clippy::cast_precision_loss)] -pub fn run_worker( - protocol: Protocol, - target: SocketAddr, - iterations: u64, - warmup: u64, - concurrency: usize, - payload_bytes: usize, -) -> anyhow::Result { +/// What one worker process measures. +#[derive(Clone, Debug)] +pub struct WorkerRun { + pub protocol: Protocol, + pub target: SocketAddr, + pub iterations: u64, + pub warmup: u64, + pub concurrency: usize, + pub payload_bytes: usize, + pub landlock: bool, + pub file_path: PathBuf, + /// `file_path` is the synthetic provider file of this size; verify it. + pub provider_file_bytes: Option, +} + +/// Confine the calling process with a Landlock domain that handles every +/// ABI v3 filesystem right and allows them all beneath `/`. +/// +/// Nothing is denied, so the worker behaves as before; each open now runs +/// the Landlock hook, which is what this measures. The production baseline +/// allows each root entry separately instead, which walks the same +/// hierarchy. +fn confine_with_landlock() -> anyhow::Result<()> { + use landlock::{ + ABI, Access as _, AccessFs, PathBeneath, PathFd, Ruleset, RulesetAttr as _, + RulesetCreatedAttr as _, RulesetStatus, + }; + let access = AccessFs::from_all(ABI::V3); + let status = Ruleset::default() + .handle_access(access) + .context("handle Landlock filesystem access")? + .create() + .context("create Landlock ruleset")? + .add_rule(PathBeneath::new( + PathFd::new("/").context("open filesystem root")?, + access, + )) + .context("allow the root hierarchy")? + .restrict_self() + .context("enter Landlock domain")?; + if status.ruleset != RulesetStatus::FullyEnforced { + bail!("Landlock ABI v3 is not fully enforced on this kernel"); + } + Ok(()) +} + +/// What the workers of a mixed protocol share: every worker keeps operating +/// until the last one has measured its `iterations`, so each class is +/// measured only while the other is running. +struct MixedRun { + /// Workers that have not yet measured `iterations` operations. + below_quota: AtomicUsize, + stop: AtomicBool, +} + +/// Ends a mixed run when its worker stops for any reason. A worker that +/// failed or panicked never reaches its quota; without this its peers would +/// wait for it forever. +struct StopMixedRun<'a>(&'a MixedRun); + +impl Drop for StopMixedRun<'_> { + fn drop(&mut self) { + self.0.stop.store(true, Ordering::Release); + } +} + +/// Latency samples a mixed worker keeps. It keeps operating, and counting, +/// past this many while a slower class finishes. +const MIXED_SAMPLE_LIMIT: usize = 1_000_000; + +/// What one worker thread measured. +struct Measured { + operations: u64, + samples: Vec, +} + +/// The bytes a read of `payload_bytes` from the benchmark file must return, +/// for runs that check them. +fn expected_read(run: &WorkerRun) -> anyhow::Result>> { + let reference = if let Some(bytes) = run.provider_file_bytes { + // The whole file, against content this worker derives itself. + let expected = synthetic_provider_file(bytes).into_bytes(); + let served = std::fs::read(&run.file_path).context("read provider file")?; + if served != expected { + bail!( + "provider file {} does not hold the synthetic content", + run.file_path.display() + ); + } + expected + } else if run.protocol.is_mixed() { + std::fs::read(&run.file_path).context("read benchmark file")? + } else { + // The original protocols on an ordinary file check nothing. + return Ok(None); + }; + let length = reference.len().min(run.payload_bytes); + Ok(Some(Arc::from(&reference[..length]))) +} + +pub fn run_worker(run: &WorkerRun) -> anyhow::Result> { + let WorkerRun { + protocol, + target, + iterations, + warmup, + concurrency, + payload_bytes, + landlock, + .. + } = *run; + if landlock { + confine_with_landlock()?; + } + let file_path: Arc = Arc::from(run.file_path.as_path()); + let expected = expected_read(run)?; + let classes = protocol.classes(); + let workers = classes.len() * concurrency; + let mixed = protocol.is_mixed().then(|| { + Arc::new(MixedRun { + below_quota: AtomicUsize::new(workers), + stop: AtomicBool::new(false), + }) + }); let preparation_deadline = Instant::now() + START_TIMEOUT; let (sender, receiver) = mpsc::channel(); - let mut threads = Vec::with_capacity(concurrency); - let mut starters = Vec::with_capacity(concurrency); - let mut readiness = Vec::with_capacity(concurrency); - for _ in 0..concurrency { - let sender = sender.clone(); - let (start_tx, start_rx) = mpsc::channel(); - let (ready_tx, ready_rx) = mpsc::channel(); - starters.push(start_tx); - readiness.push(ready_rx); - threads.push(thread::spawn(move || { - let result = worker_loop( - WorkerOptions { - protocol, - target, - iterations, - warmup, - payload_bytes, - preparation_deadline, - }, - start_rx, - ready_tx, - ); - let _ = sender.send(result); - })); + let mut threads = Vec::with_capacity(workers); + let mut starters = Vec::with_capacity(workers); + let mut readiness = Vec::with_capacity(workers); + for (class, class_protocol) in classes.iter().copied().enumerate() { + for _ in 0..concurrency { + let sender = sender.clone(); + let (start_tx, start_rx) = mpsc::channel(); + let (ready_tx, ready_rx) = mpsc::channel(); + starters.push(start_tx); + readiness.push(ready_rx); + let options = WorkerOptions { + protocol: class_protocol, + target, + iterations, + warmup, + payload_bytes, + preparation_deadline, + file_path: Arc::clone(&file_path), + expected: expected.clone(), + mixed: mixed.clone(), + }; + threads.push(thread::spawn(move || { + let result = worker_loop(&options, start_rx, ready_tx); + let _ = sender.send((class, result)); + })); + } } drop(sender); + let join = |threads: Vec>| { + threads.into_iter().try_for_each(|worker| { + worker + .join() + .map_err(|_| anyhow::anyhow!("worker panicked")) + }) + }; let preparation = readiness.into_iter().try_for_each(|ready| { ready .recv_timeout(preparation_deadline.saturating_duration_since(Instant::now())) @@ -236,11 +626,7 @@ pub fn run_worker( // Releasing the senders wakes workers that completed preparation while // a peer failed, so every thread can be joined without hanging. drop(starters); - for worker in threads { - worker - .join() - .map_err(|_| anyhow::anyhow!("worker panicked"))?; - } + join(threads)?; return Err(error); } let started = Instant::now(); @@ -249,38 +635,71 @@ pub fn run_worker( .send(()) .context("benchmark worker exited before measurement")?; } - let mut samples = Vec::new(); - for result in receiver { - samples.extend(result?); + // The channel closes once every worker has reported or died. A worker + // that fails ends a mixed run for its peers, so this never waits on a + // worker that cannot finish. + let outcomes = receiver.into_iter().collect::>(); + let seconds = started.elapsed().as_secs_f64(); + join(threads)?; + let mut measured = classes + .iter() + .map(|_| Measured { + operations: 0, + samples: Vec::new(), + }) + .collect::>(); + for (class, outcome) in outcomes { + let worker = outcome?; + measured[class].operations += worker.operations; + measured[class].samples.extend(worker.samples); } - let elapsed = started.elapsed(); - for worker in threads { - worker - .join() - .map_err(|_| anyhow::anyhow!("worker panicked"))?; - } - samples.sort_unstable(); - let operations = iterations.saturating_mul(concurrency as u64); - let seconds = elapsed.as_secs_f64(); - Ok(BenchmarkReport { + Ok(classes + .iter() + .zip(measured) + .map(|(class, measured)| class_report(run, *class, measured, seconds)) + .collect()) +} + +/// The report of one class of operation: the whole run, or one class of a +/// mixed protocol. +#[allow(clippy::cast_precision_loss)] +fn class_report( + run: &WorkerRun, + class: Protocol, + mut measured: Measured, + seconds: f64, +) -> BenchmarkReport { + measured.samples.sort_unstable(); + let samples = &measured.samples; + let mixed = run.protocol.is_mixed(); + BenchmarkReport { layer: Layer::Native, - protocol, - iterations: operations, - concurrency, - payload_bytes, - capability_scope: protocol.capability_scope().to_string(), + protocol: run.protocol, + iterations: measured.operations, + concurrency: run.concurrency, + payload_bytes: run.payload_bytes, + capability_scope: class.capability_scope().to_string(), elapsed_ms: seconds * 1_000.0, - operations_per_second: operations as f64 / seconds, + operations_per_second: measured.operations as f64 / seconds, throughput_mbit_per_second: throughput_mbit_per_second( - protocol, - operations, - payload_bytes, + class, + measured.operations, + run.payload_bytes, seconds, ), - latency_ns_p50: percentile(&samples, 50), - latency_ns_p95: percentile(&samples, 95), - latency_ns_p99: percentile(&samples, 99), - }) + latency_ns_p50: percentile(samples, 50), + latency_ns_p95: percentile(samples, 95), + latency_ns_p99: percentile(samples, 99), + latency_ns_max: samples.last().copied().unwrap_or(0), + landlock: run.landlock, + task_memory_reads: None, + sync_wakeup: false, + notify_receivers: None, + file_path: class.is_file().then(|| run.file_path.clone()), + provider_file_bytes: run.provider_file_bytes.filter(|_| class.is_file()), + class: mixed.then_some(class), + class_workers: mixed.then_some(run.concurrency), + } } /// Report application bytes transferred by the measured operation. A TCP @@ -294,8 +713,13 @@ fn throughput_mbit_per_second( seconds: f64, ) -> f64 { let bytes_per_operation = match protocol { - Protocol::TcpConnect => 0, + // A mixed protocol is reported per class, never as a whole. + Protocol::TcpConnect + | Protocol::FileOpenClose + | Protocol::MixedFileTcpConnect + | Protocol::MixedFileTcpStream => 0, Protocol::TcpStream => payload_bytes.saturating_mul(2), + Protocol::FileOpenRead | Protocol::FileRead => payload_bytes, }; operations as f64 * bytes_per_operation as f64 * 8.0 / seconds / 1_000_000.0 } @@ -305,11 +729,42 @@ impl Protocol { match self { Self::TcpConnect => "implemented local TCP socket/connect interception", Self::TcpStream => "implemented established TCP fast path", + Self::FileOpenClose => "read-only file open and close", + Self::FileOpenRead => "read-only file open, one read, and close", + Self::FileRead => "read on an already-open descriptor; no open is measured", + Self::MixedFileTcpConnect | Self::MixedFileTcpStream => { + "file opens and TCP operations at the same time" + } + } + } + + const fn is_mixed(self) -> bool { + matches!(self, Self::MixedFileTcpConnect | Self::MixedFileTcpStream) + } + + /// Whether the protocol opens the benchmark file. + const fn opens_file(self) -> bool { + self.is_file() || self.is_mixed() + } + + /// The operations worker threads run: the protocol itself, or the two + /// classes of a mixed protocol, each on `concurrency` threads. + fn classes(self) -> Vec { + match self { + Self::MixedFileTcpConnect => vec![Self::FileOpenRead, Self::TcpConnect], + Self::MixedFileTcpStream => vec![Self::FileOpenRead, Self::TcpStream], + single => vec![single], } } + + const fn is_file(self) -> bool { + matches!( + self, + Self::FileOpenClose | Self::FileOpenRead | Self::FileRead + ) + } } -#[derive(Clone, Copy)] struct WorkerOptions { protocol: Protocol, target: SocketAddr, @@ -317,13 +772,31 @@ struct WorkerOptions { warmup: u64, payload_bytes: usize, preparation_deadline: Instant, + file_path: Arc, + /// What a read of the benchmark file must return, when checked. + expected: Option>, + /// Present for a class of a mixed protocol. + mixed: Option>, +} + +/// What one worker thread keeps across operations. +struct Prepared { + payload: Vec, + response: Vec, + tcp: Option, + /// Opened before measurement for [`Protocol::FileRead`]. + file: Option, + /// What a file read must return, when checked. + expected: Option>, + /// Whether a TCP echo must equal its request. + check_echo: bool, } fn worker_loop( - options: WorkerOptions, + options: &WorkerOptions, start: mpsc::Receiver<()>, ready: mpsc::Sender>, -) -> anyhow::Result> { +) -> anyhow::Result { let WorkerOptions { protocol, target, @@ -331,24 +804,36 @@ fn worker_loop( warmup, payload_bytes, preparation_deadline, - } = options; + ref file_path, + ref expected, + ref mixed, + } = *options; let prepared = (|| -> anyhow::Result<_> { - let payload = vec![0x5a; payload_bytes]; - let mut tcp = if matches!(protocol, Protocol::TcpStream) { - Some(connect(target)?) - } else { - None + let mut prepared = Prepared { + payload: vec![0x5a; payload_bytes], + response: vec![0_u8; payload_bytes], + tcp: if matches!(protocol, Protocol::TcpStream) { + Some(connect(target)?) + } else { + None + }, + file: if matches!(protocol, Protocol::FileRead) { + Some(File::open(file_path).context("open benchmark file")?) + } else { + None + }, + expected: expected.clone(), + check_echo: mixed.is_some(), }; - let mut response = vec![0_u8; payload_bytes]; for _ in 0..warmup { if Instant::now() >= preparation_deadline { bail!("benchmark warmup exceeded preparation deadline"); } - one_operation(protocol, target, &payload, &mut response, tcp.as_mut())?; + one_operation(protocol, target, file_path, &mut prepared)?; } - Ok((payload, response, tcp)) + Ok(prepared) })(); - let (payload, mut response, mut tcp) = match prepared { + let mut prepared = match prepared { Ok(prepared) => { ready .send(Ok(())) @@ -363,30 +848,101 @@ fn worker_loop( }; start.recv().context("benchmark measurement cancelled")?; let mut samples = Vec::with_capacity(usize::try_from(iterations).unwrap_or(0)); - for _ in 0..iterations { + let Some(mixed) = mixed else { + for _ in 0..iterations { + let started = Instant::now(); + one_operation(protocol, target, file_path, &mut prepared)?; + samples.push(u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX)); + } + return Ok(Measured { + operations: iterations, + samples, + }); + }; + // Keep operating until every worker of both classes has its quota, so + // no class is measured after the other went idle. + let _stop = StopMixedRun(mixed); + let mut operations = 0_u64; + while !mixed.stop.load(Ordering::Acquire) { let started = Instant::now(); - one_operation(protocol, target, &payload, &mut response, tcp.as_mut())?; - samples.push(u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX)); + one_operation(protocol, target, file_path, &mut prepared)?; + if samples.len() < MIXED_SAMPLE_LIMIT { + samples.push(u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX)); + } + operations += 1; + if operations == iterations && mixed.below_quota.fetch_sub(1, Ordering::AcqRel) == 1 { + mixed.stop.store(true, Ordering::Release); + } } - Ok(samples) + Ok(Measured { + operations, + samples, + }) } fn one_operation( protocol: Protocol, target: SocketAddr, - payload: &[u8], - response: &mut [u8], - tcp: Option<&mut TcpStream>, + file_path: &Path, + prepared: &mut Prepared, ) -> io::Result<()> { match protocol { Protocol::TcpConnect => { connect(target)?; } Protocol::TcpStream => { - let stream = tcp.expect("TCP stream initialized"); - stream.write_all(payload)?; - stream.read_exact(response)?; + let stream = prepared.tcp.as_mut().expect("TCP stream initialized"); + stream.write_all(&prepared.payload)?; + stream.read_exact(&mut prepared.response)?; + if prepared.check_echo && prepared.response != prepared.payload { + return Err(content_mismatch("TCP echo differs from its request")); + } + } + Protocol::FileOpenClose => { + drop(File::open(file_path)?); + } + Protocol::FileOpenRead => { + let mut file = File::open(file_path)?; + let read = file.read(&mut prepared.response)?; + require_data(read)?; + require_expected(&prepared.response[..read], prepared.expected.as_deref())?; } + Protocol::FileRead => { + let file = prepared.file.as_ref().expect("benchmark file opened"); + let read = file.read_at(&mut prepared.response, 0)?; + require_data(read)?; + require_expected(&prepared.response[..read], prepared.expected.as_deref())?; + } + Protocol::MixedFileTcpConnect | Protocol::MixedFileTcpStream => { + unreachable!("a mixed protocol runs as its classes") + } + } + Ok(()) +} + +fn content_mismatch(message: &str) -> io::Error { + io::Error::new(io::ErrorKind::InvalidData, message) +} + +/// Fail a read whose bytes are not the ones the run established, where it +/// established any. +fn require_expected(read: &[u8], expected: Option<&[u8]>) -> io::Result<()> { + match expected { + Some(expected) if read != expected => Err(content_mismatch( + "benchmark file content changed or is not the expected file", + )), + _ => Ok(()), + } +} + +/// A file shorter than the payload still yields a valid operation; an empty +/// one measures nothing. +fn require_data(read: usize) -> io::Result<()> { + if read == 0 { + return Err(io::Error::new( + io::ErrorKind::UnexpectedEof, + "benchmark file is empty", + )); } Ok(()) } @@ -439,6 +995,222 @@ mod tests { let address = listener.local_addr().expect("test listener address"); drop(listener); - assert!(run_worker(Protocol::TcpStream, address, 1, 1, 2, 8).is_err()); + assert!(run_worker(&worker_run(Protocol::TcpStream, address)).is_err()); + } + + fn worker_run(protocol: Protocol, target: SocketAddr) -> WorkerRun { + WorkerRun { + protocol, + target, + iterations: 1, + warmup: 1, + concurrency: 2, + payload_bytes: 8, + landlock: false, + file_path: PathBuf::from("/etc/passwd"), + provider_file_bytes: None, + } + } + + /// The single report of a protocol that is not mixed. + fn one_report(run: &WorkerRun) -> BenchmarkReport { + let mut reports = run_worker(run).unwrap(); + assert_eq!(reports.len(), 1); + reports.pop().unwrap() + } + + fn options( + layers: &str, + protocols: &str, + provider_file_bytes: Option, + ) -> BenchmarkOptions { + BenchmarkOptions { + layers: Layer::selection(layers).unwrap(), + protocols: Protocol::selection(protocols).unwrap(), + iterations: 1, + warmup: 0, + concurrency: 1, + payload_bytes: 64, + landlock: false, + file_path: PathBuf::from("/etc/passwd"), + sync_wakeup: false, + notify_receivers: 1, + provider_file_bytes, + } + } + + #[test] + fn provider_file_runs_refuse_what_cannot_serve_or_open_one() { + for (layers, protocols, bytes) in [ + // Only the filtered layer injects a provider file. + ("native", "file", 4_096), + ("all", "file", 4_096), + ("attribution", "file", 4_096), + ("notify-only", "file-open-read", 4_096), + // Only file operations open one. + ("filtered", "all", 4_096), + ("filtered", "tcp-connect", 4_096), + // The broker's own bounds. + ("filtered", "file", 0), + ("filtered", "file", 65_537), + ] { + let error = run(options(layers, protocols, Some(bytes))) + .expect_err("an incompatible provider-file run must be refused"); + assert!( + error.to_string().contains("provider-file"), + "{layers} {protocols} {bytes}: {error}" + ); + } + } + + #[test] + fn provider_file_workers_verify_what_they_read() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("config.toml"); + std::fs::write(&path, synthetic_provider_file(256)).unwrap(); + for protocol in [Protocol::FileOpenRead, Protocol::FileRead] { + let run = WorkerRun { + provider_file_bytes: Some(256), + ..file_run(protocol, &path) + }; + let report = one_report(&run); + assert_eq!(report.provider_file_bytes, Some(256)); + assert_eq!(report.file_path.as_deref(), Some(path.as_path())); + // Any other content is an error, not a measurement. + let wrong = WorkerRun { + provider_file_bytes: Some(255), + ..run + }; + assert!(run_worker(&wrong).is_err()); + } + } + + #[test] + fn mixed_protocols_report_each_class_measured_together() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("data"); + std::fs::write(&path, b"0123456789abcdef").unwrap(); + for (protocol, tcp_class, echo) in [ + (Protocol::MixedFileTcpConnect, Protocol::TcpConnect, false), + (Protocol::MixedFileTcpStream, Protocol::TcpStream, true), + ] { + let fixture = start_tcp_fixture(echo).unwrap(); + let run = WorkerRun { + iterations: 32, + warmup: 2, + concurrency: 2, + file_path: path.clone(), + ..worker_run(protocol, fixture.target) + }; + let reports = run_worker(&run).unwrap(); + assert_eq!(reports.len(), 2, "{protocol:?}"); + let classes = [Protocol::FileOpenRead, tcp_class]; + for (report, class) in reports.iter().zip(classes) { + assert_eq!( + serde_json::to_value(report.protocol).unwrap(), + serde_json::to_value(protocol).unwrap() + ); + assert_eq!( + serde_json::to_value(report.class).unwrap(), + serde_json::to_value(class).unwrap() + ); + assert_eq!(report.class_workers, Some(2)); + // Every worker of the class met its quota; a class that + // finished first kept running until the other did. + assert!(report.iterations >= 64, "{class:?}: {}", report.iterations); + assert!(report.operations_per_second > 0.0); + assert!(report.latency_ns_p50 > 0, "{class:?}"); + assert!(report.latency_ns_p50 <= report.latency_ns_p99); + assert!(report.latency_ns_p99 <= report.latency_ns_max); + assert_eq!(report.file_path.is_some(), class.is_file()); + } + // Both classes were measured over the same window. + assert!((reports[0].elapsed_ms - reports[1].elapsed_ms).abs() < f64::EPSILON); + } + } + + #[test] + fn a_failing_class_ends_a_mixed_run_instead_of_hanging_it() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("data"); + std::fs::write(&path, b"0123456789abcdef").unwrap(); + let listener = TcpListener::bind("127.0.0.1:0").expect("bind test listener"); + let closed = listener.local_addr().expect("test listener address"); + drop(listener); + // No warmup: the connects fail during measurement, while the file + // workers, which could never be told to stop otherwise, are running. + let run = WorkerRun { + iterations: u64::MAX, + warmup: 0, + concurrency: 2, + file_path: path, + ..worker_run(Protocol::MixedFileTcpConnect, closed) + }; + assert!(run_worker(&run).is_err()); + + // A worker that panics ends the run the same way. + let mixed = Arc::new(MixedRun { + below_quota: AtomicUsize::new(2), + stop: AtomicBool::new(false), + }); + let panicking = { + let mixed = Arc::clone(&mixed); + thread::spawn(move || { + let _stop = StopMixedRun(&mixed); + panic!("worker failure"); + }) + }; + assert!(panicking.join().is_err()); + assert!(mixed.stop.load(Ordering::Acquire)); + } + + fn file_run(protocol: Protocol, path: &Path) -> WorkerRun { + WorkerRun { + iterations: 64, + file_path: path.to_path_buf(), + ..worker_run( + protocol, + SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0), + ) + } + } + + #[test] + fn file_protocols_measure_every_requested_operation() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("data"); + std::fs::write(&path, b"0123456789abcdef").unwrap(); + for protocol in [ + Protocol::FileOpenClose, + Protocol::FileOpenRead, + Protocol::FileRead, + ] { + let report = one_report(&file_run(protocol, &path)); + // Two threads, 64 iterations each. + assert_eq!(report.iterations, 128, "{protocol:?}"); + assert!(report.latency_ns_p50 > 0, "{protocol:?}"); + assert!(report.latency_ns_p50 <= report.latency_ns_p99); + assert!(report.latency_ns_p99 <= report.latency_ns_max); + assert!(!report.landlock); + } + // Only a read transfers payload. + let open_only = throughput_mbit_per_second(Protocol::FileOpenClose, 1_000, 64, 1.0); + assert!(open_only.abs() < f64::EPSILON); + assert!(throughput_mbit_per_second(Protocol::FileRead, 1_000, 64, 1.0) > 0.0); + + // A missing file fails every operation; an empty one fails the reads. + let missing = directory.path().join("missing"); + let empty = directory.path().join("empty"); + std::fs::write(&empty, b"").unwrap(); + for protocol in [ + Protocol::FileOpenClose, + Protocol::FileOpenRead, + Protocol::FileRead, + ] { + assert!(run_worker(&file_run(protocol, &missing)).is_err()); + } + for protocol in [Protocol::FileOpenRead, Protocol::FileRead] { + assert!(run_worker(&file_run(protocol, &empty)).is_err()); + } } } diff --git a/crates/openshell-sandbox/src/provider_files.rs b/crates/openshell-sandbox/src/provider_files.rs index e7687ff379..376b210077 100644 --- a/crates/openshell-sandbox/src/provider_files.rs +++ b/crates/openshell-sandbox/src/provider_files.rs @@ -1,14 +1,25 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Read-only provider files served on demand through seccomp FD injection. +//! Read-only provider files. +//! +//! Every workload file open notifies the broker, which serves a path beneath +//! the provider prefix by injecting a read-only descriptor for a sealed +//! in-memory file and continues every other open. +//! +//! Two things are kept apart. A [`Revision`] is one immutable version of one +//! file, and a [`Snapshot`] is the complete set a publication replaces at +//! once; neither knows how bytes reach the workload. What a revision prepares +//! at publication is its backing, a [`SealedMemfd`], created once when the +//! revision is published and reopened for each workload open. use std::collections::HashMap; -use std::fs::{File, Permissions}; -use std::io::{self, Seek as _, SeekFrom, Write as _}; -use std::os::fd::AsRawFd as _; -use std::os::unix::fs::PermissionsExt as _; -use std::sync::{Arc, RwLock}; +use std::fs::File; +use std::io::{self, Write as _}; +use std::os::fd::{AsFd, AsRawFd as _, OwnedFd}; +use std::sync::{Arc, OnceLock, RwLock}; + +use rustix::fs::{Mode, OFlags, SealFlags}; use crate::linux::seccomp_notify::{Notification, NotificationListener}; use crate::linux::task_memory; @@ -16,16 +27,204 @@ use crate::linux::task_memory; const PREFIX: &str = "/run/openshell/providers/"; const PROC_PREFIX: &str = "/proc/"; const MAX_FILE_BYTES: usize = 65_536; -const MAX_TOTAL_BYTES: usize = 262_144; +pub const MAX_TOTAL_BYTES: usize = 262_144; const MAX_PATH_BYTES: usize = 4_096; -type Snapshot = HashMap>; +/// The mode a provider file reports. It keeps the file's metadata private +/// to the sandbox identity; the seals and the read-only descriptor protect +/// its content. +const FILE_MODE: Mode = Mode::from_bits_truncate(0o600); + +/// One immutable version of one provider file: its bytes, and what was +/// prepared at publication to serve them. +struct Revision { + content: Box<[u8]>, + backing: Backing, +} + +/// A complete generation of provider files, by path. +type Snapshot = HashMap>>; + +/// Build the generation that holds exactly `desired`. +/// +/// A path whose bytes are unchanged keeps its revision, and with it the +/// backing every earlier open of that path used. Every other file gets a new +/// revision from `prepare`. Nothing is published here: if `prepare` fails, +/// the revisions already prepared are dropped and `current` is untouched. +fn next_snapshot( + current: &Snapshot, + desired: HashMap, + mut prepare: impl FnMut(&[u8]) -> io::Result, +) -> io::Result> { + desired + .into_iter() + .map(|(path, content)| { + let revision = match current.get(&path) { + Some(revision) if *revision.content == *content.as_bytes() => Arc::clone(revision), + _ => Arc::new(Revision { + backing: prepare(content.as_bytes())?, + content: content.into_bytes().into_boxed_slice(), + }), + }; + Ok((path, revision)) + }) + .collect() +} + +/// This process's own descriptor table in procfs. +/// +/// Opening a descriptor's entry here yields a new open file description of +/// the same file. The directory is opened once and verified to be procfs, so +/// a later reopen walks no path. +struct OwnDescriptors(OwnedFd); + +impl OwnDescriptors { + fn open() -> io::Result { + let directory = rustix::fs::open( + "/proc/self/fd", + OFlags::PATH | OFlags::DIRECTORY | OFlags::CLOEXEC, + Mode::empty(), + )?; + if rustix::fs::fstatfs(&directory)?.f_type != rustix::fs::PROC_SUPER_MAGIC { + return Err(io::Error::other("/proc/self/fd is not on procfs")); + } + Ok(Self(directory)) + } + + /// A new read-only open file description of the file `descriptor` refers + /// to, positioned at its start. Duplicating `descriptor` instead would + /// share one file offset between every holder. + fn reopen_read_only(&self, descriptor: impl AsFd) -> io::Result { + Ok(rustix::fs::openat( + &self.0, + rustix::path::DecInt::new(descriptor.as_fd().as_raw_fd()), + OFlags::RDONLY | OFlags::CLOEXEC | OFlags::NOCTTY, + Mode::empty(), + )?) + } +} + +/// The sealed in-memory file behind one revision. +/// +/// The content is written once and sealed against writes, resizing, and +/// further sealing, so no descriptor of the file, however it was opened, can +/// change it. The only descriptor kept here is itself read-only: the +/// writable one `memfd_create` returns is closed before the revision is +/// published. +struct SealedMemfd { + anchor: OwnedFd, + descriptors: Arc, +} + +impl SealedMemfd { + fn create(content: &[u8], descriptors: &Arc) -> io::Result { + let writable = rustix::fs::memfd_create( + "openshell-provider", + rustix::fs::MemfdFlags::CLOEXEC | rustix::fs::MemfdFlags::ALLOW_SEALING, + )?; + rustix::fs::fchmod(&writable, FILE_MODE)?; + let mut writable = File::from(writable); + writable.write_all(content)?; + rustix::fs::fcntl_add_seals( + &writable, + SealFlags::SEAL | SealFlags::WRITE | SealFlags::GROW | SealFlags::SHRINK, + )?; + let anchor = descriptors.reopen_read_only(&writable)?; + Ok(Self { + anchor, + descriptors: Arc::clone(descriptors), + }) + } + + /// A descriptor for one workload open: its own read-only open file + /// description, positioned at the start of the file. + fn open(&self) -> io::Result { + match self.descriptors.reopen_read_only(&self.anchor) { + // Every open of a revision shares one file, which the workload + // owns as well and may have made unreadable through a descriptor + // it holds. Its mode is not what protects the content, so + // restore it rather than fail opens until the next revision. + Err(error) if error.raw_os_error() == Some(libc::EACCES) => { + rustix::fs::fchmod(&self.anchor, FILE_MODE)?; + self.descriptors.reopen_read_only(&self.anchor) + } + opened => opened, + } + } +} -/// A complete provider-file generation. Open handlers clone the selected -/// content before dropping the lock, so updates never block on a child open. +/// The published generation of provider files. +#[derive(Default)] +struct VirtualFiles { + /// Replaced as a whole. Open handlers clone the selected revision before + /// dropping the lock, so an update never waits on a workload open and a + /// revision outlives its generation for as long as an open uses it. + current: RwLock>>, + /// Opened by the first publication that needs it. + descriptors: OnceLock>, +} + +impl VirtualFiles { + fn descriptors(&self) -> io::Result<&Arc> { + if let Some(descriptors) = self.descriptors.get() { + return Ok(descriptors); + } + let opened = Arc::new(OwnDescriptors::open()?); + Ok(self.descriptors.get_or_init(|| opened)) + } + + fn snapshot(&self) -> Arc> { + Arc::clone( + &self + .current + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner), + ) + } + + fn revision(&self, path: &str) -> Option>> { + self.current + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .get(path) + .cloned() + } + + fn publish(&self, desired: HashMap) -> io::Result<()> { + self.publish_with(desired, |content| { + SealedMemfd::create(content, self.descriptors()?) + }) + } + + /// Replace the generation with exactly `desired`, or leave it untouched. + /// + /// Every new revision is prepared before the publication lock is taken, + /// which is then held only to exchange the generation. Concurrent + /// publications each publish a complete generation, and the last to + /// take the lock is the one new opens see. + fn publish_with( + &self, + desired: HashMap, + prepare: impl FnMut(&[u8]) -> io::Result, + ) -> io::Result<()> { + let next = Arc::new(next_snapshot(&self.snapshot(), desired, prepare)?); + let replaced = std::mem::replace( + &mut *self + .current + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner), + next, + ); + // Closes the backing of revisions no open still uses, after the lock. + drop(replaced); + Ok(()) + } +} + +/// The provider files of one sandbox. #[derive(Clone, Default)] pub struct ProviderFiles { - current: Arc>>, + files: Arc, } impl ProviderFiles { @@ -51,15 +250,7 @@ impl ProviderFiles { pub(crate) fn replace(&self, desired: HashMap) -> io::Result<()> { Self::validate(&desired)?; - let next = desired - .into_iter() - .map(|(path, content)| (path, Arc::<[u8]>::from(content.into_bytes()))) - .collect(); - *self - .current - .write() - .unwrap_or_else(std::sync::PoisonError::into_inner) = Arc::new(next); - Ok(()) + self.files.publish(desired) } pub(crate) fn handle_open( @@ -73,15 +264,33 @@ impl ProviderFiles { } else { notification.args[0] }; - if handle_thread_comm_open(listener, notification, path_address)? { - return Ok(()); + // Every workload open reaches the listener, and nearly all of them + // are ordinary paths the kernel decides. One exact copy of the + // path's first six bytes tells those apart from the two kinds of path + // served here; an ordinary open costs nothing else, and in + // particular no read of an `openat2` argument structure. + // + // Six bytes are enough. `/proc/` is six bytes, and every provider + // path starts with the first six bytes of its prefix and is longer + // than that, so both kinds always have six readable bytes. A copy + // that fails is an unreadable pointer or a shorter path that ends + // before an unmapped page: neither is served here, and the kernel + // resolves the path or reports its own EFAULT. + // + // The copy only classifies. Both served kinds read the whole path + // again and decide from that, and neither is ever continued: the + // kernel and Landlock authorize every open that is. + let mut start = [0_u8; PROC_PREFIX.len()]; + if task_memory::read_exact(notification.tid, path_address, &mut start).is_err() { + return listener.respond_continue(notification.id); } - // Every workload open reaches the listener. Copy only the reserved - // prefix for ordinary paths; full path reads are rare. - let mut prefix = [0_u8; PREFIX.len()]; - if task_memory::read_exact(notification.tid, path_address, &mut prefix).is_err() - || prefix != PREFIX.as_bytes() - { + if start == PROC_PREFIX.as_bytes() { + if handle_thread_comm_open(listener, notification, path_address)? { + return Ok(()); + } + return listener.respond_continue(notification.id); + } + if start != PREFIX.as_bytes()[..PROC_PREFIX.len()] { return listener.respond_continue(notification.id); } // A failed or non-absolute lookup is left to the kernel. In particular, @@ -92,13 +301,7 @@ impl ProviderFiles { if !path.starts_with(PREFIX) { return listener.respond_continue(notification.id); } - let content = self - .current - .read() - .unwrap_or_else(std::sync::PoisonError::into_inner) - .get(&path) - .cloned(); - let Some(content) = content else { + let Some(revision) = self.files.revision(&path) else { return listener.respond_errno(notification.id, libc::ENOENT); }; let flags = match open_flags(¬ification) { @@ -120,7 +323,7 @@ impl ProviderFiles { if flags & (libc::O_DIRECTORY | libc::O_PATH | libc::O_DIRECT) != 0 { return listener.respond_errno(notification.id, libc::EINVAL); } - let file = sealed_memfd(&content)?; + let file = revision.backing.open()?; listener.add_fd_and_send( notification.id, file.as_raw_fd(), @@ -170,6 +373,10 @@ fn validate_path(path: &str) -> io::Result<()> { /// the target's own thread group, so a substituted path or reused thread ID /// cannot rename another process's thread through the descriptor. Returns /// `false` when the open is not such a request and normal mediation applies. +/// +/// Called only for a path whose first six bytes were seen to be `/proc/`. +/// That is a hint, not authority: the path is read whole here and decided +/// from that copy alone. fn handle_thread_comm_open( listener: &NotificationListener, notification: Notification, @@ -183,12 +390,6 @@ fn handle_thread_comm_open( if access == libc::O_RDONLY { return Ok(false); } - let mut prefix = [0_u8; PROC_PREFIX.len()]; - if task_memory::read_exact(notification.tid, path_address, &mut prefix).is_err() - || prefix != PROC_PREFIX.as_bytes() - { - return Ok(false); - } let Ok(path) = read_path(notification.tid, path_address) else { return Ok(false); }; @@ -296,34 +497,52 @@ fn open_flags(notification: &Notification) -> io::Result { } } -fn sealed_memfd(content: &[u8]) -> io::Result { - let fd = rustix::fs::memfd_create( - "openshell-provider", - rustix::fs::MemfdFlags::CLOEXEC | rustix::fs::MemfdFlags::ALLOW_SEALING, - )?; - let mut file = File::from(fd); - // Keep metadata private as well as the returned descriptor read-only. - file.set_permissions(Permissions::from_mode(0o600))?; - file.write_all(content)?; - file.seek(SeekFrom::Start(0))?; - rustix::fs::fcntl_add_seals( - &file, - rustix::fs::SealFlags::SEAL - | rustix::fs::SealFlags::WRITE - | rustix::fs::SealFlags::GROW - | rustix::fs::SealFlags::SHRINK, - )?; - // memfd_create returns O_RDWR. Reopen the sealed object through our own - // procfs descriptor so the child receives an actual O_RDONLY description. - File::open(format!("/proc/self/fd/{}", file.as_raw_fd())) -} - #[cfg(test)] mod tests { - use super::{ProviderFiles, comm_target, sealed_memfd}; + use super::{MAX_FILE_BYTES, ProviderFiles, SealedMemfd, VirtualFiles, comm_target}; use std::collections::HashMap; - use std::io::Read as _; - use std::os::unix::fs::PermissionsExt as _; + use std::fs::File; + use std::io::{self, Read as _, Seek as _, SeekFrom}; + use std::os::fd::AsRawFd as _; + use std::os::unix::fs::{FileExt as _, MetadataExt as _, PermissionsExt as _}; + use std::sync::Arc; + use std::sync::atomic::{AtomicBool, Ordering}; + + const CONFIG: &str = "/run/openshell/providers/acme/client.toml"; + const KEY: &str = "/run/openshell/providers/acme/key.pem"; + const TOKEN: &str = "/run/openshell/providers/other/token"; + + fn files(entries: &[(&str, &str)]) -> HashMap { + entries + .iter() + .map(|(path, content)| ((*path).to_string(), (*content).to_string())) + .collect() + } + + fn virtual_files(provider_files: &ProviderFiles) -> &VirtualFiles { + &provider_files.files + } + + /// What the broker injects for a workload open of `path`, or `None` + /// where it answers `ENOENT`. + fn open(provider_files: &ProviderFiles, path: &str) -> Option { + let revision = virtual_files(provider_files).revision(path)?; + Some(File::from(revision.backing.open().expect("open revision"))) + } + + fn read(provider_files: &ProviderFiles, path: &str) -> Option { + let mut content = String::new(); + open(provider_files, path)? + .read_to_string(&mut content) + .unwrap(); + Some(content) + } + + /// The identity of the file a descriptor refers to. + fn file_identity(file: &File) -> (u64, u64) { + let metadata = file.metadata().unwrap(); + (metadata.dev(), metadata.ino()) + } #[test] fn comm_target_accepts_only_the_callers_own_thread_names() { @@ -363,17 +582,303 @@ mod tests { } #[test] - fn memfd_is_read_only_and_positioned_at_start() { - let mut file = sealed_memfd(b"version = 1\n").unwrap(); - assert_eq!(file.metadata().unwrap().permissions().mode() & 0o777, 0o600); - let flags = rustix::fs::fcntl_getfl(&file).unwrap(); + fn each_open_is_its_own_read_only_description_of_one_sealed_file() { + let provider_files = ProviderFiles::default(); + provider_files + .replace(files(&[(CONFIG, "0123456789")])) + .unwrap(); + let mut first = open(&provider_files, CONFIG).unwrap(); + let mut second = open(&provider_files, CONFIG).unwrap(); + + // Both opens are positioned at the start, and neither moves the + // other: a duplicated descriptor would share one offset. + let mut head = [0_u8; 4]; + first.read_exact(&mut head).unwrap(); + assert_eq!(&head, b"0123"); + let mut whole = String::new(); + second.read_to_string(&mut whole).unwrap(); + assert_eq!(whole, "0123456789"); + let mut rest = String::new(); + first.read_to_string(&mut rest).unwrap(); + assert_eq!(rest, "456789"); + second.seek(SeekFrom::Start(8)).unwrap(); + assert_eq!(first.stream_position().unwrap(), 10); + + // One file serves every open of a revision. + assert_eq!(file_identity(&first), file_identity(&second)); + for file in [&first, &second] { + assert_eq!(file.metadata().unwrap().mode() & 0o7777, 0o600); + assert_eq!( + rustix::fs::fcntl_getfl(file).unwrap() & rustix::fs::OFlags::ACCMODE, + rustix::fs::OFlags::RDONLY + ); + let seals = rustix::fs::fcntl_get_seals(file).unwrap(); + assert!(seals.contains( + rustix::fs::SealFlags::SEAL + | rustix::fs::SealFlags::WRITE + | rustix::fs::SealFlags::GROW + | rustix::fs::SealFlags::SHRINK + )); + assert!(file.write_at(b"changed", 0).is_err()); + assert!(file.set_len(0).is_err()); + } + + // The workload owns the file and can reopen its descriptor for + // writing. The seals, not the descriptor's access mode, are what + // keep the content, and they cannot be lifted. + let link = format!("/proc/self/fd/{}", first.as_raw_fd()); + for options in [ + std::fs::OpenOptions::new().write(true).clone(), + std::fs::OpenOptions::new().read(true).write(true).clone(), + std::fs::OpenOptions::new().append(true).clone(), + ] { + if let Ok(writable) = options.open(&link) { + assert!(writable.write_at(b"changed", 0).is_err()); + assert!(io::Write::write_all(&mut &writable, b"changed").is_err()); + assert!(writable.set_len(0).is_err()); + assert!(writable.set_len(1 << 20).is_err()); + } + } + assert!( + std::fs::OpenOptions::new() + .write(true) + .truncate(true) + .open(&link) + .is_err() + ); + assert_eq!(read(&provider_files, CONFIG).unwrap(), "0123456789"); + + // The descriptor the runtime keeps for the revision is read-only too. + let revision = virtual_files(&provider_files).revision(CONFIG).unwrap(); assert_eq!( - flags & rustix::fs::OFlags::ACCMODE, + rustix::fs::fcntl_getfl(&revision.backing.anchor).unwrap() + & rustix::fs::OFlags::ACCMODE, rustix::fs::OFlags::RDONLY ); - let mut read = String::new(); - file.read_to_string(&mut read).unwrap(); - assert_eq!(read, "version = 1\n"); - assert!(std::io::Write::write_all(&mut file, b"changed").is_err()); + } + + #[test] + fn a_cleared_mode_does_not_deny_later_opens() { + let provider_files = ProviderFiles::default(); + provider_files.replace(files(&[(KEY, "key-1")])).unwrap(); + let held = open(&provider_files, KEY).unwrap(); + // The workload owns the shared file and can change its mode. + held.set_permissions(std::fs::Permissions::from_mode(0o000)) + .unwrap(); + assert_eq!(read(&provider_files, KEY).unwrap(), "key-1"); + // An identical update keeps the revision; it must stay readable. + provider_files.replace(files(&[(KEY, "key-1")])).unwrap(); + assert_eq!(read(&provider_files, KEY).unwrap(), "key-1"); + } + + #[test] + fn updates_and_detach_change_later_opens_and_keep_open_descriptors() { + let provider_files = ProviderFiles::default(); + assert!(open(&provider_files, CONFIG).is_none()); + provider_files + .replace(files(&[(CONFIG, "version = 1\n"), (KEY, "key-1")])) + .unwrap(); + let old_config = open(&provider_files, CONFIG).unwrap(); + let old_key = open(&provider_files, KEY).unwrap(); + + // Change one file, keep one, attach one. + provider_files + .replace(files(&[ + (CONFIG, "version = 2\n"), + (KEY, "key-1"), + (TOKEN, "token"), + ])) + .unwrap(); + let new_config = open(&provider_files, CONFIG).unwrap(); + assert_ne!(file_identity(&new_config), file_identity(&old_config)); + assert_eq!(read(&provider_files, CONFIG).unwrap(), "version = 2\n"); + assert_eq!(read(&provider_files, TOKEN).unwrap(), "token"); + // An unchanged file keeps its revision: the same file as before. + assert_eq!( + file_identity(&open(&provider_files, KEY).unwrap()), + file_identity(&old_key) + ); + + // An identical update changes nothing an open can observe. + provider_files + .replace(files(&[ + (CONFIG, "version = 2\n"), + (KEY, "key-1"), + (TOKEN, "token"), + ])) + .unwrap(); + assert_eq!( + file_identity(&open(&provider_files, CONFIG).unwrap()), + file_identity(&new_config) + ); + assert_eq!( + file_identity(&open(&provider_files, KEY).unwrap()), + file_identity(&old_key) + ); + + // Detach one provider, then everything. + provider_files.replace(files(&[(TOKEN, "token")])).unwrap(); + assert!(open(&provider_files, CONFIG).is_none()); + assert!(open(&provider_files, KEY).is_none()); + assert_eq!(read(&provider_files, TOKEN).unwrap(), "token"); + provider_files.replace(HashMap::new()).unwrap(); + assert!(open(&provider_files, TOKEN).is_none()); + + // Reattaching the same bytes is a new revision, not the detached one. + provider_files.replace(files(&[(KEY, "key-1")])).unwrap(); + assert_ne!( + file_identity(&open(&provider_files, KEY).unwrap()), + file_identity(&old_key) + ); + + // Descriptors opened before any of this still read what they opened. + for (held, expected) in [ + (old_config, "version = 1\n"), + (new_config, "version = 2\n"), + (old_key, "key-1"), + ] { + let mut content = vec![0_u8; expected.len() + 1]; + let length = held.read_at(&mut content, 0).unwrap(); + assert_eq!(&content[..length], expected.as_bytes()); + } + } + + #[test] + fn a_refused_publication_leaves_the_published_generation() { + let provider_files = ProviderFiles::default(); + provider_files + .replace(files(&[(CONFIG, "version = 1\n"), (KEY, "key-1")])) + .unwrap(); + let published = [CONFIG, KEY].map(|path| { + ( + file_identity(&open(&provider_files, path).unwrap()), + read(&provider_files, path).unwrap(), + ) + }); + let unchanged = || { + for (path, (identity, content)) in [CONFIG, KEY].into_iter().zip(&published) { + assert_eq!(&read(&provider_files, path).unwrap(), content); + assert_eq!( + &file_identity(&open(&provider_files, path).unwrap()), + identity + ); + } + assert!(open(&provider_files, TOKEN).is_none()); + }; + + // Input the validation refuses, alongside otherwise valid changes. + let oversized = "x".repeat(MAX_FILE_BYTES + 1); + let mut too_many = files(&[(CONFIG, "version = 2\n")]); + for index in 0..64 { + too_many.insert(format!("/run/openshell/providers/many/{index}"), "x".into()); + } + for refused in [ + files(&[(CONFIG, "version = 2\n"), (TOKEN, &oversized)]), + files(&[(CONFIG, "version = 2\n"), ("/etc/passwd", "x")]), + files(&[ + (CONFIG, "version = 2\n"), + ("/run/openshell/providers/a/../b", "x"), + ]), + too_many, + ] { + let error = provider_files.replace(refused).unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::InvalidInput); + unchanged(); + } + + // A revision that cannot be prepared, after another one was. + let files_state = virtual_files(&provider_files); + let mut prepared = 0; + let error = files_state + .publish_with( + files(&[(CONFIG, "version = 2\n"), (KEY, "key-2"), (TOKEN, "token")]), + |content| { + prepared += 1; + if prepared == 2 { + return Err(io::Error::from_raw_os_error(libc::EMFILE)); + } + SealedMemfd::create(content, files_state.descriptors()?) + }, + ) + .unwrap_err(); + assert_eq!(error.raw_os_error(), Some(libc::EMFILE)); + assert_eq!(prepared, 2); + unchanged(); + + // Publication still works afterwards. + provider_files.replace(files(&[(TOKEN, "token")])).unwrap(); + assert_eq!(read(&provider_files, TOKEN).unwrap(), "token"); + assert!(open(&provider_files, CONFIG).is_none()); + } + + #[test] + fn concurrent_publications_and_opens_observe_only_complete_revisions() { + const PATHS: [&str; 3] = [CONFIG, KEY, TOKEN]; + const GENERATIONS: usize = 150; + // Every file of a generation carries the generation's token on + // every line, so a partial or mixed file is recognizable. + fn generation(token: usize) -> HashMap { + let document = format!("{token:08}\n").repeat(512); + files(&PATHS.map(|path| (path, document.as_str()))) + } + let token_of = |content: &str| -> usize { + let first = content + .lines() + .next() + .expect("a published file is not empty"); + assert_eq!(content.len(), 512 * 9, "partial file"); + assert!(content.lines().all(|line| line == first), "mixed file"); + first.parse().expect("token") + }; + + let provider_files = ProviderFiles::default(); + provider_files.replace(generation(0)).unwrap(); + let stop = Arc::new(AtomicBool::new(false)); + let readers = (0..3) + .map(|_| { + let provider_files = provider_files.clone(); + let stop = Arc::clone(&stop); + std::thread::spawn(move || { + let mut opened = 0_usize; + while !stop.load(Ordering::Acquire) { + let held = PATHS.map(|path| { + let mut file = + open(&provider_files, path).expect("the file is always published"); + let mut content = String::new(); + file.read_to_string(&mut content).unwrap(); + (file, token_of(&content)) + }); + opened += held.len(); + // Descriptors opened earlier keep their revision + // while newer generations are published. + for (file, token) in held { + let mut again = vec![0_u8; 512 * 9]; + file.read_exact_at(&mut again, 0).unwrap(); + assert_eq!(token_of(std::str::from_utf8(&again).unwrap()), token); + } + } + opened + }) + }) + .collect::>(); + let publishers = [1_000, 2_000].map(|base| { + let provider_files = provider_files.clone(); + std::thread::spawn(move || { + for step in 0..GENERATIONS { + provider_files.replace(generation(base + step)).unwrap(); + } + }) + }); + for publisher in publishers { + publisher.join().expect("publisher"); + } + stop.store(true, Ordering::Release); + for reader in readers { + assert!(reader.join().expect("reader") > 0); + } + // What stays published is one publisher's last generation, whole. + let last = PATHS.map(|path| token_of(&read(&provider_files, path).unwrap())); + assert!(last[0] == 1_000 + GENERATIONS - 1 || last[0] == 2_000 + GENERATIONS - 1); + assert_eq!(last, [last[0]; 3]); } } diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index bf9e9e0b83..3f00bf27a3 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -370,6 +370,12 @@ Common findings: generations dial host CID 2 over vsock; libkrun forwards to a protected host Unix socket, while QEMU uses a per-guest host port. Keep supervisor and runtime binaries compatible when creating new generations. +- Provider files are missing or stale in a sandbox: the sandbox runtime + delivers provider files by intercepting workload opens beneath + `/run/openshell/providers` and injecting a sealed in-memory descriptor. The + workload container has no provider-file volume or mount, so a missing mount + is not a fault; debug it as an enforcement-probe or provider-attachment + problem. - Sandbox startup reports an `isolation control contract` failure: inspect the supervisor and sandbox-runtime image versions and update them together. An older runtime or incompatible isolation capability set fails before workload startup. See the published Sandbox Runtimes documentation for trusted image override precedence. - The sandbox fails its enforcement probe: inspect the sandbox log for the exact nested seccomp user-notification, task-memory, Landlock, loopback DNS, or socket-injection check that failed. A runtime may return `ENOSYS` for `process_vm_readv` and `process_vm_writev` while satisfying the production parent-to-workload-child task-memory probe through `/proc//mem`; only failure of both backends is fatal. Do not add capabilities or switch to an unconfined seccomp profile; use a runtime whose default profile permits the unprivileged probe. - A GPU sandbox fails because Docker reports no discovered NVIDIA CDI devices: verify `.DiscoveredDevices` contains entries such as `nvidia.com/gpu=all`, verify `/etc/cdi` or `/var/run/cdi` contains a generated NVIDIA spec, and check that `nvidia-cdi-refresh.service` and `nvidia-cdi-refresh.path` from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so `inactive (dead)` can be normal after a successful run; use `systemctl status` and `journalctl` to distinguish success from a skipped or failed refresh. Restart `nvidia-cdi-refresh.service` to regenerate missing or stale CDI specs, then restart or reload Docker and re-check `docker info`. @@ -1750,7 +1756,6 @@ Shared serving leases last at most fifteen seconds from issuance, independently of configuration renewal; include membership failure detection when measuring a partition's stale-owner interval. Storage failures prevent renewal. - Application ingress consumers call the separate `openshell.v1.ServiceDiscovery` service, rather than `OpenShell`. Discovery snapshots share a one-second cache per workspace (16 cached workspaces, four concurrent collections). The gateway