Repository navigation
Description
Summary
opencode-pty (packages/process-daemon) leaks one zombie process per finished agent shell command, and a long-running server stabilises at 25 Z processes (25 = EXITED_LIMIT in packages/core/src/shell.ts). Every zombie's PPID is the opencode-pty daemon. Only an explicit terminate/eviction reaps them, and the client only evicts when the finished-command history overflows, so a long session never gets back to 0.
This is not the PTY path. PTYs use child.wait() in the wait thread and are reaped correctly (verified 35 create/remove → 0 Z).
Why it happens
service/pipe/unix.rs intentionally keeps pipe children as zombies:
wait_exit()useswaitid(P_PID, …, WEXITED | WNOWAIT)— it observes the exit but does not reap.- The zombie keeps the PID and process-group ID reserved, so a later
killpgcan never hit an unrelated process. - Only
TerminalHandle::shutdown()(→Process::reap()→waitpid) actually reaps.
On the client side (packages/core/src/shell.ts):
EXITED_LIMIT = 25keeps the last 25 finished commands forshell.gethistory/output.evict()only runs whenexitOrder.length > 25.
So: 1 command → 1 Z; commands 2…25 → linearly up to 25; after that it stays pinned at 25 forever (no TTL), and only explicit DELETE /api/shell/:id or a server restart drops them.
Reproduction (aarch64 Android, 2.3.0, isolated opencode serve)
warmup Z=1
+10 normal exits Z=11
+10 non-zero exits Z=21
+5 timeouts (sleep 30, timeout=800) Z=25
+5 cancels (DELETE while running) Z=25
+20 more normal exits Z=25 (stays)
idle 30s Z=25 (no TTL)
stop server Z=0
ps sample:
26386 26380 Z sh <defunct> [sh] <defunct>
26394 26380 Z sh <defunct> [sh] <defunct>
26401 26380 Z sh <defunct> [sh] <defunct>
26380 is the opencode-pty daemon. comm is sh, or sleep when the command was a timeout (sh -c execs the last command).
Deleting finished commands one by one immediately drops the count (5 → 4 → 3 → 2 → 1 → 0), confirming reap works and the only problem is when it is called.
What does not work as a reap condition
kill(-pgid, 0) cannot tell "group is empty" on Linux/Android: even when the group only holds the zombie it returns 0. (macOS returns ESRCH, which is presumably why the current comment says it's fine there.) Verified on device with waitid(WNOWAIT) + kill(-pgid, 0).
SIGCHLD = SIG_IGN / waitpid(-1) would lose the exit code and break Exited events.
Proposed fix (patch below)
After the exit message is emitted, reap the child only when the process group has no live (non-zombie) members:
- Linux/Android: scan
/proc/*/statforpgrp == pid && state != 'Z', excluding the child itself. If/proccannot be read, return "unknown" and keep the zombie. - macOS:
killpg(pid, 0)→ESRCHmeans empty. - Poll for up to 2 s (100 ms interval), so leftovers that die right after the command still allow a reap.
- If live members remain (e.g.
cmd &), keep the zombie — eviction/terminatestill kills the group and reaps, so the safety property is unchanged. ProcessInfo, the registry, the output file,wait/get/list, exit codes andExitedevents are untouched.
This turns the common case (plain commands) into Z == 0 while preserving the "reserve the group for later killpg" behaviour for commands that intentionally leave descendants.
Patch against v2.3.0 (also adds three unit tests: empty group reaps, live member keeps the zombie, unknown state keeps it):
--- a/src/service/pipe/unix.rs
+++ b/src/service/pipe/unix.rs
@@ -20,7 +20,7 @@
use std::os::unix::process::CommandExt;
use std::process::{Command, Stdio};
use std::sync::{Arc, Mutex, PoisonError, RwLock};
-use std::time::Instant;
+use std::time::{Duration, Instant};
use anyhow::{Context, Result, bail};
use nix::errno::Errno;
@@ -29,6 +29,72 @@
use super::{DRAIN_AFTER_EXIT, Exit, OutputSink, start_pipe};
use crate::service::{CreatePipe, ProcessSignal, TerminalHandle, TerminalId};
+/// How long an exited pipe child may keep its zombie while its group still has
+/// live members, and how often that is re-checked.
+const REAP_WINDOW: Duration = Duration::from_secs(2);
+const REAP_POLL: Duration = Duration::from_millis(100);
+
+/// Whether `group` still has a live (non-zombie) member other than the exited
+/// child itself. `None` means the platform could not tell and callers must keep
+/// the zombie.
+#[cfg(target_os = "linux")]
+fn group_has_live_members(group: u32) -> Option<bool> {
+ let Ok(pid) = i32::try_from(group) else {
+ return None;
+ };
+ let entries = std::fs::read_dir("/proc").ok()?;
+ let mut scanned = false;
+ for entry in entries.flatten() {
+ let name = entry.file_name();
+ let Some(member) = name.to_str().and_then(|value| value.parse::<i32>().ok()) else {
+ continue;
+ };
+ // A member that exits between readdir and read leaves no stat file.
+ let Ok(stat) = std::fs::read_to_string(entry.path().join("stat")) else {
+ continue;
+ };
+ scanned = true;
+ let Some((state, pgrp)) = parse_stat(&stat) else {
+ continue;
+ };
+ if pgrp == pid && member != pid && state != 'Z' {
+ return Some(true);
+ }
+ }
+ // An unreadable /proc must not look like an empty group.
+ if scanned { Some(false) } else { None }
+}
+
+/// macOS has no `/proc`, but its `killpg(pid, 0)` reports `ESRCH` for a group
+/// that only holds zombies (the opposite of Linux), so it is a valid test.
+#[cfg(not(target_os = "linux"))]
+fn group_has_live_members(group: u32) -> Option<bool> {
+ let Ok(pid) = i32::try_from(group) else {
+ return None;
+ };
+ // SAFETY: signal 0 performs the permission/existence check only.
+ let result = unsafe { libc::killpg(pid, 0) };
+ if result == 0 {
+ return Some(true);
+ }
+ match Errno::last() {
+ Errno::ESRCH => Some(false),
+ _ => None,
+ }
+}
+
+/// Reads state and process-group id from `/proc/<pid>/stat`. The comm field can
+/// contain spaces and parentheses, so split at the last `)`.
+#[cfg(target_os = "linux")]
+fn parse_stat(stat: &str) -> Option<(char, i32)> {
+ let rest = stat.get(stat.rfind(')')? + 1..)?;
+ let mut fields = rest.split_whitespace();
+ let state = fields.next()?.chars().next()?;
+ let _ppid = fields.next()?;
+ let pgrp = fields.next()?.parse().ok()?;
+ Some((state, pgrp))
+}
+
/// Serializes pipe creation against `spawn`. macOS has no `pipe2`, so std
/// creates a pipe and then marks it close-on-exec; a fork in between would
/// leak the pipe into another child, where it could hold output or cancel
@@ -235,6 +301,32 @@
}
*reaped = true;
}
+
+ /// Reaps the exited child once its process group has no live members.
+ ///
+ /// The zombie keeps the PID and process-group ID reserved so a later
+ /// `killpg` can never hit an unrelated process. Once every member is gone
+ /// there is nothing left to signal, so the reservation protects nothing.
+ /// A group with live members keeps the zombie: the client may still kill
+ /// the leftovers when it evicts or terminates the process.
+ pub(super) fn reap_when_group_empty(&self) {
+ let deadline = Instant::now() + REAP_WINDOW;
+ loop {
+ match group_has_live_members(self.pid) {
+ // Reap only when we could actually prove the group is empty.
+ Some(false) => {
+ self.reap();
+ return;
+ }
+ // Live leftovers, or a platform we cannot inspect: keep it.
+ _ => {}
+ }
+ if Instant::now() >= deadline {
+ return;
+ }
+ std::thread::sleep(REAP_POLL);
+ }
+ }
}
/// Reads errno right after a `kill`/`killpg` result. A zombie group with no
@@ -418,3 +510,67 @@
}
}
}
+
+#[cfg(all(test, target_os = "linux"))]
+mod reap_tests {
+ use super::*;
+ use std::os::unix::process::CommandExt;
+ use std::process::Command;
+
+ fn spawn_child(script: &str) -> Process {
+ let child = Command::new("/bin/sh")
+ .arg("-c")
+ .arg(script)
+ .process_group(0)
+ .spawn()
+ .expect("spawn test child");
+ let pid = child.id();
+ // Dropping `Child` neither waits nor kills; the daemon owns reaping.
+ drop(child);
+ Process {
+ pid,
+ reaped: Mutex::new(false),
+ }
+ }
+
+ fn wait_without_reaping(pid: u32) -> Exit {
+ wait_exit(pid).expect("wait for test child")
+ }
+
+ fn state_of(pid: u32) -> Option<char> {
+ let stat = std::fs::read_to_string(format!("/proc/{pid}/stat")).ok()?;
+ parse_stat(&stat).map(|(state, _)| state)
+ }
+
+ fn exists(pid: u32) -> bool {
+ std::path::Path::new(&format!("/proc/{pid}")).exists()
+ }
+
+ #[test]
+ fn reaps_an_exited_child_whose_group_is_empty() {
+ let process = spawn_child("true");
+ wait_without_reaping(process.pid());
+ assert_eq!(state_of(process.pid()), Some('Z'));
+ process.reap_when_group_empty();
+ assert!(!exists(process.pid()), "the zombie must be gone");
+ }
+
+ #[test]
+ fn keeps_the_zombie_while_the_group_still_has_members() {
+ let process = spawn_child("sleep 5 & exit 0");
+ wait_without_reaping(process.pid());
+ // Polls for REAP_WINDOW, but a live `sleep` in the group must stop it.
+ process.reap_when_group_empty();
+ assert_eq!(state_of(process.pid()), Some('Z'), "a live member needs the group reserved");
+ process.kill();
+ process.reap();
+ assert!(!exists(process.pid()), "explicit kill + reap must still work");
+ }
+
+ #[test]
+ fn an_unknown_group_state_keeps_the_zombie() {
+ // A pid that is not a child can be observed but never reaped; the
+ // conservative branch must leave it alone rather than block.
+ assert_eq!(group_has_live_members(u32::MAX), None);
+ }
+}
--- a/src/service/pipe.rs
+++ b/src/service/pipe.rs
@@ -211,12 +211,18 @@
};
joins.push(reader);
let wait_actor_tx = actor_tx.clone();
+ let reap_after_exit = Arc::clone(&process);
let waiter = thread::Builder::new()
.name(format!("opencode-pty-wait-{id}"))
.spawn(move || {
let Waiter { exit, after } = waiter();
let _ = wait_actor_tx.send(exit_message(exit));
drop(after);
+ // The exit status is already reported, so the zombie's only job is
+ // keeping the process group reserved for signalling. With every
+ // member gone there is nothing left to signal, and the PID can be
+ // released; groups with live members keep the zombie for eviction.
+ reap_after_exit.reap_when_group_empty();
})
.context("failed to spawn terminal child waiter");
let waiter = match waiter {
--- a/src/service/pipe/windows.rs
+++ b/src/service/pipe/windows.rs
@@ -177,6 +177,10 @@
self.lock().handles.take();
}
+ /// Windows has no zombies: an exited process keeps nothing reserved, and
+ /// the handle is closed by `reap` at shutdown. Nothing to do.
+ pub(super) fn reap_when_group_empty(&self) {}
+
fn take_signal(&self) -> Option<&'static str> {
self.lock().killed.take()
}
Verification after the patch (same device)
- 50 mixed commands (normal / non-zero / timeout / cancel) →
Z == 0after every batch - 50 PTY create/remove →
Z == 0 - idle 10 s →
Z == 0 - regression checks:
exitcode preserved,status=timeout,status=killed,shell.outputstill readable - build:
aarch64-unknown-linux-muslstatic binary; ghostty'sbuild.zigrequires zig 0.16.x (not 0.17), linked with cargo-zigbuild
Happy to send this as a PR if the approach looks right.
Side note (worth a doc line)
The daemon socket path is
$OPENCODE_PTY_SOCKET_DIR/opencode-pty-<uid>/<32hex>.sock
(+56 bytes on top of the socket dir). With a long TMPDIR the daemon exits with error: path must be shorter than SUN_LEN, and opencode silently falls back to spawned shells — which hides this bug entirely during testing. Adding an explicit warning when the daemon cannot start would have saved me an hour.
Activity
opencodejr commentedon Nov 15, 2026
Thanks for the detailed write-up and the patch. I think this is a duplicate of #63817, which covers the same root cause: exited pipe children stay as zombies under
opencode-pty daemonuntil they are evicted from the list of recent finished commands or terminated. It has been reproduced on 2.3.0.Possible duplicates and related issues, ordered by likelihood:
<defunct>underopencode-pty daemonon 2.3.0, with the cap of 25 recent finished commands explained in its commentsI'm closing this issue now as a duplicate of #63817. Your proposed condition, reaping once the process group has no live members, and your on-device measurements would be very useful there, so please add them to #63817 (or link a PR from it). If you think this is a different problem, reply here and the issue will reopen.