Skip to content

process-daemon: pipe children stay as zombies (steady-state 25 Z) until eviction; safe to reap once the process group is empty #64076

Closed as duplicate

Description

@jcz2020

Summary

opencode-pty (packages/process-daemon) leaks one zombie process per finished agent shell command, and a long-running server stabilises at 25 Z processes (25 = EXITED_LIMIT in packages/core/src/shell.ts). Every zombie's PPID is the opencode-pty daemon. Only an explicit terminate/eviction reaps them, and the client only evicts when the finished-command history overflows, so a long session never gets back to 0.

This is not the PTY path. PTYs use child.wait() in the wait thread and are reaped correctly (verified 35 create/remove → 0 Z).

Why it happens

service/pipe/unix.rs intentionally keeps pipe children as zombies:

  • wait_exit() uses waitid(P_PID, …, WEXITED | WNOWAIT) — it observes the exit but does not reap.
  • The zombie keeps the PID and process-group ID reserved, so a later killpg can never hit an unrelated process.
  • Only TerminalHandle::shutdown() (→ Process::reap() → waitpid) actually reaps.

On the client side (packages/core/src/shell.ts):

  • EXITED_LIMIT = 25 keeps the last 25 finished commands for shell.get history/output.
  • evict() only runs when exitOrder.length > 25.

So: 1 command → 1 Z; commands 2…25 → linearly up to 25; after that it stays pinned at 25 forever (no TTL), and only explicit DELETE /api/shell/:id or a server restart drops them.

Reproduction (aarch64 Android, 2.3.0, isolated opencode serve)

warmup                              Z=1
+10 normal exits                    Z=11
+10 non-zero exits                  Z=21
+5 timeouts (sleep 30, timeout=800) Z=25
+5 cancels (DELETE while running)   Z=25
+20 more normal exits               Z=25   (stays)
idle 30s                            Z=25   (no TTL)
stop server                         Z=0

ps sample:

26386 26380 Z    sh <defunct>    [sh] <defunct>
26394 26380 Z    sh <defunct>    [sh] <defunct>
26401 26380 Z    sh <defunct>    [sh] <defunct>

26380 is the opencode-pty daemon. comm is sh, or sleep when the command was a timeout (sh -c execs the last command).

Deleting finished commands one by one immediately drops the count (5 → 4 → 3 → 2 → 1 → 0), confirming reap works and the only problem is when it is called.

What does not work as a reap condition

kill(-pgid, 0) cannot tell "group is empty" on Linux/Android: even when the group only holds the zombie it returns 0. (macOS returns ESRCH, which is presumably why the current comment says it's fine there.) Verified on device with waitid(WNOWAIT) + kill(-pgid, 0).

SIGCHLD = SIG_IGN / waitpid(-1) would lose the exit code and break Exited events.

Proposed fix (patch below)

After the exit message is emitted, reap the child only when the process group has no live (non-zombie) members:

  • Linux/Android: scan /proc/*/stat for pgrp == pid && state != 'Z', excluding the child itself. If /proc cannot be read, return "unknown" and keep the zombie.
  • macOS: killpg(pid, 0) → ESRCH means empty.
  • Poll for up to 2 s (100 ms interval), so leftovers that die right after the command still allow a reap.
  • If live members remain (e.g. cmd &), keep the zombie — eviction/terminate still kills the group and reaps, so the safety property is unchanged.
  • ProcessInfo, the registry, the output file, wait/get/list, exit codes and Exited events are untouched.

This turns the common case (plain commands) into Z == 0 while preserving the "reserve the group for later killpg" behaviour for commands that intentionally leave descendants.

Patch against v2.3.0 (also adds three unit tests: empty group reaps, live member keeps the zombie, unknown state keeps it):

--- a/src/service/pipe/unix.rs
+++ b/src/service/pipe/unix.rs
@@ -20,7 +20,7 @@
 use std::os::unix::process::CommandExt;
 use std::process::{Command, Stdio};
 use std::sync::{Arc, Mutex, PoisonError, RwLock};
-use std::time::Instant;
+use std::time::{Duration, Instant};
 
 use anyhow::{Context, Result, bail};
 use nix::errno::Errno;
@@ -29,6 +29,72 @@
 use super::{DRAIN_AFTER_EXIT, Exit, OutputSink, start_pipe};
 use crate::service::{CreatePipe, ProcessSignal, TerminalHandle, TerminalId};
 
+/// How long an exited pipe child may keep its zombie while its group still has
+/// live members, and how often that is re-checked.
+const REAP_WINDOW: Duration = Duration::from_secs(2);
+const REAP_POLL: Duration = Duration::from_millis(100);
+
+/// Whether `group` still has a live (non-zombie) member other than the exited
+/// child itself. `None` means the platform could not tell and callers must keep
+/// the zombie.
+#[cfg(target_os = "linux")]
+fn group_has_live_members(group: u32) -> Option<bool> {
+    let Ok(pid) = i32::try_from(group) else {
+        return None;
+    };
+    let entries = std::fs::read_dir("/proc").ok()?;
+    let mut scanned = false;
+    for entry in entries.flatten() {
+        let name = entry.file_name();
+        let Some(member) = name.to_str().and_then(|value| value.parse::<i32>().ok()) else {
+            continue;
+        };
+        // A member that exits between readdir and read leaves no stat file.
+        let Ok(stat) = std::fs::read_to_string(entry.path().join("stat")) else {
+            continue;
+        };
+        scanned = true;
+        let Some((state, pgrp)) = parse_stat(&stat) else {
+            continue;
+        };
+        if pgrp == pid && member != pid && state != 'Z' {
+            return Some(true);
+        }
+    }
+    // An unreadable /proc must not look like an empty group.
+    if scanned { Some(false) } else { None }
+}
+
+/// macOS has no `/proc`, but its `killpg(pid, 0)` reports `ESRCH` for a group
+/// that only holds zombies (the opposite of Linux), so it is a valid test.
+#[cfg(not(target_os = "linux"))]
+fn group_has_live_members(group: u32) -> Option<bool> {
+    let Ok(pid) = i32::try_from(group) else {
+        return None;
+    };
+    // SAFETY: signal 0 performs the permission/existence check only.
+    let result = unsafe { libc::killpg(pid, 0) };
+    if result == 0 {
+        return Some(true);
+    }
+    match Errno::last() {
+        Errno::ESRCH => Some(false),
+        _ => None,
+    }
+}
+
+/// Reads state and process-group id from `/proc/<pid>/stat`. The comm field can
+/// contain spaces and parentheses, so split at the last `)`.
+#[cfg(target_os = "linux")]
+fn parse_stat(stat: &str) -> Option<(char, i32)> {
+    let rest = stat.get(stat.rfind(')')? + 1..)?;
+    let mut fields = rest.split_whitespace();
+    let state = fields.next()?.chars().next()?;
+    let _ppid = fields.next()?;
+    let pgrp = fields.next()?.parse().ok()?;
+    Some((state, pgrp))
+}
+
 /// Serializes pipe creation against `spawn`. macOS has no `pipe2`, so std
 /// creates a pipe and then marks it close-on-exec; a fork in between would
 /// leak the pipe into another child, where it could hold output or cancel
@@ -235,6 +301,32 @@
         }
         *reaped = true;
     }
+
+    /// Reaps the exited child once its process group has no live members.
+    ///
+    /// The zombie keeps the PID and process-group ID reserved so a later
+    /// `killpg` can never hit an unrelated process. Once every member is gone
+    /// there is nothing left to signal, so the reservation protects nothing.
+    /// A group with live members keeps the zombie: the client may still kill
+    /// the leftovers when it evicts or terminates the process.
+    pub(super) fn reap_when_group_empty(&self) {
+        let deadline = Instant::now() + REAP_WINDOW;
+        loop {
+            match group_has_live_members(self.pid) {
+                // Reap only when we could actually prove the group is empty.
+                Some(false) => {
+                    self.reap();
+                    return;
+                }
+                // Live leftovers, or a platform we cannot inspect: keep it.
+                _ => {}
+            }
+            if Instant::now() >= deadline {
+                return;
+            }
+            std::thread::sleep(REAP_POLL);
+        }
+    }
 }
 
 /// Reads errno right after a `kill`/`killpg` result. A zombie group with no
@@ -418,3 +510,67 @@
         }
     }
 }
+
+#[cfg(all(test, target_os = "linux"))]
+mod reap_tests {
+    use super::*;
+    use std::os::unix::process::CommandExt;
+    use std::process::Command;
+
+    fn spawn_child(script: &str) -> Process {
+        let child = Command::new("/bin/sh")
+            .arg("-c")
+            .arg(script)
+            .process_group(0)
+            .spawn()
+            .expect("spawn test child");
+        let pid = child.id();
+        // Dropping `Child` neither waits nor kills; the daemon owns reaping.
+        drop(child);
+        Process {
+            pid,
+            reaped: Mutex::new(false),
+        }
+    }
+
+    fn wait_without_reaping(pid: u32) -> Exit {
+        wait_exit(pid).expect("wait for test child")
+    }
+
+    fn state_of(pid: u32) -> Option<char> {
+        let stat = std::fs::read_to_string(format!("/proc/{pid}/stat")).ok()?;
+        parse_stat(&stat).map(|(state, _)| state)
+    }
+
+    fn exists(pid: u32) -> bool {
+        std::path::Path::new(&format!("/proc/{pid}")).exists()
+    }
+
+    #[test]
+    fn reaps_an_exited_child_whose_group_is_empty() {
+        let process = spawn_child("true");
+        wait_without_reaping(process.pid());
+        assert_eq!(state_of(process.pid()), Some('Z'));
+        process.reap_when_group_empty();
+        assert!(!exists(process.pid()), "the zombie must be gone");
+    }
+
+    #[test]
+    fn keeps_the_zombie_while_the_group_still_has_members() {
+        let process = spawn_child("sleep 5 & exit 0");
+        wait_without_reaping(process.pid());
+        // Polls for REAP_WINDOW, but a live `sleep` in the group must stop it.
+        process.reap_when_group_empty();
+        assert_eq!(state_of(process.pid()), Some('Z'), "a live member needs the group reserved");
+        process.kill();
+        process.reap();
+        assert!(!exists(process.pid()), "explicit kill + reap must still work");
+    }
+
+    #[test]
+    fn an_unknown_group_state_keeps_the_zombie() {
+        // A pid that is not a child can be observed but never reaped; the
+        // conservative branch must leave it alone rather than block.
+        assert_eq!(group_has_live_members(u32::MAX), None);
+    }
+}
--- a/src/service/pipe.rs
+++ b/src/service/pipe.rs
@@ -211,12 +211,18 @@
     };
     joins.push(reader);
     let wait_actor_tx = actor_tx.clone();
+    let reap_after_exit = Arc::clone(&process);
     let waiter = thread::Builder::new()
         .name(format!("opencode-pty-wait-{id}"))
         .spawn(move || {
             let Waiter { exit, after } = waiter();
             let _ = wait_actor_tx.send(exit_message(exit));
             drop(after);
+            // The exit status is already reported, so the zombie's only job is
+            // keeping the process group reserved for signalling. With every
+            // member gone there is nothing left to signal, and the PID can be
+            // released; groups with live members keep the zombie for eviction.
+            reap_after_exit.reap_when_group_empty();
         })
         .context("failed to spawn terminal child waiter");
     let waiter = match waiter {
--- a/src/service/pipe/windows.rs
+++ b/src/service/pipe/windows.rs
@@ -177,6 +177,10 @@
         self.lock().handles.take();
     }
 
+    /// Windows has no zombies: an exited process keeps nothing reserved, and
+    /// the handle is closed by `reap` at shutdown. Nothing to do.
+    pub(super) fn reap_when_group_empty(&self) {}
+
     fn take_signal(&self) -> Option<&'static str> {
         self.lock().killed.take()
     }

Verification after the patch (same device)

  • 50 mixed commands (normal / non-zero / timeout / cancel) → Z == 0 after every batch
  • 50 PTY create/remove → Z == 0
  • idle 10 s → Z == 0
  • regression checks: exit code preserved, status=timeout, status=killed, shell.output still readable
  • build: aarch64-unknown-linux-musl static binary; ghostty's build.zig requires zig 0.16.x (not 0.17), linked with cargo-zigbuild

Happy to send this as a PR if the approach looks right.

Side note (worth a doc line)

The daemon socket path is
$OPENCODE_PTY_SOCKET_DIR/opencode-pty-<uid>/<32hex>.sock
(+56 bytes on top of the socket dir). With a long TMPDIR the daemon exits with error: path must be shorter than SUN_LEN, and opencode silently falls back to spawned shells — which hides this bug entirely during testing. Adding an explicit warning when the daemon cannot start would have saved me an hour.

Activity

  1. opencodejr commented on Nov 15, 2026

    @opencodejr
    Contributor

    Thanks for the detailed write-up and the patch. I think this is a duplicate of #63817, which covers the same root cause: exited pipe children stay as zombies under opencode-pty daemon until they are evicted from the list of recent finished commands or terminated. It has been reproduced on 2.3.0.

    Possible duplicates and related issues, ordered by likelihood:

    I'm closing this issue now as a duplicate of #63817. Your proposed condition, reaping once the process group has no live members, and your on-device measurements would be very useful there, so please add them to #63817 (or link a PR from it). If you think this is a different problem, reply here and the issue will reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions