Skip to content

packaging: cocoon-agent keeps running when the OOM killer kills one of its exec children - #21

Merged
CMGS merged 1 commit into
mainfrom
m6/agent-oom-continue
Oct 23, 2026
Merged

CMGS merged 1 commit into
mainfrom
m6/agent-oom-continue

Conversation

@CMGS

@CMGS CMGS commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

packaging/cocoon-agent.service now sets OOMPolicy=continue.

Why

Commands run through cocoon-agent (cocoon vm exec, kubectl exec) live in cocoon-agent.service's cgroup, because KillMode=control-group keeps them there. With the default OOMPolicy=stop, one kernel OOM kill of any such command makes systemd stop the whole unit. That kills the agent and every other in-flight exec, and vsock exec fails until Restart=always brings the agent back.

The same mechanism was reproduced on silkd.service, which had the same unit shape (cocoonstack/sandbox#377). On a 7.9 GiB guest with no swap, a growing memory hog was started through the daemon's exec:

  • The kernel kills only the hog (Killed process … (python3) … oom_score_adj:0).
  • systemd logs "A process of this unit has been killed by the OOM killer", then "Failed with result 'oom-kill'", then "Scheduled restart job". NRestarts goes up by one and MainPID changes.
  • A vsock probe gets EOF for about 2.5 s.
  • A sleep started earlier through the same daemon dies with the unit.

The same guest runs cocoon-agent.service with OOMPolicy=stop and adj 0, so a hog started through cocoon vm exec takes the agent down the same way.

With OOMPolicy=continue, the OOM killer still picks the hog, and the agent and its other execs keep running. systemctl stop still tears the cgroup down as before. The setting has no runtime cost.

Checks

  • systemd-analyze verify (systemd 255) on the unit: rc 0.
  • The os-image copies in cocoonstack/cocoon (Ubuntu and Debian) and cocoonstack/sandbox get the same line.

@CMGS
CMGS merged commit 624c167 into main Oct 23, 2026
2 checks passed
@CMGS
CMGS deleted the m6/agent-oom-continue branch October 23, 2026 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant