Skip to content

Preserve container exits and OOM backoff - #2045

Merged
luke-lombardi merged 3 commits into
mainfrom
ll/tini-embed
Oct 21, 2026
Merged

luke-lombardi merged 3 commits into
mainfrom
ll/tini-embed

Conversation

@luke-lombardi

@luke-lombardi luke-lombardi commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Reap detached runsc children, preserve terminal exit/OOM evidence through cleanup, and bound final log drains without changing failure backoff.

Validated Linux/race tests and staging exits/OOMs/stops, functions/endpoints, sandbox survival, logging/checkpoint/restore, and fresh image builds. Pins the approved merged gVisor runtime.

@luke-lombardi
luke-lombardi marked this pull request as ready for review October 21, 2026 04:13

@jackf-beamcloud jackf-beamcloud left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks right: tini re-exec only when the worker is PID 1 (tini is already in the worker image), final-metrics evidence covers the lost wait RPC case including init vs guest OOM, and the terminalOOM flag stops later cleanup stops from reclassifying an OOM. Non-blocking: Wait does an unbounded <-done after forceDelete with a WithoutCancel ctx and no cmd.WaitDelay on the restore command, so a stray inherited stdout fd would hang it; and restoreOutput entries leak if a restored container never reaches Wait.

@luke-lombardi

Copy link
Copy Markdown
Contributor Author

Addressed both follow-ups in 6b06ffa: bounded post-exit draining (including detached Run), automatic restore-tracking cleanup, and joining pending output after deletion. Orphan-FD regressions cover Run/Wait/Delete/no Wait and fail without the fixes.

Full Linux/race and staging checks passed again. Restored stdout/stderr remained live after 9.18s, with all 130 final-tail lines preserved.

@jackf-beamcloud jackf-beamcloud left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6b06ffa covers both earlier notes: WaitDelay on the restore command plus the read deadline on the output pipe bound post-exit draining, and restore tracking now removes itself via CompareAndDelete when the restore CLI exits, so containers that never reach Wait don't leak entries. Orphan-fd tests for Run/Wait/Delete/no-Wait look right. Non-blocking: once the 5s deadline hits, output an orphan writes after that is dropped, which seems like the right trade.

@luke-lombardi
luke-lombardi merged commit 4353a04 into main Oct 21, 2026
5 checks passed
@luke-lombardi
luke-lombardi deleted the ll/tini-embed branch October 21, 2026 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants