Skip to content

fix(installer): give TestGen time to stop before docker kills it - #91

Merged
rboni-dk merged 1 commit into
mainfrom
feat/testgen-stop-grace-period
Aug 26, 2026
Merged

fix(installer): give TestGen time to stop before docker kills it#91
rboni-dk merged 1 commit into
mainfrom
feat/testgen-stop-grace-period

Conversation

@rboni-dk

Copy link
Copy Markdown
Contributor

Why

TestGen's scheduler stops cooperatively: on SIGTERM it asks running jobs to stop at a checkpoint, waits up to TG_JOB_SHUTDOWN_TIMEOUT (default 60s), and records where each one got to so the next start continues from there.

Docker's default stop grace period is 10s, so a stop, restart, or upgrade killed the container mid-wait — the checkpoint was never written and the run started over from nothing. A large profiling run can be many minutes of work.

Measured

Same job, same 30s columns, only the grace period differs:

Grace Stop took Exit Outcome
10s (old default) 10s 137 SIGKILL stranded -> canceled next start, 38 columns discarded
90s (this PR) 53s 0 interrupted, checkpoint kept, resumed

docker compose down behaves the same (52s, checkpoint kept), which is the path tg upgrade and tg delete use.

90s clears the 60s wait plus the final write, and matches what TestGen ships in its own compose files and Helm chart.

What changed

The compose template alone would reach only new installs, so this lands in three places:

  • Generated compose file - fresh Docker installs.
  • tg upgrade - patches the compose file in place. CreateComposeFileStepBase reuses an existing file, so this is the only path that reaches current users. Scoped to the engine service by indentation via a new find_in_block helper, and matched on the image key rather than its value so a private-mirror install is patched too.
  • Pip mode - stop_app_tree force-killed the tree after 10s. It now waits the same grace period, reports whether the stop was clean, and swallows a second Ctrl+C rather than letting it re-signal the tree (TestGen reads a second signal as "hurry up" and cuts the job mid-pause).

The force-kill path also sweeps orphans now: run-app all starts its ui/scheduler children in their own sessions, so killpg on the parent left them holding the port and data directory, breaking the next tg start.

Known limits

  • Windows still force-kills - no catchable signal is deliverable without CREATE_NEW_PROCESS_GROUP, and TestGen's own _forward_signal_to_child force-kills children there regardless. The Ctrl+C messaging is gated on this so it neither promises a wait it cannot honor nor warns about a lost job on every stop.
  • A column longer than TG_JOB_SHUTDOWN_TIMEOUT is still cut. Verified: a 150s column overruns the 90s grace and is SIGKILLed. No grace period fixes a step with no interior stop check.

Testing

207 unit/integration tests pass; ruff clean. Behaviour verified end-to-end against a locally built TestGen image carrying the cooperative-shutdown work, across five scenarios (fast columns, slow columns, the 10s counterfactual, a column exceeding the grace, and compose down).

🤖 Generated with Claude Code

TestGen's scheduler stops cooperatively: on SIGTERM it asks running jobs to
stop at a checkpoint, waits up to TG_JOB_SHUTDOWN_TIMEOUT (default 60s), and
records where each one got to so the next start continues from there.

Docker's default stop grace period is 10s, so a stop, restart, or upgrade
killed the container mid-wait. The checkpoint was never written, the run was
canceled, and the next launch started it over from nothing. Measured against a
profiling run whose columns take 30s: with 10s the container is SIGKILLed and
the next start cancels the run, discarding 38 columns of work; with 90s it
stops cleanly in 53s and resumes. 90s clears the 60s wait plus the final write,
and matches what TestGen ships in its own compose files and Helm chart.

Applied in three places, since the compose template alone reaches only new
installs:

- The generated compose file, for fresh Docker installs.
- tg upgrade, which patches the compose file in place — existing installs keep
  their file forever, so this is the only path that reaches them. Scoped to the
  engine service by indentation, and matched on the image key rather than its
  value, so a private mirror is patched too.
- Pip mode, where stop_app_tree force-killed the tree after 10s. It now waits
  the same grace period, reports whether the stop was clean, and swallows a
  second Ctrl+C rather than letting it re-signal the tree — TestGen reads a
  second signal as "hurry up" and cuts the job mid-pause.

The force-kill path also sweeps orphans now: run-app all starts its ui and
scheduler children in their own sessions, so killpg on the parent left them
holding the port and the data directory, breaking the next tg start.

Windows still force-kills, as it has no catchable signal to deliver without
CREATE_NEW_PROCESS_GROUP. The Ctrl+C messaging is gated on that so it neither
promises a wait it cannot honor nor warns about a lost job on every stop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rboni-dk
rboni-dk merged commit 0fb33f1 into main Aug 26, 2026
3 checks passed
@rboni-dk
rboni-dk deleted the feat/testgen-stop-grace-period branch August 26, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants