fix(installer): give TestGen time to stop before docker kills it - #91
Merged
Conversation
TestGen's scheduler stops cooperatively: on SIGTERM it asks running jobs to stop at a checkpoint, waits up to TG_JOB_SHUTDOWN_TIMEOUT (default 60s), and records where each one got to so the next start continues from there. Docker's default stop grace period is 10s, so a stop, restart, or upgrade killed the container mid-wait. The checkpoint was never written, the run was canceled, and the next launch started it over from nothing. Measured against a profiling run whose columns take 30s: with 10s the container is SIGKILLed and the next start cancels the run, discarding 38 columns of work; with 90s it stops cleanly in 53s and resumes. 90s clears the 60s wait plus the final write, and matches what TestGen ships in its own compose files and Helm chart. Applied in three places, since the compose template alone reaches only new installs: - The generated compose file, for fresh Docker installs. - tg upgrade, which patches the compose file in place — existing installs keep their file forever, so this is the only path that reaches them. Scoped to the engine service by indentation, and matched on the image key rather than its value, so a private mirror is patched too. - Pip mode, where stop_app_tree force-killed the tree after 10s. It now waits the same grace period, reports whether the stop was clean, and swallows a second Ctrl+C rather than letting it re-signal the tree — TestGen reads a second signal as "hurry up" and cuts the job mid-pause. The force-kill path also sweeps orphans now: run-app all starts its ui and scheduler children in their own sessions, so killpg on the parent left them holding the port and the data directory, breaking the next tg start. Windows still force-kills, as it has no catchable signal to deliver without CREATE_NEW_PROCESS_GROUP. The Ctrl+C messaging is gated on that so it neither promises a wait it cannot honor nor warns about a lost job on every stop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
aarthy-dk
approved these changes
Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
TestGen's scheduler stops cooperatively: on SIGTERM it asks running jobs to stop at a checkpoint, waits up to
TG_JOB_SHUTDOWN_TIMEOUT(default 60s), and records where each one got to so the next start continues from there.Docker's default stop grace period is 10s, so a stop, restart, or upgrade killed the container mid-wait — the checkpoint was never written and the run started over from nothing. A large profiling run can be many minutes of work.
Measured
Same job, same 30s columns, only the grace period differs:
cancelednext start, 38 columns discardedinterrupted, checkpoint kept, resumeddocker compose downbehaves the same (52s, checkpoint kept), which is the pathtg upgradeandtg deleteuse.90s clears the 60s wait plus the final write, and matches what TestGen ships in its own compose files and Helm chart.
What changed
The compose template alone would reach only new installs, so this lands in three places:
tg upgrade- patches the compose file in place.CreateComposeFileStepBasereuses an existing file, so this is the only path that reaches current users. Scoped to theengineservice by indentation via a newfind_in_blockhelper, and matched on theimagekey rather than its value so a private-mirror install is patched too.stop_app_treeforce-killed the tree after 10s. It now waits the same grace period, reports whether the stop was clean, and swallows a second Ctrl+C rather than letting it re-signal the tree (TestGen reads a second signal as "hurry up" and cuts the job mid-pause).The force-kill path also sweeps orphans now:
run-app allstarts its ui/scheduler children in their own sessions, sokillpgon the parent left them holding the port and data directory, breaking the nexttg start.Known limits
CREATE_NEW_PROCESS_GROUP, and TestGen's own_forward_signal_to_childforce-kills children there regardless. The Ctrl+C messaging is gated on this so it neither promises a wait it cannot honor nor warns about a lost job on every stop.TG_JOB_SHUTDOWN_TIMEOUTis still cut. Verified: a 150s column overruns the 90s grace and is SIGKILLed. No grace period fixes a step with no interior stop check.Testing
207 unit/integration tests pass; ruff clean. Behaviour verified end-to-end against a locally built TestGen image carrying the cooperative-shutdown work, across five scenarios (fast columns, slow columns, the 10s counterfactual, a column exceeding the grace, and
compose down).🤖 Generated with Claude Code