fix(cache): a test-temp run lives exactly as long as its owner - #271
Open
ebursztein wants to merge 3 commits into
Open
ebursztein wants to merge 3 commits into
ebursztein wants to merge 3 commits into
Conversation
/var/tmp/capsem-tests/<namespace> reached 42 GB: one ~7 GB run-<pid> per
release precheck plus ~1,900 .run-<pid>.lock files back to early September.
`cache prune test-temp` offered 21 MB; only a cold clean reclaimed it.
Two defects, one on each side of the lease:
- Nothing removed a run when it ended. The launcher leased run-<pid> and let
the lease drop at exit, leaving the directory and its lock file behind on
success and failure alike. gatelaunch now leases the run before creating it
and, at exit, removes the run and its lease through
operations.reclaim_generation -- the same guarded, journaled removal a
prune uses. A killed process leaves a run no lease holds.
- Prune treated a dead run like a retained generation: it aged for 24 hours
under a 200 GiB maximum. For a leased ephemeral stage an unleased
generation now has "no live owner" and is reclaimed whatever its age or
size. Inventory attaches each lease file to its generation, so both leave
together, and reports a lease whose generation is gone as lease_only: it is
collected ("orphaned lease") and never counts toward maximum_count.
plan_clean now removes a generation's members too.
Removal takes each generation's lease exclusively, so a reused pid that
leased after the plan keeps its directory (ApplyResult.busy). retain_path
waits out that exclusive hold instead of failing, and retakes a lease file
that was unlinked under it.
With dead runs reclaimed on sight, test-temp holds only live runs, so its maximum is a statement about how much live scratch is reasonable, not a retention budget. 200 GiB on a 484 GB disk that also carries a 180 GiB Cargo target bounded nothing: the disk filled first. 32 GiB is about four times the largest run observed (a 7.2 GB release precheck), enough for a gate plus several concurrent bounded suites; more than that is a leak that enforcement should report rather than absorb. Warm stays 8 GiB.
pytest's default retention keeps every tmp_path until the next session wipes the basetemp, so a run's scratch peaked at the sum of the whole suite (0.8 GB for build_system/tests alone). With "failed", only failing and in-flight tests hold their directories, and the failing ones remain for inspection until the run ends.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #271 +/- ##
=========================================
- Coverage 65.3% 65.3% -0.1%
=========================================
Files 1452 1452
Lines 127965 128076 +111
Branches 91872 91872
=========================================
+ Hits 83614 83637 +23
- Misses 39361 39446 +85
- Partials 4990 4993 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On 2026-09-28,
/var/tmp/capsem-tests/6e48ba3b/(thetest-tempstage) had grown to 42 GB: onerun-<pid>/of about 7.2 GB per release precheck, plus about 1,900.run-<pid>.lockfiles going back to 2026-09-04.just cache prune test-tempoffered 21 MB. Onlyjust cache clean test-temp --applygot the space back.Root causes
gatelaunchcreatesrun-<pid>(the gate's or the bounded wrapper'sTMPDIRand pytest--basetemp) and leases.run-<pid>.lock. The lease is dropped at process exit, but the directory and the lock file were left behind every time, on success and on failure. Every launcher did this:capsem-gate,capsem-cache,run-bounded-command.py, and every test that starts one.ephemeralstrategy the planner used the same rules asgenerational: a run with no lease was reclaimed only aftermaximum_age_hours(24) or once the stage passed its 200 GiB maximum. Lock files were not matched bymanaged_globs, so they were never collectable at all.contained_environmentcreated the run directory before any lease existed. Tests that build a temporary checkout therefore leftrun-<pid>directories in one new namespace per test. This is where most of the ~350 namespace directories under/var/tmp/capsem-testscame from.Fix
gatelaunch.hold_environmenttakes the lease first, then creates the run. At exit it hands the run tooperations.reclaim_generation, which releases the lease and removes the run and its lease through the same guarded, journaledapply_prunepath. A process that is killed leaves a run that no lease holds.ephemeralstage, an unleased generation is now reported as "no live owner" and is reclaimed whatever its age or size.lease_onlyentry: it is collected as "orphaned lease" and never counts towardmaximum_count.plan_cleannow also removes a generation's members.apply_pruneholds each generation's lease exclusively while removing it. If a reused pid leased the run after the plan was made, the run is kept and reported inApplyResult.busy.retain_pathnow waits for that exclusive hold instead of raising, and re-takes a lease file that was unlinked while it waited.test-tempmax goes from 200 GiB to 32 GiB, with the reasoning inconfig/cache.toml. The stage now holds only live runs, and 32 GiB is about 4x the largest run seen.tmp_path_retention_policy = "failed". On this box, the peak scratch forbuild_system/testsdrops from 0.8 GB to 0.3 GB.Evidence
build_system/testsrun each left theirrun-<pid>behind (the latter 801 MB).node_modulesand assets, the contract suites peak at 0.8 GB and 76 MB. With this change, a run no longer outlives its owner whatever its size.Tests
build_system/tests/cache/test_scratch_runs.py:build_system/tests: the only failures are the 23 environmental ones that also fail onorigin/mainon this machine (test_mock_server_launcherandtest_protocol_fixture_recorderneed thecapsem-mock-serverbinary).tests/citadel -n 4passes: 1255 tests.ruffandcapsem-gate lintare clean.Follow-up (not in this PR)
External namespaces from retired authorities, such as deleted worktrees that exported
CAPSEM_CACHE_AUTHORITY, are never inventoried by any other authority. Their dead runs stay behind until someone cleans them by hand.🤖 Generated with Claude Code