Conversation
Size Report
Startup median (7 runs, lower is better):
|
|
Release now checks ownership, but at 2ec6027 a stale reclaim can still give the lock to two contenders. In Release can now reject with A smaller-design question: could each acquisition write a random token into Coverage fails because of this change: |
|
All four points addressed. The reclaim race and the release callers in 7d4b8f3, the claim token in 9ccc2bc. The rename race. After the rename, what arrived is now compared against what was judged: the inode and device from the Release callers. The cancel path in The The claim token. Adopted, and it turned out to close a hole the inode check left. Each acquisition writes a random The coverage rule. I changed the rule. Gate at 9ccc2bc: |
|
At 9ccc2bc the inode and claim-token checks stop the race from the last review, and the five release sites you changed now keep the task's own error. Two gaps remain. The reclaim still has a window where the lock path is empty while a live lock sits under the Four callers still release in A smaller-design question: could a short reclaim mutex (for example CI is green and there are no conflicts. |
|
Pushed 1. The rename window is gone because the rename is gone. A reclaim no longer moves anything: it takes
The contender you asked for, injected at the moment the judge is admitted: It clears the dead claim, publishes itself, and the acquirer must time out with 2. Four
3. Not every caller is task-shaped, and one should stay best-effort. Gate: |
|
Verified at
|
|
Reviewed at e62cdd8. Reclaim now runs under the A failed runner build can now show a lock error instead of the build error. runner-artifact.ts#L527 still releases in Reclaim can still remove a lock that a new contender just took. A similar question at process-lock.ts#L273: for an abandoned directory with no record, store-filesystem.ts#L117 still says a stale reclaim renames a lock aside, and store.test.ts:131 still builds a The two-contender test at process-lock.test.ts#L269 runs both acquirers in one process, and reclaim is synchronous, so the mutex is never contended; with Would a smaller shape work under the mutex? A directory with a record could be judged by its claim alone, and an empty one by CI is green on e62cdd8, and there are no conflicts. The next step is the build-error masking in runner-artifact.ts and the reclaim that removes a path after |
e62cdd8 to
0f37589
Compare
|
Addressed at A failed build reports the build. A reclaim that finds the path gone touches nothing. An abandoned directory with no record is Names. The two-contender test asks with a second of grace now: with zero, the mutex the winner holds reads as abandoned and the test measures a reclaim that skipped the mutex. Where the mutex is held (default 5s grace) the test also asserts the contender kept polling rather than giving up after one attempt. Reclaim is synchronous in-process, so genuine two-process contention is not producible there; the interleaving is injected in a contender that claims the path during a reclaim keeps its lock. Smaller shape taken. What each remaining decision is worth, by mutation:
Two reclaim tests are new (a claim published while a reclaim holds the mutex outlives the empty directory it filled, a lock directory made anew…) alongside the three redirect tests. |
|
Reviewed at 0f37589, as a follow-up to the review at e62cdd8. The claim token and the rmdir-only clear under the mutex are in, and the build error now wins over a release error because the precedence lives in From the code, a release that fails for a reason other than a lost race can wedge the device-set lock until the daemon restarts. Release returns "unverified" on any non-ENOENT read or unlink error, for example EMFILE or EACCES (process-lock.ts#L40, with the call sites at L143, L162 and L168). The lock directory stays, and its record still names this live daemon with a token that nothing holds. The bare Is the separate best-effort helper still needed? It reads as a second copy of the All 18 checks pass at 0f37589, and there are no conflicts. The next step is a recovery rule for a lock with a spent claim token, plus the narrower catch. |
0f37589 to
bb64eac
Compare
|
Reviewed at bb64eac, as a follow-up to the review at 0f37589. The delta takes the smaller design the last review asked for: one module-level set and a handle method instead of the standalone helper. One problem is left in A question on the live-claim set: it is per module instance. If Coverage fails only on Next: fix the error precedence in |
bb64eac to
ef508bc
Compare
|
Reviewed again at A release that fails for a reason other than a race no longer wedges the lock, and the reason it could not is now recorded. You were right that the fix had to be about the claim and not the removal. A token is issued per acquisition and held in The swallow is narrowed and it moved. No separate helper. The standalone best-effort function is gone; the handle carries The double honesty point from the same round was load-bearing, not cosmetic: two fixtures that pretended to be a rival process were naming Mutations, each run against the tree with only that change:
Gate at Honest gaps. The wedge needs a |
… evicts a live owner Release removed the lock directory unconditionally, so a contender that reclaimed the lock while its previous holder was away had that holder delete the exclusion belonging to whoever holds it now. Release removes only while the record inside still names the acquirer, and says so with a typed reason when it cannot read far enough to tell, rather than clearing a lock it might not own. A record that could not be read looked exactly like a record that was never written, so a live owner whose owner.json is unreadable or malformed aged out at the grace window and lost a lock it was holding. Only ENOENT now speaks for an unwritten record; any other read failure, and any record that does not name a live-shaped process, is an owner of unknown identity and the lock stands. Both wait paths carry a hint naming the directory, because nothing else in the repository removes it. A stale reclaim moves the abandoned directory aside under a unique `.lock` name before removing it. Two plain removals both report success, because the second is a silent no-op on a path the first already cleared, and both contenders then continue as though they had freed the lock. mkdir's EEXIST, unchanged, still arbitrates the lock itself. A reclaim that fails for a reason the fallback cannot settle waits and retries instead of surfacing an errno, and where the win32 fallback does force a removal it re-reads the record first, because a refused rename means a live contender may have claimed the path since. The allocation store keeps lane locks beside the lanes they guard, so it now recognises any `.lock` directory as a lock instead of only the `.lane.lock` suffix.
…before removal A rename addresses whatever stands at the lock path now, not the directory whose record was read. A contender that reclaimed first and published a live owner in the meantime had its live directory moved aside by the loser and then removed, which handed the same lock to two holders. What arrives is now compared against what was judged — same inode, and no live owner inside — and a directory that turns out to belong to someone else is renamed back rather than deleted. `releaseProcessLock` can now refuse to clear a lock it cannot prove it owns, so the callers that run a task under a lock stop letting that verdict displace the task's own failure: the unverifiable release is reported when the task succeeded, and suppressed when the task already failed, because the lock's stale-clear path resolves a lock that is still standing while nothing else recovers why the task failed. The abandoned cache-lock acquire releases without leaving a rejected promise nobody awaits. Co-authored-by: Apex by Callstack <noreply@callstack.com>
…ts process Two records can name the same process and still be different acquisitions of the same path, which is the distinction a release and a reclaim both need. Each acquisition publishes a random claim token with its owner record: release matches the token rather than pid and start time alone, and a reclaim compares the token of the record inside the directory it moved with the one it judged before renaming. The publication-ownership rule now says what it means for the process lock: its owner record goes through the shared publication owner and it writes no file by hand, while its renames are addressed only between the lock path and the reclaimed name. Reclaiming a lock directory is a different claim of ownership from publishing a file into one. Co-authored-by: Apex by Callstack <noreply@callstack.com>
…aks over its caller
Rename-aside made a reclaim a two-step transaction: move the abandoned directory out from under the
lock path, judge it there, and move it back if it turned out to belong to somebody else. Between
those steps the lock path is simply absent, and a contender polling the path reads that as free: it
claims the path and publishes, and the rename back returns `ENOTEMPTY` into a `catch {}`. The judged
directory ends up nobody's.
Removal happens in place now, behind `mkdir <lock>.reclaim`, and every branch decides again from what
is on disk rather than from a `Stats` read before the mutex was taken:
- a directory whose record named a dead claim goes only while that record is still there answering
to the same claim token. A token is a random id no later acquisition repeats, so a contender that
claimed the path in between walks away holding its lock;
- a directory with no record has no claim to attribute its contents to, so `rmdir` is the only call
made on it and its age is asked again. `ENOTEMPTY` is a publication saying so, and a directory
dated a moment ago is an acquisition that has not published yet, not an abandoned one;
- a path that is not a directory is unlinked, and `unlink` itself answers `EISDIR` for the one case
this branch must not touch;
- a path that has already gone is left alone, so the caller's next `mkdir` simply wins it.
`releaseProcessLock` lost its recursive removal for the same reason: it unlinks the record it
verified and removes the directory only while empty.
Callers that were answering "which of two failures do I report?" by hand with `finally` now go
through one shape: `withProcessLock({ acquire, task })` releases best-effort when the task failed and
strictly when it did not. `runner-artifact.ts`, `runner-cache.ts`, `managed-allocation/src/store.ts`
and `store-lock.ts` each had a release that could speak `ownerReleaseUnverified` over the task's own
failure. `managed agent-browser setup gives the lock back on every path out` pins a second bug that
shape was hiding: setup returned early when the package was already installed, skipping the only
`await release()` and leaving the lock for the stale-clear path to notice five seconds later.
The XCTest device-set redirect had that shape twice more, and neither site is a task: a launch that
failed waits on the redirect in `runner-session.ts`, and a teardown in `runner-disposal.ts`. Both call
`releaseXcodebuildSimulatorSetRedirectBestEffort` now, and the build path calls
`withXcodebuildSimulatorSetRedirect`, so the `xcodebuild` failure is the error a caller reads. The
parked directory is gone from the vocabulary as well: a stale reclaim holds `<lock>.reclaim.lock`,
which is what the `managed-allocation` store comment and its lock-scan test now say, and the
two-contender test asks with a second of grace rather than none, because a zero grace reads the
winner's own mutex as abandoned.
…ocess A release that could not verify ownership — the `unlink` of the record refused by EACCES or EMFILE, the record unreadable — left the lock directory standing with a record naming this live pid. The next acquire from the same process read that record, found a live owner, and waited 30 s for it; the runner build or launch behind it failed with "Timed out waiting for ...". The only thing that would end the wait was restarting the daemon, because the pid and start time the reclaim reads outlive the claim that was written with them. So the claim, not the process, is what the reclaim has to date. A token is issued per acquisition, and the moment a release is asked for, nobody inside this process is acting on it — whether the removal afterwards succeeds or not. `clearStaleProcessLock` now reads a record naming this pid under a token this process no longer holds as the dead claim it is, and takes the path back instead of waiting for a restart nothing is going to perform. The failed release is recorded as `process_lock_release_unverified` rather than vanishing into the caller that swallowed it. Two doubles that pretended to be another process were naming this pid with a token it never issued, which is now precisely the shape that says "spent, not rival"; they name `process.ppid` — another live process, which is what a contender has to be. The mutation that turns the new test red is deleting the spent-claim disjunct from the reclaim decision: the second acquire then waits out its whole timeout exactly as the report describes.
…ts own release
The swallow written last round was too wide. `release()` does two things: it restores the host's own
`~/Library/Developer/XCTestDevices` from the backup, and it gives the device-set lock back. The
second is the one whose failure a teardown can afford, because the claim is spent and a reclaim from
this process now reads it as dead. The first is a fact about this machine — without it the symlink
stays pointed at the agent-device set and every later `simctl` run sees the wrong devices — and it
was going into the same `catch {}` that runner-session and runner-disposal already had.
Both sites now call one handle method, `releaseBestEffort`, which drops only an AppError carrying
`ownerReleaseUnverified` and rethrows anything else; `release` stays strict for the build path, where
`withProcessLock` owns which of two failures gets reported. A test where the restore's `renameSync`
answers EACCES pins the rethrow, and the mutation is the bare `catch {}` returning. The standalone
best-effort helper is gone, so there is one place that decides what a teardown forgives.
The size ratchet refused this PR's six-line growth in `runner-session.test.ts`, which is 1,955 lines at the merge-base and already over the tripwire, and its own remedy is the right one here: the handle the launch hands the session was being fabricated four times over, twice in that file alone, with a per-file spy that each file then counted calls on. It now lives once in `runner-session-fixtures.ts`, the module this test family already shares, and the family is five lines shorter than it was. One spy answers both give-backs on purpose. The session-level tests ask whether the host's device set came back when a launch failed or a session was disposed; which of the two doors it came back through is the thing this PR changed, and that is pinned where the handle is made, in `runner-device-set.test.ts`, where each door is a separate test.
4f18834 to
d055691
Compare
|
Reviewed at 4f18834, as a follow-up to the review at bb64eac. The delta adjusts the spent-claim commit and adds one shared device-set redirect double, but the redirect give-back problem is still open. In runner-device-set.ts#L149, The no-redirect branch looks like it has a related gap. At runner-device-set.ts#L109, Could the redirect handle have one ordered give-back instead: restore, then release, where the restore error always wins and an unverified release is only logged? Then Two doc fixes. The JSDoc at runner-session-fixtures.ts#L189 has a garbled sentence ("the the host's device set came back, and which door..."). The PR body says "Four files; one is outside host-kit", but the diff touches 30 files: All checks passed at 4f18834 and there are no conflicts. The newer head d055691 is the same patch rebased onto main, so this applies to it unchanged. Next: make the restore error win in |
…e-backs
The close-finalization and session-close tests arrived from `main` answering the device-set redirect
with `{ release }`, which was the entire handle the day they were written. Disposal now asks the
session for `releaseBestEffort`, so those tests failed the moment this branch met them:
`session.simulatorSetRedirect?.releaseBestEffort is not a function`. The double is what is incomplete
here, not the teardown; guarding the call for a handle that does not implement its own type would only
teach the next caller to hand one over.
Both files now take the handle from `runner-session-fixtures.ts`, the module their neighbours already
use for it, and drop the per-file spy that only ever appeared in the setup. That the gap went
unnoticed by typecheck is the untyped `vi.fn()` standing in for `acquireXcodebuildSimulatorSetRedirect`,
which answers anything; every redirect double in the tree now references the shared handle, and the
`{ release }` literals left are the type's own field and the object the owning module builds.
|
Rebased on Those tests caught a real gap between the two branches. The double is what is incomplete, not the teardown, so Why typecheck passed over it: Gate at |
|
Checked at f334710. The new commit only moves the two close tests onto the shared redirect double, and In All checks pass and there are no conflicts. Next: make the restore error win in one ordered give-back, and add the test where both the rename and the |
A pid identifies a process, and the spent-claim rule reads a record that names one. Two bundles of `process-lock.ts` loaded in the same process — a nested install, a vendored copy in another package's setup — share that pid and its start time, and neither can see the other's `liveClaimTokens`. The rule as written would therefore read the other copy's live claim as a spent one and clear a lock somebody is holding, which is a worse failure than the 30 s wait the rule exists to end. The record now carries which loading issued the claim, and a spent claim requires that id to be this one. A record without it — written before claims carried an issuer, or by code that never did — is not evidence of a spent claim and stays subject to the liveness answer, so nothing that reclaimed a lock before stops reclaiming it now. The field is optional in the parser rather than part of the required shape, because a record from an older daemon must still parse rather than read as unreadable.
`release()` reconciled the host's device set inside a `try` and handed the lock back in a `finally`, so when both failed — and they fail together, being two writes to the same directory — the `finally` threw `ownerReleaseUnverified` over the restore's EACCES, and the best-effort door then dropped it as the one failure it is allowed to drop. A `~/Library/Developer/XCTestDevices` left pointing at this simulator's set is a fact about the machine that outlives the request, and the only report named the lock instead. The strict door was wrong too, in the other direction: it reported a lock problem for a restore problem. There is one give-back now, and it runs in one order: restore, then release. The restore failure always outranks the release, and the lock goes back regardless so the next acquire does not wait on a claim nobody is acting on. Whatever the release could not do is recorded either way, with the unverified release left to the log at the lock that already keeps it, so the two doors differ in exactly one thing: whether the caller also throws that failure. The same rule reaches the two paths that have no handle to give back. The simulator whose set already is `XCTestDevices` used to run its release inside the `try`, where an unverified ownership check fell into the catch, reconciled, released again, and raised "Failed to redirect XCTest device set path" for a redirect that was never needed; it now carries on and the lock's own log line stands. And a redirect that never got installed reports the redirect even when the clean-up it runs on the way out fails too, instead of the clean-up taking the report. The precedence rule stays in the handle rather than moving into `withProcessLock`: that helper knows about locks, and which of "the host's device set could not be restored" and "the claim could not be verified" a caller should hear is a fact about this redirect, not about locking. The rule was checked where it bites. Both faults at once — `renameSync` refusing the restore and the record refusing the unlink — now reach either door as EACCES, with the lock still standing as proof the pair really happened; reverting the order to the old `finally` turns that test red through both doors. A no-redirect simulator with an unverifiable release answers a null handle, and putting the release back inside the `try` brings the misleading message back verbatim. A redirect whose install and clean-up both fail still says "Failed to redirect XCTest device set path" with the install's EPERM inside it. The redirect double's doc comment, which had lost its way between two edits, says what it now means, and the CHANGELOG describes the `withProcessLock` migration this PR carries rather than only the lock fix that started it.
|
Reviewed at The restore error now wins, and there is one ordered give-back. Where the precedence rule lives: the handle, not The no-redirect branch followed the same rule. It ran The new tests. Both faults at once (restore The bundled-copy question from Doc fixes. The fixtures comment lost a clause mid-edit and now says what the double actually does. The body overstated the diff as four files; it says 32 and names the migration, and the CHANGELOG gained the entries it was missing: the Gate at |
|
Reviewed at 868d0ad, as a follow-up to the review at f334710. The handle's give-back and the no-redirect branch now let a restore error win, but the install-failure catch still hides one. In Would one ordered give-back helper do for all three sites? It would restore, then release, and return All 18 checks pass and there are no conflicts. Next: surface the restore error from the install catch, with its test. |
Summary
acquireProcessLockreleased its lock directory unconditionally, so a contender that reclaimed the lockwhile the previous holder was away had that holder delete the exclusion belonging to whoever holds it
now. Release now removes only while
owner.jsonstill names the acquirer and reports a typedownerReleaseUnverifiedotherwise; an unreadable record is an owner of unknown identity, not an absentone, and no longer ages out at the grace window. A reclaim re-decides in place, and a record naming this
live process is dead once the claim inside it is spent — claims are issued per acquisition and stamped
with the loading of this module that issued them, so a second bundled copy cannot have a live claim
cleared under it.
Lock-and-run callers stop answering "which of my two failures does the caller hear" by hand: the work
inside the lock outranks a lock that could not be handed back, and work that succeeded still reports it.
That moves the Apple runner's artifact, cache, lease, session and disposal paths, the
managed-allocation and device-claim stores, atomic publishes, the Swift recording cache and the
agent-browser setup onto
withProcessLock, and so changes which error those callers report. The XCTestdevice-set redirect adds the order inside its own give-back: restore, then release, restore first.
Closes #2523. 32 files across
host-kit,managed-allocation,capture-kit,platform-apple,platform-weband the daemon.Validation
Rebased on
origin/mainate3cbc91a1a; tested at868d0ad713.pnpm check:affected --runpassed: 622test files, 4648 tests, typecheck, lint, layering and Fallow clean; the two
scripts/gates it does notselect pass locally.
Mutations, one at a time, each red only where claimed:
finallythat let the release outrank the restoretryreleaseBestEffortswallow every errorEACCESrestore test goes quietGaps. The wedge needs a refused
unlink, so it is fault-injected: no device run yieldsEACCESon itsown, and the live coverage is the CI lanes that build and launch through this redirect. The two-loading
case is pinned with a foreign issuer id, not a second real bundle. No win32 evidence here.