Skip to content

fix(record): bound a cold video-validator compile by the stop's finalize budget - #4150

Merged
thymikee merged 11 commits into
mainfrom
fix/ios-record-stop-timeout
Nov 3, 2026
Merged

thymikee merged 11 commits into
mainfrom
fix/ios-record-stop-timeout

Conversation

@thymikee

@thymikee thymikee commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Summary

Since #4065 moved recording into smoke:capture-close, the iOS smoke lane failed 3 times at record stop with xcrun timed out after 30000ms. It is swiftc doing a cold compile of the AVFoundation video validator, under a fixed 30 s bound.

record stop already has a 70 s finalization budget meant to cover this compile (#3697). The finalizer now hands it to the playability checks. Callers share one compile, each waits within its own budget, and the last to leave cancels it with SIGINT, so swift-driver also stops its frontend jobs (SIGKILL left them running). A spent budget, or a validator run that times out, reads as not playable. The overlay decides on the verdict the wait returns. 9 files.

No-deadline callers keep the fixed 30 s compile wait: Android pullPlayable (platform-runtime-screen-recording-android-host.ts) and the live-device assertions (test/integration/live-device-e2e/assertions.ts) call isPlayableVideo without a deadline.

Validation

  • pnpm check:affected --run --head a1af133177 (rebased on main): passed, exactCommit=true. Earlier unrelated flakes (stale-ref-warning, Revyl stray GET /, fixed on main by refactor(providers): share one lease table across cloud providers #4158) passed on rerun.
  • New tests fail on reverted source.
  • Live, iPhone 17e simulator, cold isolated Swift cache, CPU oversubscribed (see PR comment):
    • At cdf951620a, record stop passed with a 66.5 s compile (stop 73.2 s).
    • A forced 5 s abandon returned video_validator_compile_timeout. No swiftc/swift-frontend, .lock or temp dir was left, and the next stop compiled and passed.
  • Self-review: cold reread only.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agent-device-website Ready Ready Preview Nov 2, 2026 9:10pm UTC

Request Review

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.70 MB 5.70 MB +960 B
Package (unpacked) 5.70 MB 5.70 MB +960 B
Package (download) 1.74 MB 1.74 MB +377 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.6 ms 33.0 ms +4.4 ms
CLI --help 61.3 ms 67.3 ms +5.9 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@thymikee
thymikee force-pushed the fix/ios-record-stop-timeout branch from 8e2f9fe to 08f3f3d Compare November 1, 2026 20:28

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Apex Sentinel

Verdict: approve

Since the last review: 4 new · 1 still open · 5 no longer raised

5 finding(s): 3 minor, 2 nit. Verdict: approve.

🟡 Minor (3)

Where Finding
packages/capture-kit/src/recording/overlay.ts:120 🆕 exportOverran is not set when the composited video fails playability verification before the budget expires
packages/capture-kit/src/recording/video.ts:117 🆕 Validator-run timeout now returns false before the isSwiftVideoValidatorUnavailable fallback that used to see its stderr
packages/capture-kit/src/recording/video.ts:138 🆕 waitForValidatorCompile cancels a compile it started without incrementing waiters first
🔵 Nit (2)
Where Finding
packages/capture-kit/src/recording/swift-cache.ts:19 🆕 Comment on kill describes only the timeout path, but SWIFTC_STOP now also serves request cancellation
packages/capture-kit/src/recording/video.ts:50 Stale doc comment above videoValidatorCompiles still describes the removed timeout behavior

Reply under an inline comment with @apex-sentinel-staging note ... or @apex-sentinel-staging pivot ... to re-check that finding.
Comment @apex-sentinel-staging review on this conversation to run a full review.

Earlier reviews

Review of cdf95162 · 02.11.2026 20:48 UTC

Apex Sentinel

Verdict: approve

Since the last review: 3 new · 3 still open · 6 no longer raised

6 finding(s): 4 minor, 2 nit. Verdict: approve.

🟡 Minor (4)

Where Finding
packages/capture-kit/src/recording/overlay.ts:116 🆕 Post-export playability check is now clamped to the 5 s exit grace, so a slow validator discards a good composite
packages/capture-kit/src/recording/video.ts:193 Concurrent validator-compile waiters lose the shared compile and reap the in-flight swiftc after the first timeout
packages/capture-kit/src/recording/video.ts:194 🆕 Abandoning a shared compile now also stops the compile that previously completed and was cached
src/platform-runtime-screen-recording-finalizer-host.ts:36 A budget spent before the validator runs is reported to the user as an unplayable recording
🔵 Nit (2)
Where Finding
packages/capture-kit/src/recording/video.ts:105 🆕 isPlayableVideo catch clause gains a body with no comment; the 30s/10s split lives only in tests
packages/capture-kit/src/recording/video.ts:193 Validator-compile cancellation is not recorded as a distinct command failure reason
Review of afefe6b0 · 02.11.2026 20:18 UTC

Apex Sentinel

Verdict: request_changes

Since the last review: 7 new · 2 still open · 3 no longer raised

9 finding(s): 1 major, 4 minor, 4 nit. Verdict: request_changes.

Required fixes

  • waitForPlayableVideo now rejects with the compile-timeout error, skipping every remaining poll and bypassing the unplayableRecording hint

🟠 Major (1)

Where Finding
packages/capture-kit/src/recording/video.ts:230 🆕 waitForPlayableVideo now rejects with the compile-timeout error, skipping every remaining poll and bypassing the unplayableRecording hint

🟡 Minor (4)

Where Finding
packages/capture-kit/src/recording/video.ts:106 🆕 A validator run that hits its own bound throws for every caller with 10 s+ of budget left
packages/capture-kit/src/recording/video.ts:110 Validator run timeout mapped to false hides a genuinely broken video when the deadline had only milliseconds left
packages/capture-kit/src/recording/video.ts:138 Shared compile drops the 30s timeoutMs, so the Swift-cache lock is now held on a 120s bound
packages/capture-kit/src/recording/video.ts:172 🆕 Unreferenced VALIDATOR_COMPILE_WAIT_MS in the error the caller receives
🔵 Nit (4)
Where Finding
packages/capture-kit/src/recording/swift-cache.ts:90 🆕 compileSwiftSourceFile (the overlay export compile) gains no cancel path, so the SIGINT cleanup this PR adds stops at the validator
packages/capture-kit/src/recording/swift-cache.ts:177 🆕 Swift cache lock acquire timeout still equals the compile timeout, so a canceled wait can make another process's healthy compile time this one out
packages/capture-kit/src/recording/video.ts:130 🆕 A validator compile interrupted by another caller now reads as 'playable' instead of failing
packages/capture-kit/src/recording/video.ts:154 🆕 Shared-compile bookkeeping relies on a settled flag that races the last-waiter abort, so a completed compile can be dropped from the map and re-joined on the next call
Review of 009d67cd · 02.11.2026 19:50 UTC

Apex Sentinel

Verdict: approve

Since the last review: 3 new · 2 still open · 3 no longer raised

5 finding(s): 1 minor, 4 nit. Verdict: approve.

🟡 Minor (1)

Where Finding
packages/capture-kit/src/recording/overlay.ts:86 🆕 The overlay's own compile lost its 30 s bound and inherits the finalization budget, so it can overrun the 90 s daemon envelope
🔵 Nit (4)
Where Finding
packages/capture-kit/src/recording/video.ts:115 🆕 A validator run SIGKILLed at a clamped deadline reads as a Swift-validator failure and reports an unplayable recording as playable
packages/capture-kit/src/recording/video.ts:137 🆕 Shared validator compile drops the compiler's own timeout bound
packages/capture-kit/src/recording/overlay.ts:120 Overlay reports a spent budget as an export overrun after the playability wait, masking the compile-timeout error's details
packages/capture-kit/src/recording/video.ts:93 webm skips the new spent-budget guard, unlike mp4
Review of 43963597 · 02.11.2026 19:18 UTC

Apex Sentinel

Verdict: approve

Since the last review: 3 new · 2 still open · 13 no longer raised

5 finding(s): 2 minor, 3 nit. Verdict: approve.

🟡 Minor (2)

Where Finding
packages/capture-kit/src/recording/video.ts:103 🆕 Android's playable probe now waits up to 30 s for a validator compile where it used to get a cached path
packages/capture-kit/src/recording/video.ts:193 A compile aborted for want of waiters can fail every validator check in the next 2 s, and the overlay treats that as an unplayable video
🔵 Nit (3)
Where Finding
packages/capture-kit/src/recording/overlay.ts:120 🆕 Overlay discards a composited video when a validator check throws, instead of using the deadline verdict the wait returns
packages/capture-kit/src/recording/video.ts:103 🆕 Validator run timeout is clamped against the caller deadline, not just the 10 s default
packages/capture-kit/src/recording/video.ts:152 The cache-root key is computed twice, once to insert and once to clean up
Review of 68380584 · time not recorded

Apex Sentinel

Verdict: request_changes

Since the last review: 4 new · 2 still open · 9 no longer raised

6 finding(s): 1 major, 2 minor, 3 nit. Verdict: request_changes. reached max rounds (1).

Required fixes

  • Decide between: Deadline-clamped validator run rejects and escapes as an exec timeout instead of reading as not playable / waitForPlayableVideo's new deadline makes the overlay's post-export verification fail spuriously on slow exports

🟠 Major (1)

Where Finding
packages/capture-kit/src/recording/video.ts:108 🆕 Deadline-clamped validator run rejects and escapes as an exec timeout instead of reading as not playable

🟡 Minor (2)

Where Finding
packages/capture-kit/src/recording/video.ts:136 🆕 Cold validator compile is no longer bounded by swiftc's own timeout; the wait is bounded only while a caller waits
packages/capture-kit/src/recording/video.ts:224 waitForPlayableVideo's new deadline makes the overlay's post-export verification fail spuriously on slow exports
🔵 Nit (3)
Where Finding
packages/capture-kit/src/recording/video.ts:212 waitForPlayableVideo now runs attempts + 1 validator passes, doubling the validator run for an unplayable file
packages/capture-kit/src/recording/video.ts:103 🆕 Run timeout clamps the deadline twice in one nested expression
packages/capture-kit/src/recording/video.ts:143 🆕 shared.settled is set from a .then(...) handler on shared.compiled, and waitForValidatorCompile's finally r…

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 5 new findings on 08f3f3d. The verdict and the full list are in the summary comment.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 8 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 6 new findings on 3a127d3. The verdict and the full list are in the summary comment.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 4 new findings and 1 moved to a new line on 6838058. The verdict and the full list are in the summary comment.

@thymikee

thymikee commented Nov 1, 2026

Copy link
Copy Markdown
Member Author

The fix reads well at 6838058, but the new cancel path has no live run yet, so it is not ready to merge. CI shows no failing job; the only non-passing check is the review bot's pending request-changes status.

The only live measurement in the PR table is from the first commit. The head now uses a shared compile where the last waiter aborts swiftc (video.ts:130), and record stop is a device-facing path. An abandoned compile is the one new behavior that can leave an orphaned swiftc, a stale .lock dir or a temp output on a real host. Please run once at 6838058 on a cold Swift cache with the CPU oversubscribed. First, show an iOS record start then record stop that passes with the compile over 30 s, with output showing the stop time and the compile time. Second, force an abandon (for example a throwaway Deadline.fromTimeoutMs(5_000) through isPlayableVideo) and show the video_validator_compile_timeout error. Then show that pgrep -fl swiftc is empty and no .lock or temp dir is left under the Swift cache bin directory, and that a following stop compiles again and passes.

Not blocking, and you can take or leave these: (1) if the compile ends with a few ms of budget left, the clamped validator run in video.ts:105 times out and isPlayableVideo throws instead of returning false, so return false when the deadline is nearly spent or fix the doc comment; (2) Android pullPlayable in platform-runtime-screen-recording-android-host.ts:112 and assertions.ts:92 call isPlayableVideo with no deadline, so they keep the fixed 30 s compile wait, which is worth a line in the PR body; (3) overlay.ts:120 ignores the boolean from waitForPlayableVideo and pays an extra validator pass, so use the returned value instead of deadline.isExpired(); (4) swift-cache.test.ts:70 should assert the request-canceled error, cover a canceled waiter on another process's lock, and have one finalizer test where the compile timeout reaches complete(); (5) the comment at video.ts:49 should say abort kills this process's swiftc or stops waiting for another process's compile, and the nested Math.min at lines 105-108 can become a local const.

Of the open inline threads, the two on the clamped validator-run timeout (r4437312410, r4437338234) still apply, and three lower-priority threads (comment wording, the nested min) still apply. These do not apply at the head, so please resolve them: r4437243626, r4437312427 and r4437338242 (every waiter is bounded and the last aborts swiftc); r4437243630 (a spent budget returns an overlay warning); r4437243632 (the error carries retriable and a hint); r4437243637 (that code no longer exists); r4437312420 (the typed timeout is intended); r4437312423 (base ran the same passes); r4437312430 (the canceled rejection is consumed internally); r4437338247 (base already checked the deadline); r4437338255 (settled is set before any waiter finishes).

This review is from reading the diff. I did not run the tests or the simulator repro. I could not tie the 90 s stop envelope or the 80 s worst case to a constant on the record stop route, or confirm that built Swift modules stay cached after an abort, as the retry hint says. Before merge, please post the cold-cache live run above.

@thymikee
thymikee force-pushed the fix/ios-record-stop-timeout branch from 6838058 to 4396359 Compare November 2, 2026 19:09
@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Rebased on main and addressed the review. Head is now 4396359760.

What changed

  • The clamped validator run reads as not playable (note 1). When the deadline clamps the run below its own 10 s bound, an exec timeout returns false instead of throwing. When the budget is already spent, it returns false before the run starts. Without a deadline, a 10 s timeout still throws.
  • Overlay (note 3). The overlay now uses the boolean that waitForPlayableVideo returns. That wait now checks once per attempt and no longer adds a pass after the last one. An export that never plays now keeps the recorded video and returns the overlay warning. Only a spent budget adds the "did not finish within its budget" wording.
  • Comment and local const (note 5). Done.
  • PR body (note 2). Added a line about the Android pullPlayable and live-device assertion callers.
  • Tests (note 4). The swift-cache cancel test now asserts the request-canceled error. A new test cancels a wait on a lock that another live process holds; the lock stays in place. A new finalizer test checks that video_validator_compile_timeout reaches the caller of complete(). New video tests cover a run the deadline cut short and the poll count.
  • New fix from the live run: SIGINT for an abandoned swiftc. The first forced abandon killed swiftc but left a swift-frontend running with ppid 1. swift-driver runs each frontend job in its own process group, so killing the driver does not reach it. I tested each signal on a throwaway cold compile. After SIGTERM or SIGKILL, the frontend kept running. After SIGINT, the driver stopped its jobs. The compile now uses kill: { signal: 'SIGINT', graceMs: 2_000 } and still sends SIGKILL after the grace period.

Live run

Setup: iPhone 17e simulator F25C7515-… (iOS 27.0), built from this checkout. The state dir was isolated, and so was AGENT_DEVICE_SWIFT_CACHE_DIR, which also holds the module cache, so every cold run started from an empty cache. The host load average was already 300–700 from other work. During each stop I added 72 yes processes on this 12-core host and killed them right after; pgrep -x yes was empty afterwards. Compile times come from the daemon's exec trace (AGENT_DEVICE_EXEC_TRACE=1).

(a) A cold record start then record stop passes with the compile over 30 s. Built at d0d88deadb, which differs from the head only in the SIGINT kill option.

open exit=0
start exit=0
stop exit=0 stop_s=54.0          {"success": true, "recording": "stopped", "recorder": "confirmed"}
+5.8s  xcrun spawn_start  timeoutMs=120000
+52.1s xcrun settled 46.3s  success          <- validator compile
+52.1s video-validator-bd93ed6ddc83c518 settled 0.1s success

(b) Forced abandon at the head, from a throwaway script that is not committed. It calls isPlayableVideo(mp4, host, Deadline.fromTimeoutMs(5_000)) on a fresh cold cache:

rejected after 5014ms: {
  "code": "COMMAND_FAILED",
  "message": "The video validator did not compile within 4997ms",
  "details": { "reason": "video_validator_compile_timeout", "timeoutMs": 4997, "retriable": true, "hint": "Run record stop again: ..." }
}
--- pgrep -fl swiftc
(pgrep exit 1)
--- swift-driver/swift-frontend for this cache
0
--- ls -la swift-cache-b2/bin
total 0
drwxr-xr-x@ 2 michal  wheel   64 Nov  2 19:19 .
drwxr-xr-x@ 6 michal  wheel  192 Nov  2 19:19 ..
--- .lock or temp entries under bin
(end)

Before the SIGINT fix, the same steps left a swift-frontend -frontend -c -primary-file …/video-validator-….swift process with ppid 1. After the abandon, the module cache kept 32 .pcm files (24 MB), so the retry hint holds.

Next stop on the same cache, at the head: the stop compiled again and passed. The compile was still over 30 s because the host load peaked near 700.

stop exit=0 stop_s=73.7          {"success": true, "recorder": "confirmed"}
+35.6s xcrun spawn_start  timeoutMs=120000
+71.9s xcrun settled 36.4s  success
+72.0s video-validator-bd93ed6ddc83c518 settled 0.1s success
bin/: video-validator-bd93ed6ddc83c518 (no .lock, no temp dir)

The session was closed and the daemon was stopped afterwards.

Review threads

I resolved the 11 threads the review says do not apply, and the 4 threads this push fixes: the two clamped-run threads, the comment wording and the nested min.

Validation

pnpm check:affected --run --head 4396359760 passed with exactCommit=true. An earlier run at the same head failed once in stale-ref-warning.test.ts on a 5 s test timeout while the host load was about 700. That test covers selector refs and does not touch recording. It passed 3 times out of 3 when run alone, and it passed in the rerun.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 3 new findings and 2 moved to a new line on 4396359. The verdict and the full list are in the summary comment.

// AVFoundation is the MP4 semantic validator. It does not reliably load WebM on supported
// macOS hosts, so WebM completion is established by its EBML document type + Segment marker.
if (container === 'webm') return true;
try {

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on 009d67c.

🆕 🟡 Minor · Android's playable probe now waits up to 30 s for a validator compile where it used to get a cached path

isPlayableVideo gained an optional deadline, and every caller that does not pass one now falls back to deadline?.remainingMs() ?? VALIDATOR_COMPILE_WAIT_MS for the compile wait (video.ts:105-108). The new videoValidatorCompiles map also no longer drops a successful compile the way the old map dropped a failed one: it keeps the entry forever, so any transient compile failure permanently costs every deadline-less caller a 30 s stall in addition to its real work (the next caller starts a fresh compile and waits the full wait again). Before this diff, the same path could not wait longer than the compile's own timeoutMs: 30_000 and returned as soon as the previous attempt's promise settled.

Suggested change

Note in the isPlayableVideo doc comment that a caller without a deadline waits a fixed 30 s for a cold compile, and consider passing a short deadline from the Android pull path so the probe stays bounded.

Details

The caller that feels it is createAndroidScreenRecordingTransport (src/platform-runtime-screen-recording-android-host.ts:112,116), unchanged by this PR: its playable: result.exitCode === 0 && (await isPlayableVideo(outputPath, host)) inside pullPlayable is the one place where the probe runs for a file it has just pulled and that may not be complete yet, so a cold cache plus a validator failure turns a quick playable: false into 30 s + 120 s before it answers.

Also raised here

Also raised at this location by javascript: isPlayableVideo's new deadline semantics are not stated for the callers that pass no deadline, which now wait a fixed 30 s for a cold compile — The new doc comment on isPlayableVideo (video.ts:88-92) describes only what happens 'With a deadline'. The two production callers that pass none — createAndroidScreenRecordingTransport (src/platform-runtime-screen-recording-android-host.ts:112) and test/integration/live-device-e2e/assertions.ts:92 — are exactly the ones whose worst case this diff changed: they now wait VALIDATOR_COMPILE_WAIT_MS (30 s) plus the compile's own params.timeoutMs ?? 120_000 default in ensureSwiftExecutable (swift-cache.ts:179), because joinValidatorCompile passes no timeoutMs at all. State in that comment what a deadline-less caller is guaranteed, or name the constant it should pass instead of relying on it.

Also raised at this location by senior-developer: The compile now runs under a silent 120 s internal bound because timeoutMs is no longer passed — joinValidatorCompile (video.ts:143-148) calls compileSwiftSourceText without timeoutMs, while the removed predecessor passed timeoutMs: 30_000. That field is the only bound ensureSwiftExecutable has on the xcrun swiftc run: const timeoutMs = params.timeoutMs ?? 120_000 (swift-cache.ts:179) arms both the lock acquire and the run. The waitMs used by waitForValidatorCompile is a wait bound only — it never reaches the child. So a lone caller without a deadline (Android pullPlayable in src/platform-runtime-screen-recording-android-host.ts:112, test/integration/live-device-e2e/assertions.ts:92) gives up after 30 s while swiftc keeps compiling up to 120 s, holding the executable lock so every concurrent compile of that key times out; and a caller with a Deadline whose isExpired() goes stale (the timer is armed from remainingMs() and nothing re-checks it, video.ts:172-192) can keep a compile alive far past its budget. …

…and 1 more (javascript), left out to fit the comment.

persona: javascript, senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

'COMMAND_FAILED',
'The composited video was not verified within the budget',
);
if (!(await waitForPlayableVideo(outputPath, { host, deadline }))) {

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · exportOverran is not set when the composited video fails playability verification before the budget expires

The old code set exportOverran = true whenever the verification step found the deadline expired. The new branch sets exportOverran = deadline.isExpired(), so when waitForPlayableVideo returns false with budget still on the clock — which is now a reachable state, because a clamped runTimeoutMs that times out, or a validator that exits nonzero, both read as not playable instead of throwing — the operator gets the bare commandDescription ("Failed to add touch overlays to the recording") with no budget hint. Worse, the details still carry the copied reason: 'video_validator_compile_timeout', retriable: true and the "Run record stop again" hint from the wrapped cause, so a run that had most of the 70 s left is described both as a plain overlay failure and as a compile timeout. Reproduce: overlay export succeeds but the composited file's validator run times out within a clamped runTimeoutMs while the deadline still has seconds left.

Suggested change

Pick one description for the not-verified path (either set exportOverran for it or drop the copied compile-timeout details) so the message and the details describe the same failure.

Details

Also raised here

Also raised at this location by javascript: Overlay's budget-overrun message now depends on the wait's false verdict coinciding with isExpired() — exportProcessedVideo replaced the two-step check (await waitForPlayableVideo(...) then deadline.isExpired()) with exportOverran = deadline.isExpired() read only after the wait resolves false (overlay.ts:120-122). Because waitForPlayableVideo now also returns false for reasons other than the budget — attempts exhausted (12 polls of 150 ms at the default, video.ts:228-236), or the validator run timing out inside isPlayableVideo (video.ts:117-125) — the exportOverran flag silently relies on 'expired' and 'not playable' being the same event. When they diverge (a composited export that fails 12 validator polls with budget left, or a validator run killed by runTimeoutMs), the user loses the the export did not finish within its 70000ms budget; a lower --fps renders faster advice and gets only the plain command description, even though the budget is what made the run too short. Say which condition sets the flag instead of inferring it from the clock.

Also raised at this location by senior-developer: New video_validator_compile_timeout reason is thrown from record stop outside the recording-runtime reason contract — waitForValidatorCompile rejects with a new reason: 'video_validator_compile_timeout' (video.ts:176-188), and finalizeScreenRecording lets it propagate: the earlier unconditional waitForPlayableVideo + isPlayableVideo pair that always threw unplayableRecording() with the contract-owned RECORDING_OUTPUT_UNPLAYABLE_REASON is replaced by one if (!(await waitForPlayableVideo(...))) (src/platform-runtime-screen-recording-finalizer-host.ts:38-40). So a cold compile that outlasts the 70 s finalize budget now fails record stop with a reason string that is defined inline in capture-kit and not part of the @agent-device/contracts/screen-recording-runtime vocabulary the other stop failures use; a client or doc that branches on the recording reasons sees a value it has never seen, and the same stop can fail with either reason depending on whether the compile or the file is at fault. …

persona: senior-developer, javascript

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

@thymikee

thymikee commented Nov 2, 2026 •

Copy link
Copy Markdown
Member Author

The code looks good at 4396359, and the earlier finding from the 6838058 review (#4150 (comment)) is settled. The finalizer's budget now bounds the cold validator compile, a run the deadline cuts short reads as not playable, and the last caller to stop waiting aborts swiftc with SIGINT and then SIGKILL, so the frontend jobs end too. All Smoke Tests jobs passed at 4396359, including the iOS lane that runs record stop. The only failing check was CodeQL Analyze (python), which was cancelled during analysis after 28 s; the diff touches no Python and the same job passed on main, so a rerun should clear it. There are no conflicts.

Not blocking, take or leave: the clamped-run regression in video.test.ts:194 uses Deadline.fromTimeoutMs(5). On a loaded host more than 5 ms can pass before isPlayableVideo reaches its expiry check, so it returns false there and never reaches the new catch at video.ts:120-122, and the test would also pass with the catch reverted. A deadline with seconds left but under the 10 s run bound, for example Deadline.fromTimeoutMs(5_000), always exercises that branch, since runCmd is mocked to throw at once.

The earlier inline threads are fixed at this head or benign as written, including the two P2 threads on the deadline-clamped run (#4150 (comment), #4150 (comment)). The newer P3 and P4 threads do not apply: the sole no-deadline waiter leaves after 30 s and aborts swiftc, which matches the old bound; a joiner in the SIGINT grace window gets its own controller; and the overlay path reuses the already compiled validator. I did not run the tests or a recording; the live compile-over-30 s and forced-abandon runs are your reports.

This review covers 4396359. The newer head 009d67c only keys the shared validator compile once per wait in video.ts, which also settles the duplicated Swift root nit, so the points above still apply.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 4 findings moved to a new line on 009d67c. The verdict and the full list are in the summary comment.

);
// A budget with no time left reads as not playable; `runCmd` would read a timeout of 0 as none.
if (runTimeoutMs <= 0) return false;
const result = await runCmd(validatorPath, [filePath], {

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on afefe6b.

🆕 🔵 Nit · A validator run SIGKILLed at a clamped deadline reads as a Swift-validator failure and reports an unplayable recording as playable

isPlayableVideo now passes a deadline-clamped timeoutMs to the validator run (video.ts:115-118), and the only thing separating "the run was killed at the caller's deadline" from "the run hit the validator's own 10 s bound" is the caller-side runTimeoutMs < VALIDATOR_RUN_TIMEOUT_MS test at video.ts:121. packages/host-kit/src/internal/exec.ts makes that test incomplete in two ways, and both are newly reachable because a short timeoutMs is only possible after this diff (the predecessor always passed a fixed 10_000).

Suggested change

Stop inferring who cut the validator run short from runTimeoutMs < VALIDATOR_RUN_TIMEOUT_MS. Classify the kill from exec's own structured state: either expose an abort/deadline reason on the error exec raises (the timedOut: true shape at exec.ts:429-433 already exists and settle() at exec.ts:246-259 already knows abort.didAbort/didTimeout), or hand the validator run an AbortSignal derived from the deadline and treat isRequestCanceledError as "not playable". Also do not feed a killed validator child's partial stderr into isSwiftVideoValidatorUnavailable: run that fallback only for a run that exited on its own.

Details
  1. createTimeoutError (exec.ts:651-670) is armed only in settle() when the child already closed (exec.ts:256), and the default stop is SIGKILL — SWIFTC_STOP is passed only for xcrun swiftc in swift-cache.ts:204, not for the validator run. So a validator run that is SIGKILLed at e.g. 400 ms and whose stdio pipes stay open (any grandchild holding them, or a stream that does not close on SIGKILL) never reaches settle(); isCommandTimeoutError is never consulted and the AppError('COMMAND_FAILED', 'The command was aborted by a deadline', { timedOut: true }) from exec.ts:429-433 propagates through the catch at video.ts:129 and out of isPlayableVideo. waitForPlayableVideo propagates it too, so finalizeScreenRecording (src/platform-runtime-screen-recording-finalizer-host.ts:38) fails the whole record stop with a bare COMMAND_FAILED carrying no reason, instead of the documented and tested "reads as not playable" behavior (video.test.ts: 'a validator run the caller deadline cuts short reads as not playable' asserts only the close-on-kill path, because it mocks runCmd). A retried stop then succeeds, which is exactly the symptom the PR set out to remove.

  2. The same abort error carries details: { timedOut: true } and no timeoutMs, so isCommandTimeoutError (exec.ts:677-683, which checks only typeof error.details?.timeoutMs === 'number') answers false for it as well.

  3. When the kill does settle, createTimeoutError puts the killed child's captured stderr/stdout into details (exec.ts:662-669). For SIGKILL there is no flush, so a truncated AVFoundation/ObjC diagnostic is exactly what lands there. The video under test was written by the same AVFoundation helper that has now died, and isSwiftVideoValidatorUnavailable (video.ts:240-245) matches unable to find utility "swiftc" — if that substring survives in the killed child's partial stderr, isPlayableVideo returns true (video.ts:128) for an unplayable composited video, and overlay.ts:120-124 renames the corrupt output over the original recording. The 10 s-bound case this regex was written for flushes its stderr and exits normally, so this mixing of killed-child output into the fallback is new to this diff.

The distinguishing state already exists in exec: !abortSignal.aborted && !didTimeout. Classifying from the structured error rather than the caller's own timeout arithmetic is what AGENTS.md asks for ("Use typed reasons and details for behavior. Do not match error text.").

Also raised here

…and 1 more (senior-developer), left out to fit the comment.

persona: javascript, senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

let compiled = videoValidatorExecutablePaths.get(cacheRoot);
if (!compiled) {
compiled = compileSwiftSourceText({
function joinValidatorCompile(host: HostEnvironment, cacheRoot: string): SharedValidatorCompile {

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on cdf9516.

🟡 Minor · Shared compile drops the 30s timeoutMs, so the Swift-cache lock is now held on a 120s bound

getVideoValidatorExecutablePath compiled with timeoutMs: 30_000, and ensureSwiftExecutable uses that one value for both the lock acquire and the xcrun swiftc run (swift-cache.ts:179-181,201-206). joinValidatorCompile passes no timeoutMs, so every validator compile now runs with the 120s default: a process that merely waits for the <executable>.lock held by another daemon's compile tolerates up to 120s of lock contention where it previously gave up after 30s, and the compile itself may run 4x longer before the run's own bound fires (bounded only by cancellation when the last waiter leaves).

Suggested change
function joinValidatorCompile(host: HostEnvironment, cacheRoot: string): SharedValidatorCompile {
compiled: compileSwiftSourceText({
host,
source: VIDEO_VALIDATION_SCRIPT,
cacheName: 'video-validator',
timeoutMs: VALIDATOR_COMPILE_WAIT_MS,
signal: cancel.signal,
}),

Suggested change

Pass an explicit timeoutMs to the shared compileSwiftSourceText call (or split lock timeout from compile timeout in ensureSwiftExecutable) so the validator's lock wait keeps a bound close to the 30s it had before.

Details

persona: javascript

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Follow-up on the latest bot round. Head is now afefe6b000. The previous head, 009d67cde5, passed every CI check, including the iOS and macOS smoke tests.

Changed

  • 009d67cde5: the shared validator compile is keyed once per wait instead of calling getSwiftCacheRoot twice. The runTimeoutMs <= 0 comment now says that a spent budget reads as not playable.
  • afefe6b000 (doc only): the isPlayableVideo doc comment says that a WebM verdict is a structural read with no validator, so the deadline does not apply to it. I did not move the guard. Refusing a finished WebM because the budget ran out would be wrong, because its check takes no time. (r4447003226)

Not changed, after verification

  • r4447003203 (a deadline kill read as a validator failure). This does not hold at this head. With real runCmd, a run killed at its timeoutMs rejects in about 300 ms with a numeric details.timeoutMs. That is true for a plain child and also for a grandchild that keeps the pipes open. isCommandTimeoutError was true in both cases:
    plain child rejected after 308ms {"timeoutMs":300,"stderr":"","isCommandTimeoutError":true}
    grandchild holds pipes rejected after 306ms {"timeoutMs":300,"stderr":"partial\n","isCommandTimeoutError":true}
    
    The timedOut: true error the finding cites does not exist in the async path. For a clamped run, isPlayableVideo returns false in the runCmd catch, before the stderr fallback can run. The existing test fails if that line is removed. Separately, and already true on main: the isSwiftVideoValidatorUnavailable regex never matches real messages. Its trailing \b after a closing quote needs a word character next, so xcrun: error: unable to find utility "swiftc", not a developer tool… and no such module 'AVFoundation' both test false. That is out of scope here, and I can file it separately.
  • r4447003193 (overlay compile lost its 30 s bound). On main, the overlay compile was already bounded by Math.max(1, deadline.remainingMs()), and this PR does not change that line. For MP4, isPlayableVideo always runs the validator, compiling it first if needed, before the overlay, so the overlay still starts with what is left of the budget, as on main. A timed-out overlay compile now also gets the SIGINT stop, because SWIFTC_STOP lives in ensureSwiftExecutable, which compileSwiftSourceFile shares.

pnpm check:affected --run --head afefe6b000 passed with exactCommit=true.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 8 findings moved to a new line on afefe6b. The verdict and the full list are in the summary comment.

host,
deadline?.remainingMs() ?? VALIDATOR_COMPILE_WAIT_MS,
);
const runTimeoutMs = Math.min(

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on cdf9516.

🟡 Minor · Validator run timeout mapped to false hides a genuinely broken video when the deadline had only milliseconds left

packages/capture-kit/src/recording/video.ts:110-125: runTimeoutMs = Math.min(VALIDATOR_RUN_TIMEOUT_MS, deadline?.remainingMs()), and any timeout from that run becomes undefined → return false whenever runTimeoutMs < VALIDATOR_RUN_TIMEOUT_MS. remainingMs() is exact and can be 1–50ms with a live deadline, so the validator is killed almost at once and the answer is "not playable". waitForPlayableVideo then reports false, and src/platform-runtime-screen-recording-finalizer-host.ts:38-40 turns that into unplayableRecording ("recording was not finalized into a playable video", reason RECORDING_OUTPUT_UNPLAYABLE_REASON) for a file that was most likely fine — the finalizer's own test 'a recording confirmed playable as the budget runs out is not refused by a second check' states that a spent budget reads as not playable, so this mislabelled path is reachable at exactly that boundary. A run cut short with almost no budget left is a no verdict (like the compile-timeout error above it, which stays a retriable error), not evidence the video is broken: the error already carries the killed validator's exit, so the two cases could be told apart.

Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Final round. Head is now cdf951620a.

Changed (r4447277445): every validator run timeout now reads as not playable. The deadline-clamp condition is gone. Before this change, a run that hit its own 10 s bound threw when the caller had 10 s or more of budget left, or had no deadline. A run the deadline cut short already returned false. main also throws on the 10 s bound, so this is a consistency fix, not a regression fix. For record stop, a hung validator run now ends in the retriable unplayableRecording error instead of a bare exec timeout. The test covers both bounds.

Left as is: waitForPlayableVideo still rejects with video_validator_compile_timeout when the compile outlasts the budget. That is the typed, retriable timeout the maintainer review called intended (r4437312420).

Live run at cdf951620a

Setup: iPhone 17e simulator, built from this checkout, with an isolated state dir and an isolated AGENT_DEVICE_SWIFT_CACHE_DIR (which also holds the module cache). The host was idle (load about 5), so during each stop I ran 120 short-lived yes processes on this 12-core host and killed them afterwards. pgrep -x yes was empty afterwards. Compile times come from the daemon's exec trace.

(a) A cold record start then record stop passes with the compile over 30 s:

stop exit=0 stop_s=73.2          {"success":true,"recording":"stopped","recorder":"confirmed"}
+3.9s  xcrun spawn_start  timeoutMs=120000
+70.3s xcrun settled 66.5s  success
+70.4s video-validator-bd93ed6ddc83c518 spawn_start  timeoutMs=3148   <- run clamped to the budget left
+70.4s video-validator-bd93ed6ddc83c518 settled 0.1s success

(b) Forced abandon (a throwaway isPlayableVideo(mp4, host, Deadline.fromTimeoutMs(5_000)) on a fresh cold cache):

rejected after 5008ms: {"code":"COMMAND_FAILED","message":"The video validator did not compile within 4999ms",
  "details":{"reason":"video_validator_compile_timeout","timeoutMs":4999,"retriable":true,"hint":"Run record stop again: ..."}}
--- pgrep -fl swiftc
(pgrep exit 1)
--- swift-driver/swift-frontend for this cache: 0
--- ls -la swift-cache-final-b/bin
total 0
drwxr-xr-x@ 2 michal  wheel   64 Nov  2 21:31 .
drwxr-xr-x@ 6 michal  wheel  192 Nov  2 21:30 ..
--- .lock or temp entries under bin:
(end)
module-cache pcm files: 98

Next stop on the same cache: it compiled again and passed.

stop exit=0 stop_s=41.8          {"success":true,"recording":"stopped","recorder":"confirmed"}
+3.1s  xcrun spawn_start  timeoutMs=120000
+39.4s xcrun settled 36.4s  success
bin/: video-validator-bd93ed6ddc83c518 (no .lock, no temp dir)

Afterwards I closed the session and stopped the daemon. pnpm check:affected --run --head cdf951620a passed with exactCommit=true.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding and 5 moved to a new line on cdf9516. The verdict and the full list are in the summary comment.

@@ -117,13 +117,9 @@ async function exportProcessedVideo(params: {
exportOverran = deadline.remainingMs() <= HELPER_EXIT_GRACE_MS;

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on a1af133.

🆕 🟡 Minor · Post-export playability check is now clamped to the 5 s exit grace, so a slow validator discards a good composite

Passing deadline into waitForPlayableVideo (overlay.ts:120) now propagates it into isPlayableVideo, where the validator run is clamped to Math.min(10_000, deadline.remainingMs()) and a run that times out reads as not playable (video.ts:111-125). By construction the helper exits with about HELPER_EXIT_GRACE_MS = 5 000 ms left (overlay.ts:94-95, 114), so the very first validation attempt gets ≤5 s instead of the fixed 10 s the predecessor gave it, and every later poll gets less. On a loaded host where AVFoundation needs ~5.5 s to load the composited file, the attempt times out → false → 150 ms sleep → the deadline expires → waitForPlayableVideo returns false → exportOverran = deadline.isExpired() is true → the verified-composite rename at overlay.ts:124 never happens and the user gets "the export did not finish within its 70000ms budget" with the touch overlay silently dropped, even though the composite was fine. The predecessor ran the same check with the full 10 s. Either give the post-export check its own floor (e.g. Math.max(1, deadline.remainingMs()) with a reserved validation slice inside HELPER_EXIT_GRACE_MS), or document that the exit grace must cover the validator run.

Suggested change

Reserve part of HELPER_EXIT_GRACE_MS for the post-export playability check (or keep that one check on the fixed VALIDATOR_RUN_TIMEOUT_MS) so a composite that is playable but slow to validate is not discarded at the end of the budget.

Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

} catch (error) {
videoValidatorExecutablePaths.delete(cacheRoot);
throw error;
return await Promise.race([shared.compiled, expired]);

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on a1af133.

🆕 🟡 Minor · Abandoning a shared compile now also stops the compile that previously completed and was cached

video.ts#getVideoValidatorExecutablePath used to resolve/publish whatever compileSwiftSourceText produced, and a failed or timed-out compile was never killed — it kept running and filled the Swift module cache (which is exactly what the new hint claims: "the Swift modules this compile already built stay cached"). The new waitForValidatorCompile finally-block calls shared.cancel.abort() as soon as the last waiter leaves, which aborts runCmd for this process's own compile too and SIGINTs swiftc mid-run (swift-cache.ts:203-204, SWIFTC_STOP). Concrete case: two record stop calls overlap and the second has a small remaining finalize budget, or a caller polls and gives up — the give-up kills the compile that would have finished and been reused, so the next stop recompiles from scratch and the retry the hint promises takes longer instead of sooner. Please gate the abort on lock ownership (only stop waiting for another process's compile; leave a compile this process owns running), or say in the comment/hint that abandoning throws away the partial work.

Suggested change

Do not cancel a compile this process owns when the last waiter gives up: only stop waiting for another process's compile, or record in the comment and the error hint that giving up discards the in-flight compile so the hint's caching claim stays true.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

input: Parameters<ScreenRecordingRuntimeHost['finalize']['complete']>[0],
) {
const overlayDeadline = startRecordingOverlayBudget();
const finalizeDeadline = startRecordingOverlayBudget();

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on a1af133.

🟡 Minor · A budget spent before the validator runs is reported to the user as an unplayable recording

In the finalizer, a false from waitForPlayableVideo always throws unplayableRecording(...) (src/platform-runtime-screen-recording-finalizer-host.ts:38-40), whose message is "recording was not finalized into a playable video". With the deadline now threaded into isPlayableVideo, false is also what a never-attempted validation returns: deadline?.isExpired() (video.ts:106) or runTimeoutMs <= 0 (video.ts:116) short-circuit before the validator is spawned at all — that is the new "spent budget reads as not playable" contract asserted at video.test.ts:158-167. So a perfectly playable recording whose 70 s budget was spent by waitForStableFile plus a slow-but-successful compile and poll is refused with a message and hint ("Run record stop again: a recorder that is still finalizing its file…", lines 85-88) that point the user at the recorder rather than at the host load. The compile-timeout case was given its own retriable error (video.ts:176-188); the spent-budget-before-run case, which is the same host-load condition, was not.

Suggested change

Distinguish "the budget left no time to validate" from "the validator refused the file" — e.g. have the wait report the reason it stopped (a distinct retriable reason such as video_validator_budget_exhausted) so the user-facing hint names host load / retry rather than a recorder that is still finalizing.

Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

// AVFoundation is the MP4 semantic validator. It does not reliably load WebM on supported
// macOS hosts, so WebM completion is established by its EBML document type + Segment marker.
if (container === 'webm') return true;
try {

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on a1af133.

🆕 🔵 Nit · isPlayableVideo catch clause gains a body with no comment; the 30s/10s split lives only in tests

Two things a reader has to reconstruct here. (1) const runTimeoutMs = Math.min(...) (line 108) and the separate if (runTimeoutMs <= 0) return false; (line 111) look redundant — the guard is only needed because Math.min can return 0 and runCmd reads timeoutMs: 0 as "no timeout". The explanation sits above the later runCmd call (line 113), so the non-obvious line itself is bare. (2) The two new constants are documented as caller-side bounds (VALIDATOR_COMPILE_WAIT_MS comment mentions "waits … until that deadline instead") but nothing says that the compile itself is now launched with no timeoutMs at all, so the real wall-clock bound on swiftc for a deadline-less caller is ensureSwiftExecutable's 120s default rather than 30s; that split is asserted only in video.test.ts (expect(compile.timeoutMs).toBeUndefined()). A short comment on each line would keep the next change from "simplifying" the guard away or re-adding a timeoutMs.

Suggested change

Add a one-line comment on the runTimeoutMs computation saying Math.min can yield 0 and runCmd treats 0 as no timeout, and note on the new constants that the compile is started without timeoutMs so the caller wait, not a compile timeout, is the bound.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

waitMs,
);
});
try {

@apex-sentinel-staging apex-sentinel-staging Bot Nov 2, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer raised on a1af133.

🔵 Nit · Validator-compile cancellation is not recorded as a distinct command failure reason

waitForValidatorCompile aborts its AbortController when the last waiter leaves (video.ts:201), and the resulting rejection of compileSwiftSourceText surfaces as the generic request-canceled error. The PR introduces a distinct, retriable reason for the sibling outcome on the same path — video_validator_compile_timeout (video.ts:172-192) — but the abandoned-compile case that the PR's own test exercises (swift-cache.test.ts:174-197 asserts rejects.toSatisfy(isRequestCanceledError)) carries no reason, hint, or retriable marker. A caller or log that sees the request-canceled error cannot tell "the finalize budget was spent and we deliberately abandoned a cold compile" from an ordinary per-request cancellation, and cannot tell the user to run record stop again the way the compile-timeout hint does. The deadline path avoids this only because the local expired timer usually rejects the race first; the cancellation error is what the shared compile itself settles with. Worth an explicit reason at the abort site so the two budget outcomes are distinguishable.

Suggested change

At the abort site in waitForValidatorCompile (and/or in ensureSwiftExecutable's canceled path), reject with a structured reason (e.g. reason: 'video_validator_compile_abandoned', retriable: true, the same 'Run record stop again' hint) instead of letting the generic request-canceled error reach the caller, so the abandoned-compile outcome is distinguishable from an ordinary request cancellation.

Details

persona: javascript

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

thymikee and others added 8 commits November 2, 2026 21:59
…ize budget

`record stop` compiles the AVFoundation video validator on a cold Swift
cache under a fixed 30 s bound. On a loaded CI runner that compile takes
17 s to over 30 s, so the stop failed with "xcrun timed out after
30000ms" while most of its 70 s finalization budget was unused. #4065
moved the recording step into the iOS smoke lane, which made this
reachable on every PR.

The finalizer now hands its budget to the playability checks, and a
validator compile with a deadline runs until that deadline. Callers
without a deadline keep the 30 s bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s caller's budget

The in-flight validator compile took its timeout from the first caller, so a
concurrent `record stop` could inherit the 30 s bound. The compile is now shared
with only the Swift cache's own bound, and each caller races it against its own
budget, failing with reason `video_validator_compile_timeout`. The AVFoundation
run after the compile is clamped to the caller's remaining deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iting

A shared compile that outlived its timed-out caller had no owner: a daemon
that exited while it ran left `swiftc` reparented to launchd, past its runCmd
timeout, and left the compile's temp output in the Swift cache. The last caller
to stop waiting now aborts an unfinished compile, and the Swift cache passes
that signal to `swiftc` and to its lock wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a 1 ms validator run

Once the deadline had passed, the validator got a 1 ms timeout, so a clean
budget expiry became a thrown "timed out after 1ms". `isPlayableVideo` now
reads a deadline that passes before the validator runs as not playable, the
outcome `waitForPlayableVideo` already gives when it runs out of time.
`waitForPlayableVideo` resolves whether the file played, and the finalizer
decides on that verdict, so a file confirmed just before the budget ends is
not refused by a second check. The compile-timeout error names its retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…regression

The stub refused a second check only when handed the spent budget, so the base
form of that check, which passed no deadline, would have read playable and
passed. The stub now refuses any check, and the test asserts none ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…able

A validator run clamped to the caller's remaining budget now reads as not
playable when it times out, instead of throwing an exec timeout. The
overlay decides on the verdict waitForPlayableVideo returns, and the wait
checks once per attempt with no extra pass after the last one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s end too

swift-driver runs each swift-frontend job in its own process group. A
SIGKILL or SIGTERM on a canceled validator compile left the frontend
compiling with no parent; a live cold-cache abandon showed it. On SIGINT
the driver stops its jobs, and SIGKILL still follows after a 2 s grace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
thymikee and others added 3 commits November 2, 2026 21:59
…erdict

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A validator run that hit its own 10 s bound threw, while one the
deadline cut short read as not playable. Both are the same verdict: the
file did not prove playable in time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee
thymikee force-pushed the fix/ios-record-stop-timeout branch from cdf9516 to a1af133 Compare November 2, 2026 21:09

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 4 findings moved to a new line on a1af133. The verdict and the full list are in the summary comment.

);
// A budget with no time left reads as not playable; `runCmd` would read a timeout of 0 as none.
if (runTimeoutMs <= 0) return false;
const result = await runCmd(validatorPath, [filePath], {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · Validator-run timeout now returns false before the isSwiftVideoValidatorUnavailable fallback that used to see its stderr

isPlayableVideo used to let a runCmd timeout fall into the catch at the bottom, where isSwiftVideoValidatorError(error) inspects error.details.stderr/stdout (exactly the fields createTimeoutError in packages/host-kit/src/internal/exec.ts:662-669 fills from the killed child) and returns true when they say xcrun/swiftc is missing. That fail-open exists because a host with no working Swift toolchain must not have its recording refused. The new .catch maps any isCommandTimeoutError to undefined inside the try, so if (!result) return false answers before that fallback ever runs. Reproduce: a host whose xcrun stalls and dies printing xcrun: error: unable to find utility "swiftc" — the 10 s (or clamped runTimeoutMs) kill produces a timeout error carrying that stderr, and the verdict flips from true (predecessor) to false, so finalizeScreenRecording now throws recording was not finalized into a playable video instead of accepting the file. Intended per the PR description for the generic case, but the toolchain-missing classification is lost with it. Either re-check the timeout error with isSwiftVideoValidatorError before returning false, or note that a killed validator on a toolchain-less host now reads as unplayable.

Suggested change
const result = await runCmd(validatorPath, [filePath], {
}).catch((error: unknown) => {
if (isCommandTimeoutError(error)) {
return isSwiftVideoValidatorError(error) ? { exitCode: 0, stdout: '', stderr: '' } : undefined;
}
throw error;
});
if (!result) return false;
Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

let compiled = videoValidatorExecutablePaths.get(cacheRoot);
if (!compiled) {
compiled = compileSwiftSourceText({
function joinValidatorCompile(host: HostEnvironment, cacheRoot: string): SharedValidatorCompile {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · waitForValidatorCompile cancels a compile it started without incrementing waiters first

joinValidatorCompile publishes the map entry and starts compileSwiftSourceText before waitForValidatorCompile executes shared.waiters += 1 (video.ts:153-169 vs 138-152). Today the two run in the same synchronous block so no window exists, but the invariant "a registered compile always has at least one waiter" is invisible and unenforced: the moment the create/subscribe step is separated from the increment (an await introduced between them, or joinValidatorCompile reused by a caller that subscribes later), the last-waiter cleanup at video.ts:197-202 can abort a compile that has a live subscriber and delete the map entry under it, killing a swiftc nobody asked to cancel. joinValidatorCompile also takes a host parameter it never reads, which invites exactly that reuse. Initialize waiters: 1 at creation (and let the joining caller own the increment) or increment before videoValidatorCompiles.set.

Suggested change

Make the waiter count non-zero for the whole lifetime of a registered compile (start waiters at 1 and have the first waitForValidatorCompile release it in its finally), so the cancel-on-last-waiter rule cannot fire before a waiter is registered.

Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.


const SWIFT_CACHE_VERSION = '2';
const LOCK_RETRY_DELAY_MS = 25;
// swift-driver runs each frontend job in its own process group. It stops them on SIGINT, but

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Comment on kill describes only the timeout path, but SWIFTC_STOP now also serves request cancellation

ExecOptions.kill is documented as how 'a deadline or a cancelled request stops the child' (packages/host-kit/src/internal/exec.ts:32-38), and this PR is the first use of the field for a cancellation: SWIFTC_STOP is passed unconditionally with signal: params.signal (swift-cache.ts:201-205). The new comment above the constant explains only the swift-driver process-group reason for SIGINT and never says that SIGINT is load-bearing for the abort path too, so a reader who later passes a request-scoped signal into compileSwiftSourceFile/compileSwiftSourceText (which compileSwiftSourceFile cannot do today) would keep the default SIGKILL and re-create the orphaned swift-frontend processes this PR set out to avoid.

Suggested change

Extend the comment to state that the option applies to both the compile deadline and a caller's cancel signal, so any caller that forwards a signal keeps SIGINT.

Details

persona: javascript

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

settled: boolean;
};

// A compiled helper is valid for the Swift cache it was built into, so each cache root keeps its own.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nit · Stale doc comment above videoValidatorCompiles still describes the removed timeout behavior

The comment left in place above the new videoValidatorCompiles map ("A compiled helper is valid for the Swift cache it was built into, so each cache root keeps its own.") is directly contradicted by the two new lines added under it and by joinValidatorCompile/waitForValidatorCompile, which drop the entry when a compile fails or when the last waiter gives up — a cache root no longer keeps one compile for its lifetime. The comment also still doesn't say what this map is for now (single-flight compile sharing plus wait-counting), which is the non-obvious part of the new structure.

Suggested change

Rewrite the comment above videoValidatorCompiles to state the current contract: one in-flight compile per Swift cache root, shared by callers, dropped when it fails or when the last waiter stops waiting; a published executable is still reused from disk.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

@thymikee

thymikee commented Nov 2, 2026 •

Copy link
Copy Markdown
Member Author

The code looks good at a1af133, and the earlier findings are fixed: the cacheRoot path is computed once, the WebM doc comment now says the verdict ignores the deadline, and every validator run timeout maps to not playable whatever the bound. All checks pass, including the Apple Smoke Tests job that can exercise the iOS record stop finalize route, and there are no conflicts. I did not run the tests or a device. The live run evidence is your output at cdf9516, which has the same production files as this commit, taken before the rebase onto main.

Not blocking, and you can take or leave it: a timed-out validator run now reads as not playable for every caller, so the Android chunk pull in platform-android retries it up to 3 times and then fails with an untyped error, and the post-overlay check at finalizer-host.ts:66 reports it as an overlay warning, so one line in the PR body would cover this, and that check could also go because overlay.ts:120 already verifies the composite inside the budget. The isPlayableVideo doc comment at packages/capture-kit/src/recording/video.ts:92 also breaks mid-sentence after "A WebM" and needs a reflow.

The two open inline threads on the recovery hint when the validator is cut by the deadline (#4150 (comment) and #4150 (comment)) still stand, but the hint is correct and the note above covers them. These threads do not apply and you can resolve them: the 30 s compile wait matches main and a failed compile entry is deleted (#4150 (comment)); overlay.ts throws its own detail-less error, so no compile-timeout details are copied (#4150 (comment)); the runTimeoutMs inference was removed (#4150 (comment)); each waiter is bounded by its own wait and the last one aborts the compile (#4150 (comment)); overlay verification coming out of the 70 s budget is the intended contract (#4150 (comment)); your live run shows the compile hint holds (#4150 (comment)); the timeout comment now sits on the guard (#4150 (comment)); abort fires only with no waiters, so no caller sees the cancel rejection (#4150 (comment)); the timed-out run is the compiled validator, not xcrun (#4150 (comment)); the join and the waiter increment run in one synchronous block (#4150 (comment)); the SWIFTC_STOP comment covers timeout and cancel (#4150 (comment)); the keying comment describes current behavior (#4150 (comment)). These are fixed at this commit: #4150 (comment), #4150 (comment), #4150 (comment) and #4150 (comment).

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 2, 2026
@airowe

airowe commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Timing evidence from CI for the smoke:capture-close failure this PR targets, in case it helps review. It comes from job logs and the apple-artifacts request logs, compile = xcrun swiftc for video-validator inside record stop.

Runs Compile in record stop Recompile in the next close
12 passing Apple runs 12.6–23.7 s, median 17.3 s n/a
41457810383 (main) killed at 30 s 3.7 s
41499720011 killed at 30 s 5.3 s
41510773833 killed at 30 s 5.2 s
41523999616 killed at 30 s 3.2 s
41546007273 killed at 30 s 3.0 s
41536287527 attempt 1 18.9 s, smoke:capture-close passed n/a
41536287527 attempt 2, same SHA timed out (job log only) not read
  • The compile is slow, not hung: the recompile finishes in seconds from the module cache the killed compile built.
  • It varies by attempt on the same SHA, which points to runner load rather than code.
  • Estimate, not a measurement: the killed compiles needed about 33–45 s. The low end is 30 s plus the recompile. The high end adds the roughly 10 s before the recompile, since orphaned frontends may keep building in that gap. That fits within what the 70 s finalize budget leaves for the compile (about 68 s, the compile started about 1.4 s into the stop in the one run I read).
  • The daemon request timeout (90 s) and the harness CLI timeout (120 s) are both longer than the budget.

Limits of this evidence: this PR's three Apple runs compiled in 19.2, 14.4 and 12.6 s, so CI here has not yet exercised a compile over 30 s. Your live run is the only proof of that path. I read the diff but did not run this PR's tests or a device.

@thymikee

thymikee commented Nov 3, 2026 •

Copy link
Copy Markdown
Member Author

Thanks, this matches what we saw. The compile is slow under runner load, not hung, and it fits in the finalize budget.

  • The path where the compile runs past 30 s is covered by the live run on a cold cache under CPU load (see the live-run comment above): a 66.5 s compile, with record stop passing in 73.2 s. A forced 5 s budget gave video_validator_compile_timeout, and the next stop compiled again in 36.4 s and passed.
  • On orphaned frontends: the live abandon showed that killing swiftc with SIGKILL left swift-frontend running. The PR now cancels with SIGINT, the only signal that makes the Swift driver stop its frontend jobs. After an abandon, no swiftc or swift-frontend processes and no lock or temp directory are left behind.

@thymikee
thymikee merged commit d0e683f into main Nov 3, 2026
19 checks passed
@thymikee
thymikee deleted the fix/ios-record-stop-timeout branch November 3, 2026 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants