Skip to content

refactor(engine): type capture failures#2161

Merged
jrusso1020 merged 2 commits into
mainfrom
07-10-refactor_engine_type_capture_failures
Jul 17, 2026
Merged

refactor(engine): type capture failures#2161
jrusso1020 merged 2 commits into
mainfrom
07-10-refactor_engine_type_capture_failures

Conversation

@jrusso1020

@jrusso1020 jrusso1020 commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

What

Replace message-parsed capture retry policy with typed CaptureFailure values and peer abort.

Why

Stringified worker errors lost causes and let sibling workers continue after fatal authoring failures.

How

Classify cancellation, browser, protocol, memory, verification, authoring, and IO failures; preserve diagnostics and abort peers on the first fatal kind.

Test plan

  • capture classification and parallel peer-abort tests; producer capture-stage tests
  • Stack-wide lint, format, build, typecheck, and relevant integration gates

jrusso1020 commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from fc61671 to b683ddb Compare July 14, 2026 00:03
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_type_capture_failures branch 2 times, most recently from 7935b9d to bd29456 Compare July 14, 2026 03:20
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from b683ddb to 4e86382 Compare July 14, 2026 03:20
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_type_capture_failures branch from bd29456 to e10f87a Compare July 14, 2026 17:03
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from 1a7d917 to 7dd4686 Compare July 14, 2026 17:11
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_type_capture_failures branch from e10f87a to 9e02fbc Compare July 14, 2026 17:11
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from 7dd4686 to 8733675 Compare July 14, 2026 17:35
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_type_capture_failures branch 3 times, most recently from 533dc78 to 5d9ca34 Compare July 14, 2026 18:06
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch 2 times, most recently from 0c42de6 to f095fb4 Compare July 14, 2026 19:37
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_type_capture_failures branch from 5d9ca34 to 2d9a934 Compare July 14, 2026 19:37
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from f095fb4 to eda0bb1 Compare July 14, 2026 19:42
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_type_capture_failures branch from 2d9a934 to d564f84 Compare July 14, 2026 19:42
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from eda0bb1 to c0bd5b9 Compare July 14, 2026 19:47
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_type_capture_failures branch from d564f84 to 8c275ea Compare July 14, 2026 19:47
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from c0bd5b9 to 571341a Compare July 14, 2026 19:55
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_type_capture_failures branch from 8c275ea to 01d355d Compare July 14, 2026 19:55
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from 571341a to 606e05b Compare July 14, 2026 23:29
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_type_capture_failures branch from 01d355d to 26d4d2e Compare July 14, 2026 23:29
@jrusso1020
jrusso1020 force-pushed the 07-10-refactor_engine_manage_child_processes branch from 606e05b to 6cf4221 Compare July 14, 2026 23:41
Comment thread packages/engine/src/services/captureFailure.ts Fixed

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The typed failure boundary is a strong improvement overall: cancellation is authoritative, worker diagnostics stay immutable and attached to the representative failure, and the first fatal worker now stops sibling capture without hiding cleanup. The linear same-line I/O scan and 100k-character adversarial test also close the CodeQL finding without broadening classification.

One retry regression remains:

  • [P1] Preserve retryability for the drawElement worker watchdog timeoutpackages/engine/src/services/captureFailure.ts:57-64 enumerates protocol timeout strings but omits the engine's own drawElement worker encode timed out (frame N) failure. That error is produced by the 30-second lost-message/OOM-killed-worker watchdog at packages/engine/src/services/drawElementService.ts:674-684 (and the batch sibling at :901-906), then propagates through captureFrameRange's awaited encodeResult. On this exact head, classifyCaptureFailure(new Error("drawElement worker encode timed out (frame 42)")) returns authoring, so parallelCoordinator.ts:594-598 treats it as fatal and aborts peers, while renderOrchestrator.ts:817-821 refuses the worker-halving retry. Before this PR, the broad timed out recovery matcher covered this concrete failure. Please classify this exact watchdog failure as non-fatal/retryable and pin both classification and recovery behavior with a regression test.

Exact head reviewed: 9ad957786d14040f72b87ac2f72ac59a028e6f90. All exact-head required checks are green; this is a code-contract blocker rather than a CI failure.

Verdict: REQUEST CHANGES
Reasoning: The typed model and peer-abort mechanism are sound, but a real engine-generated worker timeout regresses from retryable to fatal, causing avoidable render failures instead of the prior recovery path.

— Magi

@jrusso1020

Copy link
Copy Markdown
Collaborator Author

Fixed Magi’s P1 on exact head 5da9f7ab3d53d7b5b95f216d63532f20cb6036dd.

  • Added a narrowly scoped drawElement worker encode timed out (frame N) protocol-timeout classification, preserving non-fatal peer behavior and Producer’s worker-halving retry.
  • Added direct Engine classification coverage and a wrapped [Parallel] Capture failed Producer recovery regression.
  • Validation: Engine classification 13/13; full Producer render-orchestrator 142/142; full Engine 1,131 passed / 3 skipped; Engine and Producer typechecks; lint/format/tracked-artifact/fallow hooks; package-subpath sync; full workspace build on exact restacked top 4deb4510c18b79c95b8c5dc1a64420257c437a03.
  • PRs refactor(engine): type capture failures #2161refactor(repo): resolve changed-code audit #2174 were restacked and republished. Exact head remains on live main c268f5ba8 with behind_by: 0 and merge base exactly live main.

Fresh exact-head CI is running. This is a substantive head change and requires a fresh exact-head review after all required checks are green.

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh exact-head re-review at 5da9f7ab3d53d7b5b95f216d63532f20cb6036dd.

The prior blocker is resolved:

  • packages/engine/src/services/captureFailure.ts:57-66 now recognizes only the concrete drawElement worker encode timed out (frame N) watchdog signature as protocol_timeout; direct execution on this head returns { kind: "protocol_timeout", fatal: false }.
  • packages/producer/src/services/renderOrchestrator.ts:817-821 therefore retains the worker-halving recovery path, and packages/producer/src/services/renderOrchestrator.test.ts:1505-1525 pins the wrapped [Parallel] Capture failed boundary rather than testing only the inner classifier.

The broader design remains sound: cancellation wins over an existing typed failure at captureFailure.ts:136-142; worker-local causes and diagnostics are preserved at parallelCoordinator.ts:493-506; and the first genuinely fatal failure aborts peers while Promise.all still waits for their teardown at parallelCoordinator.ts:589-637. The linear same-line I/O classifier retains its 100k-character adversarial regression. I also swept the engine/Producer timeout emitters again; no remaining capture-path timeout lost the old retry contract.

Verification: Engine classifier suite 13/13 locally; exact-head GitHub checks report 40 success, 8 intentional skips, 0 pending/failing; PR is mergeable and based exactly on live main.

Verdict: APPROVE
Reasoning: The exact watchdog regression is fixed narrowly and covered at both the Engine classification and Producer recovery boundaries; the typed failure and peer-abort contracts now preserve prior retry behavior without reintroducing broad string matching.

— Magi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants