Skip to content

fix(frontend): stop unit-test memory growth that times out the macOS leg - #9198

Open
aglinxinyuan wants to merge 3 commits into
apache:mainfrom
aglinxinyuan:fix/macos-frontend-test-timeouts
Open

aglinxinyuan wants to merge 3 commits into
apache:mainfrom
aglinxinyuan:fix/macos-frontend-test-timeouts

Conversation

@aglinxinyuan

@aglinxinyuan aglinxinyuan commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

This fixes the random unit-test timeouts on the macOS frontend leg (#9197) at their cause: memory that grows across spec files inside each Vitest worker.

The Angular unit-test builder runs Vitest with isolate: false, so a worker runs all its spec files against one jsdom document and one copy of Vitest. Three kinds of worker-wide reference kept every finished test's TestBed graph alive, and any one of them was enough on its own. On the 7 GB macOS runner that came to about 2.4 GB per worker, and the VM swapped until whichever test was running timed out.

  1. Root services now release what they start. These are production bugs too.
    • WorkflowActionService.ngOnDestroy() destroys the shared model.
    • SharedModel.destroy() calls wsProvider.destroy() instead of a conditional disconnect(). This releases the provider's interval, BroadcastChannel subscription and listeners, and stops a provider that never connected from retrying forever.
    • WorkflowWebsocketService.ngOnDestroy() stops the heartbeat and closes the socket.
    • JupyterPanelService.ngOnDestroy() removes its window listener.
  2. jsdom's selector cache is emptied between tests. It caches every selector with the nodes it matched, including the #root<N> query TestBed makes for each fixture. The new setup file is src/jsdom-selector-cache-setup.ts.
  3. Vitest 4's mock registry no longer keeps every mock alive. @vitest/spy held every vi.fn() / vi.spyOn() in a Set that is never pruned. A Yarn patch backports Vitest 5's WeakRef registry, since Angular 21 still requires Vitest 4. The patch carries Vitest's MIT code, so it is listed in LICENSE, with the license text in licenses-3rd-party-code/vitest.txt.

Measured locally in the macOS shape (2 workers, coverage, full suite):

main this PR
Peak footprint per worker 2.9–3.1 GB 1.5–1.9 GB
Peak footprint of the process tree 8.7–9.0 GB 5.6–6.2 GB
Vitest Duration 85–115 s 65–66 s

Notes for review:

  • Why not isolate: true? It also bounds memory, but it roughly doubled the Vitest run locally (230 s against about 100 s).
  • The spy patch is needed. Without it the process-tree peak went back to 7.4 GB. Clearing or resetting mocks at the end of each file instead reached only 6.8 / 7.0 GB.
  • The patch has to be redone on every Vitest bump. The resolutions entry names the exact @vitest/spy version, so it stops applying whenever Vitest is bumped.

Any related issues, documentation, discussions?

Closes #9197. Related: #7713 raised the test timeouts and #7975 capped the worker heap. Both treated the symptom.

How was this PR tested?

  • Run before main started requiring Node 24, on this branch's earlier commits:
    • yarn --cwd frontend run test:ci: 244 files passed, 6857 tests passed, 1 skipped.
    • The full suite twice at 2 workers with memory sampled; both runs passed all 244 files. The numbers are in the table above.
  • The new tests fail against the old code and pass with the fix:
    • SharedModel destroy: 2 tests
    • WorkflowActionService, WorkflowWebsocketService, JupyterPanelService teardown: 1 test each
  • On the current head, which includes the latest main:
    • yarn install --immutable passes.
    • The test build type-checks every spec.
    • prettier-eslint and eslint are clean on the changed files.
    • The test run itself is left to CI, because main now needs Node 24 and my machine has Node 20.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

The unit-test builder runs Vitest with isolate: false, so each worker runs
its spec files against one jsdom document and one copy of Vitest. Several
worker-wide holders kept every finished test's TestBed graph reachable, and
on the 7 GB macOS runner the resulting growth (about 2.4 GB per worker)
swapped the VM into stalls that timed out whichever test was running.

- Release what root services start: WorkflowActionService destroys its
  shared model, WorkflowWebsocketService stops its heartbeat and closes the
  socket, JupyterPanelService removes its window listener, and
  SharedModel.destroy() destroys its y-websocket provider rather than only
  disconnecting it when connected.
- Empty jsdom's selector cache between tests (new setup file).
- Backport Vitest 5's weak mock registry into @vitest/spy 4.1.11 with a
  Yarn patch, guarded by worker-memory.spec.ts.

Closes apache#9197

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 27, 2026 23:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file fix frontend Changes related to the frontend GUI ci changes related to CI docs Changes related to documentations labels Oct 27, 2026
@Yicong-Huang Yicong-Huang added release/v1.3 back porting to release/v1.3 release/v1.2 back porting to release/v1.2 labels Oct 27, 2026
@github-actions
github-actions Bot requested review from mengw15 and xuang7 October 27, 2026 23:21
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. A release/* label nominates a backport target; the branch's release manager approving this PR is what sends the fix there. The required Backport Approvals check stays red until every label below is approved, so each manager either approves or removes their own label — which is why the labels left on a merged PR are exactly the branches it reached.

Release branch Analysis
✅ release/v1.3 Already labeled — this fix is queued to backport here. @mengw15 decides: approving sends the fix here, removing this label declines it. The merge waits on one or the other.
✅ release/v1.2 Already labeled — this fix is queued to backport here. @xuang7 decides: approving sends the fix here, removing this label declines it. The merge waits on one or the other.

Auto-label run.

@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @yangzhang75, @mengw15, @zyratlo
    You can notify them by mentioning @yangzhang75, @mengw15, @zyratlo in a comment.

…st-timeouts

# Conflicts:
#	frontend/src/app/workspace/service/workflow-websocket/workflow-websocket.service.ts
@codecov-commenter

codecov-commenter commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.98%. Comparing base (c57e6b0) to head (6f85798).

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #9198   +/-   ##
=========================================
  Coverage     91.98%   91.98%           
+ Complexity     5084     5083    -1     
=========================================
  Files          1281     1281           
  Lines         55806    55810    +4     
  Branches       7076     7075    -1     
=========================================
+ Hits          51333    51337    +4     
  Misses         2732     2732           
  Partials       1741     1741           
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø) Carriedforward from b195ae7
agent-service 99.16% <ø> (ø) Carriedforward from b195ae7
amber 86.42% <ø> (-0.01%) ⬇️ Carriedforward from b195ae7
computing-unit-managing-service 63.52% <ø> (ø) Carriedforward from b195ae7
config-service 87.50% <ø> (ø) Carriedforward from b195ae7
file-service 82.29% <ø> (ø) Carriedforward from b195ae7
frontend 96.53% <100.00%> (+<0.01%) ⬆️
notebook-migration-service 83.77% <ø> (ø) Carriedforward from b195ae7
pyamber 98.54% <ø> (ø) Carriedforward from b195ae7
workflow-compiling-service 74.25% <ø> (ø) Carriedforward from b195ae7

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 10 worse · ⚪ 5 noise (<±5%) · 0 without baseline

Compared against main c57e6b0 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 449 0.274 21,044/34,134/34,134 us 🔴 +20.2% / 🔴 +147.8%
🔴 bs=100 sw=10 sl=64 935 0.571 104,841/132,479/132,479 us 🔴 +21.1% / 🔴 +42.2%
⚪ bs=1000 sw=10 sl=64 1,154 0.704 863,091/905,924/905,924 us ⚪ within ±5% / 🔴 -11.6%
Baseline details

Latest main c57e6b0 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 449 tuples/sec 509 tuples/sec 941.58 tuples/sec -11.8% -52.3%
bs=10 sw=10 sl=64 MB/s 0.274 MB/s 0.31 MB/s 0.575 MB/s -11.6% -52.3%
bs=10 sw=10 sl=64 p50 21,044 us 17,511 us 10,991 us +20.2% +91.5%
bs=10 sw=10 sl=64 p95 34,134 us 31,733 us 13,776 us +7.6% +147.8%
bs=10 sw=10 sl=64 p99 34,134 us 31,733 us 16,872 us +7.6% +102.3%
bs=100 sw=10 sl=64 throughput 935 tuples/sec 1,015 tuples/sec 1,264 tuples/sec -7.9% -26.0%
bs=100 sw=10 sl=64 MB/s 0.571 MB/s 0.619 MB/s 0.772 MB/s -7.8% -26.0%
bs=100 sw=10 sl=64 p50 104,841 us 97,771 us 86,073 us +7.2% +21.8%
bs=100 sw=10 sl=64 p95 132,479 us 109,378 us 93,135 us +21.1% +42.2%
bs=100 sw=10 sl=64 p99 132,479 us 109,378 us 111,124 us +21.1% +19.2%
bs=1000 sw=10 sl=64 throughput 1,154 tuples/sec 1,169 tuples/sec 1,304 tuples/sec -1.3% -11.5%
bs=1000 sw=10 sl=64 MB/s 0.704 MB/s 0.713 MB/s 0.796 MB/s -1.3% -11.6%
bs=1000 sw=10 sl=64 p50 863,091 us 854,601 us 841,986 us +1.0% +2.5%
bs=1000 sw=10 sl=64 p95 905,924 us 887,282 us 881,107 us +2.1% +2.8%
bs=1000 sw=10 sl=64 p99 905,924 us 887,282 us 918,208 us +2.1% -1.3%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,445.86,200,128000,449,0.274,21044.39,34133.80,34133.80
1,100,10,64,20,2139.44,2000,1280000,935,0.571,104840.86,132478.58,132478.58
2,1000,10,64,20,17328.36,20000,12800000,1154,0.704,863090.91,905923.67,905923.67

Drop the GC guard spec, the Renovate grouping rule and the TESTING.md
section, fold the selector-cache helper into its setup file, and make the
@vitest/spy patch comment self-contained.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot removed the ci changes related to CI label Oct 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file docs Changes related to documentations fix frontend Changes related to the frontend GUI release/v1.2 back porting to release/v1.2 release/v1.3 back porting to release/v1.3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

frontend: the macOS test leg times out at random because each Vitest worker's memory grows across spec files

4 participants