Skip to content

Commit 5216fd4

Browse files
authored
improvement(ci): overlap independent acceptance and recovery checks
Run extension native and access acceptance in parallel with independent fixtures, apps, databases, and reports. Overlap independent desktop-pickup deadlines and isolated recovery backoff cases, and use faster integration runners with the existing GitHub fallback. Update Playwright for the upstream WebKit compositor fix and disable focus emulation at the native blur boundary after reloads. Preserve all acceptance assertions and real timing contracts. Validated with root tests, all workspace type checks, lint, all audits, generated-artifact checks, real integration and Electron acceptance runs, and negative controls for the protected behaviors.
1 parent b6677c3 commit 5216fd4

16 files changed

Lines changed: 328 additions & 199 deletions

File tree

‎.github/CI.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ SEO depends on workspace UI and server code: landing pages reuse workspace prici
2424

2525
Affected type-checks follow the package graph too; repository audits and lint run whenever the checks workflow runs. Workspace-only unit jobs use a free GitHub runner and do not start the app test shards.
2626

27-
Database integration and lightweight HTTP suites run on free standard GitHub runners with provider-specific dependency setup. A same-source integration comparison of 16 jobs per provider (32 total) had a slowest shard of 6m 04s on 2-vCPU Blacksmith versus 6m 55s on free GitHub, with identical coverage. GitHub trades the observed 51 seconds for zero runner compute cost. This is one paired sample, not a long-run percentile estimate. These explicitly selected GitHub jobs do not follow `CI_PROVIDER`.
27+
Database integration uses 4-vCPU Blacksmith runners with sticky dependency disks, following `CI_PROVIDER` and falling back to standard GitHub runners with matching setup. A prior same-source comparison of 16 jobs per provider (32 total) had a slowest shard of 6m 04s on 2-vCPU Blacksmith versus 6m 55s on GitHub, with identical coverage. This is one paired sample, not a long-run percentile estimate. Evaluate runner changes against complete coverage and the overall CI gate, since a faster shard does not shorten a run when another job finishes later. Lightweight HTTP suites retain free standard GitHub runners and do not follow `CI_PROVIDER`.
2828

2929
GitHub runners install dependencies directly rather than transferring Bun and `node_modules` archives. In a same-source paired comparison, those restores took 26.2 seconds before an 82-millisecond install, while a clean install took 12.42 seconds. This is one paired observation; retain the policy only while repeated CI timings support it. Blacksmith keeps its sticky dependency disks. GitHub Turbo output caches remain enabled for jobs that request them, with a commit-specific key and a task/event/fork-scoped restore prefix, so a rerun can reuse an exact entry without creating another archive. Push jobs never restore pull-request entries.
3030

@@ -34,10 +34,12 @@ Every HTTP and browser group consumes the same production build in an isolated j
3434

3535
Desktop inbox protocol, assistant actions UI, and organization search UI run as three parallel HTTP jobs with isolated databases, apps, and reports. They retain one acceptance task boundary, so changing any of their harnesses selects all three jobs. Desktop inbox needs no browser. The UI jobs and browser extension install Chromium while their app starts, waiting for both before executing assertions; browser-install failures still fail the job and preserve the install log. No acceptance assertions are removed by the split.
3636

37-
Browser extension acceptance keeps native streaming and first-open capture checks on real timers. Its access-revalidation scenarios then reopen the persistent browser profile and install Playwright's clock before the app mounts. Advancing the production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. Live-provider runs retain native timing throughout.
37+
Browser extension acceptance runs native interaction and access-revalidation groups in parallel jobs with isolated databases, apps, browser profiles, and reports. Both retain the same task boundary, so an affected extension harness selects both groups. Native streaming and first-open capture checks retain real timers. Access checks install Playwright's clock before the app mounts: advancing production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. The local `browser-extension` group still runs the complete suite; live-provider runs require this complete mode and retain native timing throughout.
3838

3939
Integration shard weights come from successful reports for both push and migration provisioning paths. Refresh them with `scripts/update-integration-durations.ts`, then check the new partition against a separate complete run before adopting it. Weights only change scheduling; discovery still owns coverage. The current refresh uses runs [41348786822](https://github.com/simstudioai/sim/actions/runs/41348786822) and [41377145857](https://github.com/simstudioai/sim/actions/runs/41377145857), with [41377810718](https://github.com/simstudioai/sim/actions/runs/41377810718) held out for validation.
4040

41+
Desktop pickup checks overlap their real grace periods using distinct calls and waiters under fixed authorization and parent-run fixtures. Recovery exhaustion checks keep worker responses, injected faults, and observed requests scoped to each stream before overlapping their real backoff schedules. Their immediate-takeover checks remain serial. File isolation and serial execution between integration files remain unchanged; these suites still verify actual timeout, lease, retry-floor, and late-claim behavior against PostgreSQL and Redis.
42+
4143
SCIM's production app uses a `sim.ai` fixture hostname mapped to loopback, since `NEXT_PUBLIC_FORCE_HOSTED` is ignored in production. Readiness, HTTP tests, and internal API requests still connect to loopback. The SCIM suite verifies that ending an Enterprise subscription rejects both owner administration and an existing bearer token, and that restoring it restores access.
4244

4345
## Coverage removed or consolidated

‎.github/ci-suites.json‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,11 @@
2525
"task": "ci:e2e-desktop-inbox"
2626
},
2727
{
28-
"group": "browser-extension",
28+
"group": "browser-extension-native",
29+
"task": "ci:e2e-browser-extension"
30+
},
31+
{
32+
"group": "browser-extension-access",
2933
"task": "ci:e2e-browser-extension"
3034
},
3135
{

‎.github/scripts/http-e2e.sh‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# CI groups, which reuse the production build from `e2e-build`.
44
# The job extracts it into apps/sim/.next before running these suites.
55
#
6-
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|project-files|mobile|seo> (run from apps/sim)
6+
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-access|project-files|mobile|seo> (run from apps/sim)
77
#
88
# The job provides DATABASE_URL, BETTER_AUTH_SECRET and ENCRYPTION_KEY; each group sets the rest of
99
# its app's environment here. Reports and server logs land in $RUNNER_TEMP/e2e.
@@ -20,7 +20,7 @@
2020
# telemetry flush runs detached and still writes .next/dev).
2121
set -euo pipefail
2222

23-
group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|project-files|mobile|seo>}
23+
group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-access|project-files|mobile|seo>}
2424
report_dir="$RUNNER_TEMP/e2e"
2525
ready_timeout_seconds=600
2626
mkdir -p "$report_dir"
@@ -264,13 +264,18 @@ case "$group" in
264264
esac
265265
;;
266266

267-
browser-extension)
267+
browser-extension|browser-extension-native|browser-extension-access)
268+
extension_group=all
269+
case "$group" in
270+
browser-extension-native) extension_group=native ;;
271+
browser-extension-access) extension_group=access ;;
272+
esac
268273
export REDIS_URL=redis://127.0.0.1:6379
269274
export NEXT_PUBLIC_FORCE_HOSTED=false
270275
export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
271276
export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true
272277
export API_ENCRYPTION_KEY="$ENCRYPTION_KEY"
273-
start_browser_install browser-extension chromium
278+
start_browser_install "$group" chromium
274279
extension_identity=$(bun - <<'JS'
275280
import { generateKeyPairSync, createHash } from 'node:crypto'
276281
const { publicKey } = generateKeyPairSync('rsa', { modulusLength: 2048 })
@@ -282,11 +287,12 @@ JS
282287
)
283288
export SIM_EXTENSION_KEY="${extension_identity%%$'\n'*}"
284289
export BROWSER_EXTENSION_IDS="${extension_identity#*$'\n'}"
285-
start_app browser-extension 3007 'browser extension' record-http-status
290+
start_app "$group" 3007 'browser extension' record-http-status
286291
wait_for_browsers
287292
EXTENSION_APP_E2E_BASE_URL="$local_app_url" \
288293
EXTENSION_APP_E2E_DATABASE_URL="$DATABASE_URL" \
289-
EXTENSION_APP_E2E_REPORT_PATH="$report_dir/browser-extension/report.json" \
294+
EXTENSION_APP_E2E_GROUP="$extension_group" \
295+
EXTENSION_APP_E2E_REPORT_PATH="$report_dir/$group/report.json" \
290296
bun run test:browser-extension:e2e
291297
;;
292298

‎.github/workflows/checks.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ jobs:
7676
name: integration (${{ matrix.provision }}, ${{ matrix.shard == 'db' && 'db' || format('{0}/8', matrix.shard) }})
7777
needs: changes
7878
if: needs.changes.outputs.integration == 'true'
79-
runs-on: ubuntu-24.04
79+
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-24.04' }}
8080
timeout-minutes: 20
8181
strategy:
8282
fail-fast: false
@@ -135,7 +135,7 @@ jobs:
135135
- name: Setup workspace
136136
uses: ./.github/actions/setup
137137
with:
138-
provider: github
138+
provider: ${{ vars.CI_PROVIDER }}
139139

140140
- name: Provision a fresh database through the supported command
141141
working-directory: packages/db
@@ -219,7 +219,7 @@ jobs:
219219
--health-retries 10
220220
# Desktop inbox, browser extension, and Project Files use Redis.
221221
redis:
222-
image: ${{ (matrix.group == 'desktop-inbox' || matrix.group == 'assistant-actions' || matrix.group == 'organization-search' || matrix.group == 'browser-extension' || matrix.group == 'project-files') && 'redis:8.2-alpine' || '' }}
222+
image: ${{ (matrix.group == 'desktop-inbox' || matrix.group == 'assistant-actions' || matrix.group == 'organization-search' || startsWith(matrix.group, 'browser-extension-') || matrix.group == 'project-files') && 'redis:8.2-alpine' || '' }}
223223
ports:
224224
- 6379:6379
225225
options: >-
@@ -298,7 +298,7 @@ jobs:
298298
name: e2e-reports-${{ matrix.group }}
299299
path: |
300300
${{ runner.temp }}/e2e/**/*.json
301-
${{ runner.temp }}/e2e/browser-extension/**/*.png
301+
${{ runner.temp }}/e2e/browser-extension*/**/*.png
302302
${{ runner.temp }}/e2e/*-ui-report.json.artifacts/*.png
303303
if-no-files-found: ignore
304304
retention-days: 7

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@ name: ci
88
# A repo variable, not a committed value: during a Blacksmith outage there is no
99
# working CI to merge a switchover through. Only unset/'blacksmith' selects
1010
# Blacksmith; anything unrecognized selects GitHub so a typo can't queue jobs
11-
# against the provider you're escaping. Paid build/browser jobs and their cache actions share
12-
# this predicate. Task planning, the aggregate gate, HTTP acceptance and integration tests use free GitHub runners.
11+
# against the provider you're escaping. Integration, build and browser jobs and their cache
12+
# actions share this predicate. Task planning, the aggregate gate and HTTP acceptance use free GitHub runners.
1313
#
1414
# GitHub mode is break-glass, not a peer — cold layers, slower runs. The app image
1515
# is the one job on a paid larger runner: next build needs ~32 GB and OOM-kills

‎apps/desktop/e2e/chat-panel.spec.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,8 +114,6 @@ test('chat panel sizes survive navigation, chat switches, collapse, and layout c
114114
const shell = app
115115
const page = await shell.firstWindow()
116116
const inputSession = await page.context().newCDPSession(page)
117-
// Playwright otherwise keeps the document focused when its native window loses focus.
118-
await inputSession.send('Emulation.setFocusEmulationEnabled', { enabled: false })
119117
page.on('pageerror', (error) => errors.push(error.message))
120118
page.on('crash', () => {
121119
rendererCrashed = true
@@ -270,6 +268,8 @@ test('chat panel sizes survive navigation, chat switches, collapse, and layout c
270268
await divider.evaluate((element) => element.releasePointerCapture(1))
271269
await page.mouse.move(rect.x + 100, rect.y + 100)
272270
} else if (interruption === 'blur') {
271+
// Reloads restore Playwright's focus emulation; native blur must reach the document.
272+
await inputSession.send('Emulation.setFocusEmulationEnabled', { enabled: false })
273273
await shell.evaluate(({ BrowserWindow }) => BrowserWindow.getAllWindows()[0].blur())
274274
await expect.poll(() => page.evaluate(() => document.hasFocus())).toBe(false)
275275
} else if (interruption === 'chat switch') {

‎apps/desktop/package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@
5050
},
5151
"devDependencies": {
5252
"@electron/fuses": "1.8.0",
53-
"@playwright/test": "1.61.1",
53+
"@playwright/test": "1.64.1",
5454
"@sim/emcn": "workspace:*",
5555
"@sim/tsconfig": "workspace:*",
5656
"@types/micromatch": "4.0.10",

‎apps/docs/package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@
6767
"vfile": "6.0.3"
6868
},
6969
"devDependencies": {
70-
"@playwright/test": "1.61.1",
70+
"@playwright/test": "1.64.1",
7171
"@sim/tsconfig": "workspace:*",
7272
"@tailwindcss/postcss": "^4.3.3",
7373
"@types/node": "24.19.2",

‎apps/extension/package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
"zod": "4.3.6"
2525
},
2626
"devDependencies": {
27-
"@playwright/test": "1.61.1",
27+
"@playwright/test": "1.64.1",
2828
"@sim/tsconfig": "workspace:*",
2929
"@types/bun": "1.4.3",
3030
"@types/chrome": "0.3.6",

0 commit comments

Comments
 (0)