Skip to content

Commit dd06b90

Browse files
authored
v0.9.43: Sim links from search and file shares, synced org chart, CLI recovery guidance, image and video catalog refresh
2 parents f3596f0 + 0de05d4 commit dd06b90

378 files changed

Lines changed: 126716 additions & 4706 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/CI.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
The `changes` job runs Turborepo 2.9.14's `affectedTasks` query before installing application dependencies or provisioning services. `turbo.json` records package dependency propagation; `apps/sim/turbo.json` partitions the app into shared runtime, server, workspace UI, and acceptance tasks. `.github/ci-suites.json` turns those tasks into job matrices. The uploaded `ci-plan` artifact names changed files, selected tasks, attribution reasons, and any fallback.
44

5-
A PR compares its merge base with the checked-out merge commit. Main/staging pushes execute all checks; the parent `ci` workflow skips checks on `dev` pushes, while PRs targeting `dev` still run checks. A manual `checks` run executes all checks unless supplied a full base commit SHA to reproduce a scoped plan. Missing history, an empty diff, unknown files outside the workspace graph, root script/CI changes, or a planner error run every check. Missing selection outputs fail the aggregate gate. The gate also rejects an omitted dependency, any failed/cancelled job, and any skip that its explicit selection flag did not authorize.
5+
A PR compares the checked-out GitHub merge commit with its first parent: the actual target revision tested by that merge. The PR event's base SHA can become stale when the target advances. Scoping requires a matching GitHub merge ref and SHA, exactly two merge parents, and a usable target parent; otherwise every check runs. This works with the existing depth-two checkout without fetching older ancestry. Main/staging pushes execute all checks; the parent `ci` workflow skips checks on `dev` pushes, while PRs targeting `dev` still run checks. A manual `checks` run executes all checks unless supplied a full base commit SHA to reproduce a scoped plan. Missing history, an empty diff, unknown files outside the workspace graph, root script/CI changes, or a planner error run every check. Missing selection outputs fail the aggregate gate. The gate also rejects an omitted dependency, any failed/cancelled job, and any skip that its explicit selection flag did not authorize.
66

7-
Dedicated docs verification uses the same planner, comparing pushes with their previous commit and PRs with their merge base. Site acceptance follows `docs#build` and its transitive package dependencies. The app's docs-consumer task depends on server runtime and that docs build: backend-only changes run the platform consumer against the docs app, while site changes also run corpus and browser checks. Independent SDK and unit-only edits skip those production jobs. The required `docs HTTP` gate verifies explicit site/platform selections before accepting any skip.
7+
Dedicated docs verification uses the same planner, comparing pushes with their previous commit and PRs with the tested merge's first parent. Site acceptance follows `docs#build` and its transitive package dependencies. The app's docs-consumer task depends on server runtime and that docs build: backend-only changes run the platform consumer against the docs app, while site changes also run corpus and browser checks. Independent SDK and unit-only edits skip those production jobs. The required `docs HTTP` gate verifies explicit site/platform selections before accepting any skip.
88

99
Docs production builds run once and publish an artifact to isolated HTTP, corpus, Chromium, and WebKit jobs. Source checks keep their own checkout because generator verification mutates inputs. Content caches use separate lint/source/build namespaces; generation validates every restore, repairs missing declarations, and removes obsolete modules. The Next build cache keeps its supported incremental behavior. Content and Next caches have size limits, and Next cache files expire after seven days; successful content eviction happens after all consumers finish.
1010

@@ -36,6 +36,8 @@ Closing a same-repository pull request deletes only its GitHub merge-ref caches
3636

3737
Memory-heavy builds and browser jobs retain appropriately sized Blacksmith runners. Electron uses 4-vCPU Blacksmith: both shards passed within 14 seconds of the 8-vCPU control; the free GitHub rest shard was slower and its lease run stalled.
3838

39+
WebKit acceptance on Blacksmith uses the matching, digest-pinned Playwright browser image for mobile and docs navigation. The app, database fixtures, native modules, and harness stay on the host and retain loopback URLs. Chromium and GitHub fallback keep native browser installation and launch. Update both workspace Playwright dependencies, the image index digest, and the expected client version/revision checks together; verify the installed release's native launch timeout and real WebKit acceptance before changing the pin.
40+
3941
Every HTTP and browser group consumes the same production build in an isolated job/database. Electron also uses that build and skips its development-route warm-up. Local HTTP and Electron development runs retain their dev-server behavior. Build inputs exclude test-only files so a harness edit can reuse an unchanged production build.
4042

4143
Desktop inbox protocol, assistant actions UI, and organization search UI run as three parallel HTTP jobs with isolated databases, apps, and reports. Each harness owns its acceptance task, so an edit selects its own job. The inbox protocol task depends on server and realtime code; the browser tasks also depend on workspace UI. Desktop inbox needs no browser. The UI jobs and browser extension install Chromium while their app starts, waiting for both before executing assertions; browser-install failures still fail the job and preserve the install log. No acceptance assertions are removed by the split.
@@ -44,7 +46,7 @@ Live Electron acceptance tracks its live-app suite, fixtures, and Playwright con
4446

4547
Desktop smoke weights also stay outside the desktop runtime task. Editing weights reruns desktop partition coverage without selecting live-app acceptance.
4648

47-
Browser extension acceptance runs native interaction and access-revalidation groups in parallel jobs with isolated databases, apps, browser profiles, and reports. Both retain the same task boundary, so an affected extension harness selects both groups. Native streaming and first-open capture checks retain real timers. Access checks install Playwright's clock before the app mounts: advancing production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. The local `browser-extension` group still runs the complete suite; live-provider runs require this complete mode and retain native timing throughout.
49+
Browser extension acceptance runs navigation, composer, and access-revalidation groups in three parallel jobs with isolated databases, apps, browser profiles, and reports. All three retain the same task boundary, so an affected extension harness selects every group. Navigation and composer checks retain real timers. Access checks install Playwright's clock before the app mounts: advancing production polling timers preserves real HTTP and database authorization checks without waiting through each polling interval. Capture timestamps originate in Chromium's isolated world, so the harness restores wall time after each advance. The local `browser-extension` group still runs the complete suite; live-provider runs require this complete mode and retain native timing throughout.
4850

4951
Integration shard weights come from successful reports for both push and migration provisioning paths. Refresh them with `scripts/update-integration-durations.ts`, then check the new partition against a separate complete run before adopting it. Weights only change scheduling; discovery still owns coverage. The current refresh uses runs [41348786822](https://github.com/simstudioai/sim/actions/runs/41348786822) and [41377145857](https://github.com/simstudioai/sim/actions/runs/41377145857), with [41377810718](https://github.com/simstudioai/sim/actions/runs/41377810718) held out for validation.
5052

‎.github/ci-suites.json‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,11 @@
2525
"task": "ci:e2e-organization-search"
2626
},
2727
{
28-
"group": "browser-extension-native",
28+
"group": "browser-extension-navigation",
29+
"task": "ci:e2e-browser-extension"
30+
},
31+
{
32+
"group": "browser-extension-composer",
2933
"task": "ci:e2e-browser-extension"
3034
},
3135
{

‎.github/scripts/http-e2e.sh‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# CI groups, which reuse the production build from `e2e-build`.
44
# The job extracts it into apps/sim/.next before running these suites.
55
#
6-
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-access|project-files|mobile|seo> (run from apps/sim)
6+
# Usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-navigation|browser-extension-composer|browser-extension-access|project-files|mobile|seo> (run from apps/sim)
77
#
88
# The job provides DATABASE_URL, BETTER_AUTH_SECRET and ENCRYPTION_KEY; each group sets the rest of
99
# its app's environment here. Reports and server logs land in $RUNNER_TEMP/e2e.
@@ -20,7 +20,7 @@
2020
# telemetry flush runs detached and still writes .next/dev).
2121
set -euo pipefail
2222

23-
group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-access|project-files|mobile|seo>}
23+
group=${1:?usage: http-e2e.sh <scim|cli|stop-after|desktop-inbox|assistant-actions|organization-search|browser-extension|browser-extension-native|browser-extension-navigation|browser-extension-composer|browser-extension-access|project-files|mobile|seo>}
2424
report_dir="$RUNNER_TEMP/e2e"
2525
ready_timeout_seconds=600
2626
mkdir -p "$report_dir"
@@ -164,7 +164,11 @@ case "$group" in
164164
export NEXT_PUBLIC_CHAT_DISABLED=true
165165
export SIM_LINKS=true
166166
export PEOPLE=true
167+
export CRON_SECRET=people-directory-sync-local-e2e-secret
168+
start_browser_install people chromium
167169
start_app scim 3017 SCIM record-http-status
170+
wait_for_browsers
171+
export PEOPLE_E2E_SCREENSHOT_DIR="$report_dir/people-ui-report.json.artifacts"
168172
SCIM_E2E_BASE_URL="$local_app_url" \
169173
SCIM_E2E_DATABASE_URL="$DATABASE_URL" \
170174
SCIM_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
@@ -233,7 +237,7 @@ case "$group" in
233237
export COPILOT_TOOL_PERMISSIONS_ENABLED=true
234238
export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
235239
export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true NEXT_PUBLIC_ORGANIZATIONS_ENABLED=true
236-
if [ "$group" != desktop-inbox ]; then start_browser_install "$group" chromium; fi
240+
if [ "$group" != desktop-inbox ]; then start_browser_install "$group" --only-shell chromium; fi
237241
export INTERNAL_API_SECRET=desktop-inbox-http-ci-local-secret-at-least-32-characters
238242
export DB_TX_TRIPWIRE=throw
239243
start_app "$group" 3019 "$group" record-http-status
@@ -264,18 +268,20 @@ case "$group" in
264268
esac
265269
;;
266270

267-
browser-extension|browser-extension-native|browser-extension-access)
271+
browser-extension|browser-extension-native|browser-extension-navigation|browser-extension-composer|browser-extension-access)
268272
extension_group=all
269273
case "$group" in
270274
browser-extension-native) extension_group=native ;;
275+
browser-extension-navigation) extension_group=navigation ;;
276+
browser-extension-composer) extension_group=composer ;;
271277
browser-extension-access) extension_group=access ;;
272278
esac
273279
export REDIS_URL=redis://127.0.0.1:6379
274280
export NEXT_PUBLIC_FORCE_HOSTED=false
275281
export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
276282
export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true
277283
export API_ENCRYPTION_KEY="$ENCRYPTION_KEY"
278-
start_browser_install "$group" chromium
284+
start_browser_install "$group" --no-shell chromium
279285
extension_identity=$(bun - <<'JS'
280286
import { generateKeyPairSync, createHash } from 'node:crypto'
281287
const { publicKey } = generateKeyPairSync('rsa', { modulusLength: 2048 })
@@ -312,10 +318,19 @@ JS
312318
export INTERNAL_API_SECRET=mobile-http-ci-local-secret-at-least-32-characters
313319
export NEXT_PUBLIC_SOCKET_URL=http://127.0.0.1:3025 SOCKET_SERVER_URL=http://127.0.0.1:3025
314320
read -ra browsers <<< "${MOBILE_E2E_BROWSER:-chromium webkit}"
315-
start_browser_install mobile "${browsers[@]}"
321+
if [ "${PLAYWRIGHT_WEBKIT_WS_ENDPOINT+x}" = x ]; then
322+
if [ "${MOBILE_E2E_BROWSER:-}" != webkit ]; then
323+
echo "::error::A remote endpoint requires explicit WebKit selection." >&2
324+
exit 1
325+
fi
326+
else
327+
start_browser_install mobile --only-shell "${browsers[@]}"
328+
fi
316329
start_app mobile 3024 'mobile browser' record-http-status
317330
start_realtime mobile 3025
318-
wait_for_browsers
331+
if [ "${PLAYWRIGHT_WEBKIT_WS_ENDPOINT+x}" != x ]; then
332+
wait_for_browsers
333+
fi
319334
MOBILE_E2E_BASE_URL="$NEXT_PUBLIC_APP_URL" \
320335
MOBILE_E2E_DATABASE_URL="$DATABASE_URL" \
321336
MOBILE_E2E_AUTH_SECRET="$BETTER_AUTH_SECRET" \
Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
#!/usr/bin/env bash
2+
# Owns one pinned WebKit server; start prints its loopback endpoint, stop preserves its logs.
3+
set -euo pipefail
4+
umask 077
5+
6+
action=${1:-}
7+
state_argument=${2:-}
8+
if [ "$#" -ne 2 ] || { [ "$action" != start ] && [ "$action" != stop ]; }; then
9+
echo 'Usage: playwright-server.sh <start|stop> <state-directory-under-RUNNER_TEMP>' >&2
10+
exit 2
11+
fi
12+
: "${RUNNER_TEMP:?RUNNER_TEMP must be set}"
13+
: "${GITHUB_REPOSITORY:?GITHUB_REPOSITORY must be set}"
14+
: "${GITHUB_RUN_ID:?GITHUB_RUN_ID must be set}"
15+
: "${GITHUB_RUN_ATTEMPT:?GITHUB_RUN_ATTEMPT must be set}"
16+
: "${GITHUB_JOB:?GITHUB_JOB must be set}"
17+
if ! [[ "$GITHUB_RUN_ID" =~ ^[1-9][0-9]*$ && "$GITHUB_RUN_ATTEMPT" =~ ^[1-9][0-9]*$ && "$GITHUB_JOB" =~ ^[a-zA-Z0-9_-]+$ ]]; then
18+
echo 'Invalid GitHub job ownership metadata.' >&2
19+
exit 2
20+
fi
21+
state=$(node - "$RUNNER_TEMP" "$state_argument" <<'JS'
22+
const fs = require('node:fs')
23+
const path = require('node:path')
24+
const root = fs.realpathSync(process.argv[2])
25+
const state = path.resolve(process.argv[3])
26+
if (path.dirname(state) !== root || (fs.existsSync(state) && fs.realpathSync(state) !== state))
27+
throw new Error('State must be a direct, non-symlink child of RUNNER_TEMP')
28+
process.stdout.write(state)
29+
JS
30+
)
31+
state_key=$(node -e 'process.stdout.write(require("node:crypto").createHash("sha256").update(process.argv[1]).digest("hex"))' "$state")
32+
image='mcr.microsoft.com/playwright@sha256:2f05f8a734509a7b3f26a912639488ee08f58a7b292d14d62271bcff9038768f'
33+
starting=false
34+
35+
stop_container() {
36+
local cid inspection remaining status=0
37+
[ -f "$state/cid" ] || return 0
38+
cid=$(< "$state/cid")
39+
if ! [[ "$cid" =~ ^[0-9a-f]{64}$ ]]; then
40+
echo 'Refusing to stop an invalid container ID.' >&2
41+
return 1
42+
fi
43+
if ! inspection=$(docker container inspect "$cid" 2> "$state/inspect-error.log"); then
44+
remaining=$(docker container ls --all --no-trunc --filter "id=$cid" --format '{{.ID}}') || return 1
45+
[ -z "$remaining" ] && return 0
46+
cat "$state/inspect-error.log" >&2
47+
return 1
48+
fi
49+
if ! jq -e --arg repository "$GITHUB_REPOSITORY" --arg run "$GITHUB_RUN_ID" \
50+
--arg attempt "$GITHUB_RUN_ATTEMPT" --arg job "$GITHUB_JOB" --arg state "$state_key" \
51+
'.[0].Config.Labels | .["com.sim.ci.playwright-server"] == "true"
52+
and .["com.sim.ci.repository"] == $repository and .["com.sim.ci.run"] == $run
53+
and .["com.sim.ci.attempt"] == $attempt and .["com.sim.ci.job"] == $job
54+
and .["com.sim.ci.state"] == $state' <<< "$inspection" > /dev/null; then
55+
echo 'Refusing to stop a container owned by another job or state directory.' >&2
56+
return 1
57+
fi
58+
docker stop --timeout 10 "$cid" > "$state/stop.log" 2>&1 || status=1
59+
docker logs "$cid" > "$state/server.log" 2>&1 || status=1
60+
docker rm "$cid" > "$state/remove.log" 2>&1 || status=1
61+
return "$status"
62+
}
63+
64+
trap '
65+
status=$?
66+
trap - EXIT
67+
if [ "$status" -ne 0 ] && [ "$starting" = true ]; then
68+
stop_container || echo "Owned Playwright server cleanup failed; preserve its state and logs." >&2
69+
fi
70+
exit "$status"
71+
' EXIT
72+
trap 'exit 130' INT
73+
trap 'exit 143' TERM
74+
75+
if [ "$action" = stop ]; then
76+
stop_container
77+
exit 0
78+
fi
79+
if [ -e "$state/cid" ]; then
80+
echo 'Refusing to reuse an existing Playwright server state.' >&2
81+
exit 1
82+
fi
83+
mkdir -p -- "$state"
84+
repo_root=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)
85+
driver=$(node - "$repo_root/apps/sim" <<'JS'
86+
const fs = require('node:fs')
87+
const path = require('node:path')
88+
const packagePath = require.resolve('playwright-core/package.json', { paths: [process.argv[2]] })
89+
const root = fs.realpathSync(path.dirname(packagePath))
90+
const manifest = JSON.parse(fs.readFileSync(path.join(root, 'browsers.json'), 'utf8'))
91+
if (JSON.parse(fs.readFileSync(packagePath, 'utf8')).version !== '1.64.1' ||
92+
manifest.browsers.find(browser => browser.name === 'webkit')?.revision !== '2370')
93+
throw new Error('The pinned WebKit image requires Playwright 1.64.1, revision 2370')
94+
process.stdout.write(root)
95+
JS
96+
)
97+
docker pull "$image" > "$state/pull.log" 2>&1
98+
starting=true
99+
docker run --detach --init --cidfile "$state/cid" \
100+
--name "sim-ci-playwright-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT-$GITHUB_JOB-$$" \
101+
--label com.sim.ci.playwright-server=true --label "com.sim.ci.repository=$GITHUB_REPOSITORY" \
102+
--label "com.sim.ci.run=$GITHUB_RUN_ID" --label "com.sim.ci.attempt=$GITHUB_RUN_ATTEMPT" \
103+
--label "com.sim.ci.job=$GITHUB_JOB" --label "com.sim.ci.state=$state_key" \
104+
--network host --cpus 8 --memory 4g --memory-swap 4g --pids-limit 512 --shm-size 256m \
105+
--mount "type=bind,src=$driver,dst=/opt/playwright-core,readonly" \
106+
"$image" node /opt/playwright-core/cli.js run-server --host 127.0.0.1 --port 0 \
107+
> "$state/start.log" 2>&1
108+
cid=$(< "$state/cid")
109+
if ! [[ "$cid" =~ ^[0-9a-f]{64}$ ]]; then
110+
echo 'Docker did not return a complete container ID.' >&2
111+
exit 1
112+
fi
113+
docker container inspect "$cid" > "$state/metadata.json"
114+
deadline=$((SECONDS + 30))
115+
while [ "$SECONDS" -lt "$deadline" ]; do
116+
live=$(docker container inspect --format '{{.State.Running}} {{.State.Pid}}' "$cid")
117+
if ! [[ "$live" =~ ^true\ [1-9][0-9]*$ ]]; then
118+
docker logs "$cid" > "$state/server.log" 2>&1
119+
echo 'Playwright server exited before readiness.' >&2
120+
exit 1
121+
fi
122+
docker logs "$cid" > "$state/server.log" 2>&1
123+
endpoint=$(sed -n 's@^Listening on \(ws://127\.0\.0\.1:[1-9][0-9]*/\)$@\1@p' "$state/server.log")
124+
if [ -n "$endpoint" ]; then
125+
node - "$endpoint" <<'JS'
126+
const endpoint = process.argv[2]
127+
if (!/^ws:\/\/127\.0\.0\.1:[1-9]\d*\/$/.test(endpoint)) throw new Error('Invalid server endpoint')
128+
new URL(endpoint)
129+
process.stdout.write(`${endpoint}\n`)
130+
JS
131+
exit 0
132+
fi
133+
sleep 0.1
134+
done
135+
echo 'Playwright server did not report a live loopback endpoint within 30 seconds.' >&2
136+
exit 1

0 commit comments

Comments
 (0)