Repository navigation
Description
Investigate the Android smoke:automation-system Back-to-Settings transition without changing the shutdown scope in #4063. Follow-up to umbrella #4045.
Observed evidence
- Failed Android job, PR head
df42fdb25c49206a67cd8340b85042fb0ae2926f, actual CI merge1e4261finto65ec68be9f2b8ed8ea5deef479db334c1b73f2fb, package0.21.36-dev. - Failure artifact contains step history, daemon/request logs, coverage report and
failed-step-88-snapshot.json/PNG. live-automation-scenario.tscaptures the Automation baseline withsnapshot -i, calls bareback, then immediately callsdiff snapshot -i. The assertion at line 201 expects removed “Open automation alert” and added “Open automation lab”. Only afterwards does the test wait for Settings.- Back and diff both returned success. The diff reported zero additions/removals and 21 unchanged Automation nodes. The failure snapshot subsequently showed Settings and “Open automation lab”. Snapshot/Back/diff durations were 765/556/734 ms. These facts do not distinguish delayed navigation from stale capture or another cause.
- Current Android job passed at head
ccdc6d48d35681d335432e400ba0ffb1ed472f22, actual merge2504514. Its artifact recordssmoke:automation-systemcompleted. The scenario source and package version are identical between the two CI merges. This is a later successful run, not a controlled reproduction or proof of root cause.
Required behavior
Establish whether the failure belongs to navigation readiness, snapshot freshness, or the test's baseline/readiness contract. Preserve the meaningful removed/added assertions and snapshot baseline. Do not add sleeps, widen budgets, weaken assertions, or attribute the failure to contention without evidence. Do not change shutdown behavior to hide this failure.
If readiness is the missing precondition, use the existing readiness owner. Verify its baseline behavior on the frozen failing CI tree before changing the test. A settled Back may publish a new baseline, so an added settle flag followed by a separate diff is not automatically equivalent.
Observable completion
- Retain the failed and successful run identities and artifacts.
- In a disposable checkout, record navigation dispatch, immediate capture/diff, readiness, cache/freshness metadata and baseline publication in one bounded discriminating experiment.
- State the demonstrated cause or explicitly retain a diagnostic-only result if it cannot be reproduced.
- Fix the demonstrated boundary and add a credible regression/control that fails before the fix.
- Preserve the Settings navigation and removed/added checks. Run focused proof and serialized exact-head affected validation; record any live evidence at its named SHA.
Dependencies and limits
No implementation dependency on #4063 is established. Diagnosis comes before an implementation decision. The current green Android job does not establish that the failure is fixed, and no base-versus-head comparison establishes that #4063 caused it. Keep this issue in triage until the owning boundary is demonstrated.
Activity
apex-sentinel-staging commentedon Oct 30, 2026
Apex Sentinel · Issue review
Suggested label:
bugThe issue documents a concrete failed Android CI job with logs/artifacts and describes a real test/automation inconsistency: a back transition followed by immediate diff reported zero changes while the later snapshot showed Settings. It gives enough evidence for a maintainer to investigate and reproduce.
Required sections
thymikee commentedon Nov 2, 2026
Two more occurrences of the same signature on 2026-11-02, both on PR branches that don't touch Android. In each, the live Android emulator smoke's snapshot diff right after an action shows no change, as if the capture ran before the screen updated.
expected removed snapshot line containing Open automation alert. Summary: additions 0, removals 0, unchanged 21.expected added snapshot line containing Open automation lab. Summary: additions 0, removals 1, unchanged 20 (assertDiffLine,live-assertions.ts:63).Both passed on rerun, and main stays green. With this issue's Back-to-Settings case, that makes three occurrences of one class: the post-action snapshot diff is captured before the transition settles.
thymikee commentedon Nov 2, 2026
One more occurrence on 2026-11-02: #4160 (iOS and docs only) at af69d17, run 41460910773, job 125329396005. Same
expected added snapshot line containing Open automation lab(assertDiffLine,live-assertions.ts:63), this time with additions 0, removals 9, unchanged 12. That's the fourth time today, the second on this PR, and main stays green.