Skip to content

Diagnose Android Back-to-Settings diff capturing the previous screen #4105

Open

Description

@thymikee

Investigate the Android smoke:automation-system Back-to-Settings transition without changing the shutdown scope in #4063. Follow-up to umbrella #4045.

Observed evidence

  • Failed Android job, PR head df42fdb25c49206a67cd8340b85042fb0ae2926f, actual CI merge 1e4261f into 65ec68be9f2b8ed8ea5deef479db334c1b73f2fb, package 0.21.36-dev.
  • Failure artifact contains step history, daemon/request logs, coverage report and failed-step-88-snapshot.json/PNG.
  • live-automation-scenario.ts captures the Automation baseline with snapshot -i, calls bare back, then immediately calls diff snapshot -i. The assertion at line 201 expects removed “Open automation alert” and added “Open automation lab”. Only afterwards does the test wait for Settings.
  • Back and diff both returned success. The diff reported zero additions/removals and 21 unchanged Automation nodes. The failure snapshot subsequently showed Settings and “Open automation lab”. Snapshot/Back/diff durations were 765/556/734 ms. These facts do not distinguish delayed navigation from stale capture or another cause.
  • Current Android job passed at head ccdc6d48d35681d335432e400ba0ffb1ed472f22, actual merge 2504514. Its artifact records smoke:automation-system completed. The scenario source and package version are identical between the two CI merges. This is a later successful run, not a controlled reproduction or proof of root cause.

Required behavior

Establish whether the failure belongs to navigation readiness, snapshot freshness, or the test's baseline/readiness contract. Preserve the meaningful removed/added assertions and snapshot baseline. Do not add sleeps, widen budgets, weaken assertions, or attribute the failure to contention without evidence. Do not change shutdown behavior to hide this failure.

If readiness is the missing precondition, use the existing readiness owner. Verify its baseline behavior on the frozen failing CI tree before changing the test. A settled Back may publish a new baseline, so an added settle flag followed by a separate diff is not automatically equivalent.

Observable completion

  • Retain the failed and successful run identities and artifacts.
  • In a disposable checkout, record navigation dispatch, immediate capture/diff, readiness, cache/freshness metadata and baseline publication in one bounded discriminating experiment.
  • State the demonstrated cause or explicitly retain a diagnostic-only result if it cannot be reproduced.
  • Fix the demonstrated boundary and add a credible regression/control that fails before the fix.
  • Preserve the Settings navigation and removed/added checks. Run focused proof and serialized exact-head affected validation; record any live evidence at its named SHA.

Dependencies and limits

No implementation dependency on #4063 is established. Diagnosis comes before an implementation decision. The current green Android job does not establish that the failure is fixed, and no base-versus-head comparison establishes that #4063 caused it. Keep this issue in triage until the owning boundary is demonstrated.

Activity

  1. apex-sentinel-staging commented on Oct 30, 2026

    @apex-sentinel-staging

    Apex Sentinel · Issue review

    Suggested label: bug

    The issue documents a concrete failed Android CI job with logs/artifacts and describes a real test/automation inconsistency: a back transition followed by immediate diff reported zero changes while the later snapshot showed Settings. It gives enough evidence for a maintainer to investigate and reproduce.

    Required sections

    • Steps to reproduce (missing)
    • Expected and actual behavior (missing)
    • Version or environment (missing)
  2. thymikee commented on Nov 2, 2026

    @thymikee
    MemberAuthor

    Two more occurrences of the same signature on 2026-11-02, both on PR branches that don't touch Android. In each, the live Android emulator smoke's snapshot diff right after an action shows no change, as if the capture ran before the screen updated.

    Both passed on rerun, and main stays green. With this issue's Back-to-Settings case, that makes three occurrences of one class: the post-action snapshot diff is captured before the transition settles.

  3. added
    bugSomething isn't working
    and removed on Nov 2, 2026
  4. thymikee commented on Nov 2, 2026

    @thymikee
    MemberAuthor

    One more occurrence on 2026-11-02: #4160 (iOS and docs only) at af69d17, run 41460910773, job 125329396005. Same expected added snapshot line containing Open automation lab (assertDiffLine, live-assertions.ts:63), this time with additions 0, removals 9, unchanged 12. That's the fourth time today, the second on this PR, and main stays green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions