Skip to content

fix(annotations): reliable simulator element highlighting - #10

Merged
lauridskern merged 6 commits into
mainfrom
fix/annotation-inspection
Oct 8, 2026
Merged

lauridskern merged 6 commits into
mainfrom
fix/annotation-inspection

Conversation

@lauridskern

Copy link
Copy Markdown
Collaborator

Summary

Fixes several problems that made the iOS simulator annotation tool unreliable. Also changes the default layout so only iOS opens when both platforms have a running device.

  • No highlights at all: CoreSimulatorBridge can lose its accessibility connection after backboardd and SpringBoard restart. Every describe-ui read then returns no accessibility data. Baguette.describeUi now restarts the bridge (launchctl kickstart) and retries once, at most every 10 s per device.
  • Only coarse elements in Expo Go: Metro targets were matched by app name. Expo Go's name ("Expo Go") never matches its bundle ID (host.exp.Exponent), so React Native inspection was skipped. When no name matches on the device, the iOS simulator's foreground bundle ID is used as an exact match.
  • Offset highlights on card stacks: hit-testing picked the smallest element at a point regardless of paint order, so the slightly inset next card in a stack won over the visible one. Image, text or SVG painted later in another branch now hides earlier React Native elements at that point.
  • Highlights a few points off near the bottom right: the bezel definition describes a 400×872 pt screen cutout for a 402×874 pt iPhone. Inspection now returns the accessibility root's screen size through a typed readTree result, and the store rescales the capture and any in-progress pointer state to it.
  • Both platforms opening: with an iOS simulator and an Android device both running, the panel now opens iOS only, as docs/devices.md describes. The platform select can still show both.

Commits are split per fix. The final commit bumps the version to 0.1.139.

Telemetry

  • New ios.accessibility_bridge.restarts count with a fixed outcome (recovered, unavailable, failed). Failed restarts also report a handled simulator.accessibility_bridge error. Documented in docs/telemetry.md.
  • Existing ui.annotations.inspection, ui.annotations.tree_processing and ui.annotations.runtime_available still cover the changed inspection and hit-testing paths. No new identifiers or content are sent.

Testing

  • New tests cover the bridge restart and its rate limit, the Expo Go foreground match (mismatched app, failed lookup, name match skipping the lookup), card-stack occlusion, a later transparent overlay not hiding content, and the 400→402 screen correction including in-progress drags.
  • npm test: 605 pass. 7 tests fail the same way on main (public release, Android CLI and packaged CLI, owned startup).
  • tsc reports no new errors; two existing errors in inspection-tools.ts are gone.
  • Checked live on an iOS 27 iPhone 18 Pro simulator running an Expo Go app:
    • accessibility recovered after restarting the bridge
    • Expo Go matched its Metro target
    • the hidden card was picked at 0 of 11,430 sampled points (previously 10,554)
    • hover hit-testing takes about 7 µs
    • the server reports a 402×874 screen

CoreSimulatorBridge can lose its accessibility connection after
backboardd and SpringBoard restart, so describe-ui reports no data for
every foreground app and annotations cannot highlight elements. Restart
the bridge and retry once, at most every 10 seconds per device, and
count restart outcomes.
Target matching compared the accessibility app name with the last
bundle ID segment, so Expo Go (host.exp.Exponent) never matched and
annotations fell back to coarse accessibility elements. When no name
matches on the device, look up the iOS simulator's foreground bundle ID
and match it exactly. Split native snapshots by platform and read
Android aliases in parallel with accessibility.
Hit testing preferred the smallest element at a point regardless of
paint order, so a card stack's slightly inset next card won over the
visible card and highlights appeared offset. Image, text or SVG painted
later in another branch now hides earlier React Native elements at that
point. Index each component list once for ancestor lookups.
iOS bezel definitions can describe a smaller screen cutout than the
device, such as 400x872 points for a 402x874 iPhone, which stretched
highlights toward the bottom right. Inspection now returns the
accessibility root's screen size through a typed tree result, and the
store rescales the capture and pending pointer state to it. Drags keep
their latest point when inspection finishes mid-drag.
Opening the panel with a booted iOS simulator and a running Android
device showed both platforms side by side. Open iOS by default, matching
the documented behavior, and start from iOS before devices load. Android
still opens alone when it has the only active device, and the platform
select can show both.
@lauridskern
lauridskern merged commit 234eb5c into main Oct 8, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant