Repository navigation
fix(annotations): reliable simulator element highlighting - #10
Merged
Merged
Conversation
CoreSimulatorBridge can lose its accessibility connection after backboardd and SpringBoard restart, so describe-ui reports no data for every foreground app and annotations cannot highlight elements. Restart the bridge and retry once, at most every 10 seconds per device, and count restart outcomes.
Target matching compared the accessibility app name with the last bundle ID segment, so Expo Go (host.exp.Exponent) never matched and annotations fell back to coarse accessibility elements. When no name matches on the device, look up the iOS simulator's foreground bundle ID and match it exactly. Split native snapshots by platform and read Android aliases in parallel with accessibility.
Hit testing preferred the smallest element at a point regardless of paint order, so a card stack's slightly inset next card won over the visible card and highlights appeared offset. Image, text or SVG painted later in another branch now hides earlier React Native elements at that point. Index each component list once for ancestor lookups.
iOS bezel definitions can describe a smaller screen cutout than the device, such as 400x872 points for a 402x874 iPhone, which stretched highlights toward the bottom right. Inspection now returns the accessibility root's screen size through a typed tree result, and the store rescales the capture and pending pointer state to it. Drags keep their latest point when inspection finishes mid-drag.
Opening the panel with a booted iOS simulator and a running Android device showed both platforms side by side. Open iOS by default, matching the documented behavior, and start from iOS before devices load. Android still opens alone when it has the only active device, and the platform select can show both.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes several problems that made the iOS simulator annotation tool unreliable. Also changes the default layout so only iOS opens when both platforms have a running device.
describe-uiread then returnsno accessibility data.Baguette.describeUinow restarts the bridge (launchctl kickstart) and retries once, at most every 10 s per device.host.exp.Exponent), so React Native inspection was skipped. When no name matches on the device, the iOS simulator's foreground bundle ID is used as an exact match.readTreeresult, and the store rescales the capture and any in-progress pointer state to it.docs/devices.mddescribes. The platform select can still show both.Commits are split per fix. The final commit bumps the version to 0.1.139.
Telemetry
ios.accessibility_bridge.restartscount with a fixedoutcome(recovered,unavailable,failed). Failed restarts also report a handledsimulator.accessibility_bridgeerror. Documented indocs/telemetry.md.ui.annotations.inspection,ui.annotations.tree_processingandui.annotations.runtime_availablestill cover the changed inspection and hit-testing paths. No new identifiers or content are sent.Testing
npm test: 605 pass. 7 tests fail the same way onmain(public release, Android CLI and packaged CLI, owned startup).tscreports no new errors; two existing errors ininspection-tools.tsare gone.