fix(annotations): reliable simulator element highlighting

.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "mobile-dev",
-  "version": "0.1.134",
+  "version": "0.1.139",
   "description": "Keep a simulator beside the chat while building, running, or debugging iOS and Android apps. Stream devices, read native and Metro logs, render interactive CPU, memory, and FPS recordings, and control apps with Mobile Dev tools.",
   "author": {
     "name": "Callstack"

docs/architecture.md

@@ -48,7 +48,7 @@ An iOS JPEG decode error restarts capture through `mobile_stream_reset` with the
 
 iOS boot and shutdown calls run in order for each device and wait up to two minutes for the reported state. Boot skips the backend route if the device already runs or is booting. This avoids a second boot and Baguette's input repair on a running device. Shutdown closes that device's panel streams after the device stops.
 
-On macOS 27 with Xcode 27, the bundled Baguette can list simulators and capture frames. Device Hub can stop taps, buttons, and keys from reaching an iOS 27 device. Connections await a fresh status check. Gestures use the last completed result while one expired check refreshes in the background, so the approximately 150–200 ms command no longer delays every second of input. A new block becomes visible when that refresh completes. When blocked, it shows a notice. Ask Codex to repair input with `mobile_repair_input`. The tool runs the bundled Baguette's `heal` command. Reconnect capture afterward to use new input handles. It restarts backboardd and SpringBoard without rebooting the device. Relaunching Device Hub can block input again. The panel automatically repairs a confirmed Device Hub block when connecting and reconnects if input becomes blocked later. Repair closes running apps. It limits automatic attempts to once per device per minute; a repeated block or failed repair remains visible in the bottom bar. Model input alone does not trigger automatic repair. Baguette's boot route also repairs input after boot. Do not run the repair to diagnose video. See [Baguette's Device Hub notes](https://github.com/tddworks/baguette/blob/main/docs/features/device-hub/README.md).
+On macOS 27 with Xcode 27, the bundled Baguette can list simulators and capture frames. Device Hub can stop taps, buttons, and keys from reaching an iOS 27 device. Connections await a fresh status check. Gestures use the last completed result while one expired check refreshes in the background, so the approximately 150–200 ms command no longer delays every second of input. A new block becomes visible when that refresh completes. When blocked, it shows a notice. Ask Codex to repair input with `mobile_repair_input`. The tool runs the bundled Baguette's `heal` command. Reconnect capture afterward to use new input handles. It restarts backboardd and SpringBoard without rebooting the device. CoreSimulatorBridge can afterwards report no accessibility data for every foreground app until it restarts. UI reads that report no accessibility data restart the bridge and retry once, at most every 10 seconds per device. Relaunching Device Hub can block input again. The panel automatically repairs a confirmed Device Hub block when connecting and reconnects if input becomes blocked later. Repair closes running apps. It limits automatic attempts to once per device per minute; a repeated block or failed repair remains visible in the bottom bar. Model input alone does not trigger automatic repair. Baguette's boot route also repairs input after boot. Do not run the repair to diagnose video. See [Baguette's Device Hub notes](https://github.com/tddworks/baguette/blob/main/docs/features/device-hub/README.md).
 
 UI resource addresses include the release version so Codex can load new HTML after an update. The original simulator and workspace addresses and the old v1 through v6 simulator addresses still return the current UI. After updating, restart Codex once if it still uses an older MCP process.
 
@@ -64,7 +64,7 @@ Send to chat shows only "Apply these annotations." Edit guidance stays in assist
 
 Screen annotations use the device's accessibility tree for native and React Native apps. Click an exposed element to add a note. Clicks and manual regions work while inspection loads; an open point note adopts the real element when its bounds arrive. If runtime inspection fails, the panel retries the native accessibility tool. The note popup can select an enclosing element when the tree includes one; flat Android snapshots offer elements whose reported bounds contain the selection. Drag to mark a region when the app does not expose a view. Region annotations carry user-selected bounds, not inferred component names. React Native development apps can also supply runtime component names and measured native bounds through the MCP server. The panel makes no browser connection to Metro. The server uses an existing Metro server at `http://127.0.0.1:8081`, requires a unique device/app match and support for multiple debuggers, and closes its connection after each snapshot. It does not start Metro or take over another debugger. The `mobile_inspect_ui` tool accepts `metroUrl` and `targetId` for explicit targets.
 
-The React Native adapter reads the DevTools hook to locate mounted components, then uses the documented native element `getBoundingClientRect()` API. Native screen containers with empty DOM bounds fall back to `measureInWindow()` callbacks through a temporary CDP binding. The server removes that binding and its global function before closing the connection. Logical component bounds cover their measured native children.
+The React Native adapter reads the DevTools hook to locate mounted components, then uses the documented native element `getBoundingClientRect()` API. Native screen containers with empty DOM bounds fall back to `measureInWindow()` callbacks through a temporary CDP binding. The server removes that binding and its global function before closing the connection. Logical component bounds cover their measured native children. Hit testing follows paint order: image, text or SVG content rendered later in another branch hides earlier React Native elements at that point, so a card stack's next card stays unselectable under the top card. A later plain container does not hide content, because the snapshot has no background information. iOS bezel definitions can describe a screen cutout slightly smaller than the device, such as 400×872 points for a 402×874 iPhone. Inspection returns the accessibility root's screen size, and the panel maps element bounds and annotation points with it instead. Automatic inspection picks the Metro target on the selected device whose bundle ID ends with the accessibility app name. When that finds none, such as Expo Go (`host.exp.Exponent`), iOS simulators look up the foreground app's bundle ID and match it exactly. A backgrounded React Native app never stands in for another foreground app.
 
 Traversal uses a work queue so deeply nested navigator wrappers do not cut off visible rows. Native-stack screens marked inactive through `activityState` and `aria-hidden` are excluded, along with disconnected native elements. Decorative elements marked only `aria-hidden` remain selectable. Text labels use string children when available, and text outlines follow the component's layout bounds.
 

docs/devices.md

@@ -14,7 +14,7 @@ Android logs need `adb` from an installed Android SDK. The reader checks `ANDROI
 
 ## Device panel
 
-Open a new chat after installing. Open Mobile Dev in the sidebar or call `mobile_open_workspace` for the fullscreen view. Call `mobile_open_simulator` for the panel beside a chat. iOS opens by default. Enable Android from the toolbar to show both panels side by side. Each panel has a device dropdown, Home, App Switcher, and Screenshot. Pick a device in each panel. Use the iOS and Android toggles to show either, both, or neither simulator. Selecting a device boots it if needed, then connects its screen. A selected running device connects automatically. The panel uses Apple’s device bezel and screen mask from the installed DeviceKit assets, with a simple frame as a fallback when assets are unavailable. Use the settings button at the bottom right for appearance, text size, location, and the device frame. iOS also offers contrast; Android offers rotation. The menu shows only settings supported by the bundled backend. Click the screen to type or drag. Closing the panel closes its stream.
+Open a new chat after installing. Open Mobile Dev in the sidebar or call `mobile_open_workspace` for the fullscreen view. Call `mobile_open_simulator` for the panel beside a chat. iOS opens by default, including when an iOS device and an Android device are both running. Android opens instead when it has the only running device. Enable both platforms from the toolbar to show the panels side by side. Each panel has a device dropdown, Home, App Switcher, and Screenshot. Pick a device in each panel. Use the iOS and Android toggles to show either, both, or neither simulator. Selecting a device boots it if needed, then connects its screen. A selected running device connects automatically. The panel uses Apple’s device bezel and screen mask from the installed DeviceKit assets, with a simple frame as a fallback when assets are unavailable. Use the settings button at the bottom right for appearance, text size, location, and the device frame. iOS also offers contrast; Android offers rotation. The menu shows only settings supported by the bundled backend. Click the screen to type or drag. Closing the panel closes its stream.
 
 Use Select in the simulator toolbar to pause the screen. Hover to outline a component, then click to add a note. React Native development apps can supply runtime elements when accessibility omits a view. Drag to mark a region when neither source exposes it. Saved notes leave numbered blue bubbles. Notes attach text and available element details to your next chat message. The captured screen stays local for editing; annotations never attach screenshots. Click a bubble to edit or remove a note, or use Send to chat to send all notes for that device. If chat is unavailable, the panel keeps the notes and retries when you return. A sent or cleared batch starts again at 1.
 

docs/telemetry.md

@@ -2,6 +2,8 @@
 
 [Back to README](../README.md) · [Contributing](../CONTRIBUTING.md)
 
+Since 0.1.139, `ios.accessibility_bridge.restarts` counts restarts of the simulator's CoreSimulatorBridge after a describe-UI read reported no accessibility data, with a fixed `outcome` (`recovered`, `unavailable` or `failed`). Failed restarts also report a handled `simulator.accessibility_bridge` error. The existing `inspection.tool` errors, sampled MCP traces and `ui.annotations.inspection` timings cover the retried read. No device identifiers, accessibility content or command output are sent.
+
 Since 0.1.134, physical iOS capture runs in the shared `ios-mirror-service` Node process. Existing native packet-processing, input acknowledgement, connection, resource, and crash instrumentation stays attached to the active capture. Native resource measurements now overlap that service's Node process rather than each chat's MCP process. The service uses centralized server initialization, its packaged development/release environment, and server-owned anonymous installation/process-session identity. UI decode/render timings and sampled MCP operations retain their existing boundaries. Subscriber keyframe requests no longer invalidate the shared native queue or release another panel’s touch. The native packet-processing and recovery measurements remain on this active path; the rebuilt addon retains matching debug symbols.
 
 `ios.mirror.shared.captures` and `ios.mirror.shared.subscribers` report active counts. `ios.mirror.shared.queue_dropped` counts frames discarded by subscriber queue overflow. Counts aggregate in 30-second windows and flush on service shutdown, using the simulator surface and physical iOS context. No per-frame events, device identifiers, subscription IDs, payloads, or paths are sent. The service clears its collection timer on shutdown and honors telemetry opt-out.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "codex-mobile-dev-plugin",
-  "version": "0.1.134",
+  "version": "0.1.139",
   "private": true,
   "type": "module",
   "engines": {

src/server/baguette.ts

@@ -14,6 +14,19 @@ import { definitionDeviceState, getDefinitionDiagnostic, parseDefinitionDiagnost
 import { SimulatorUnavailableError } from "./simulator-unavailable.ts";
 import { ExpectedOperationError } from "../shared/error-reporting.ts";
 import { baguetteEnvironment } from "./baguette-runtime.ts";
+import { captureServerError, recordAccessibilityBridgeRestart } from "./telemetry.ts";
+
+type DescribedUi = { tree?: { label?: unknown } };
+const NO_ACCESSIBILITY_DATA = "no accessibility data";
+const BRIDGE_RESTART_INTERVAL = 10000;
+
+// CoreSimulatorBridge can lose its accessibility connection after backboardd and SpringBoard
+// restart, then report no data for every foreground app until it restarts.
+async function kickstartAccessibilityBridge(udid: string, signal: AbortSignal): Promise<void> {
+  await promisify(execFile)("/usr/bin/xcrun", ["simctl", "spawn", udid, "launchctl", "kickstart", "-k", "user/foreground/com.apple.CoreSimulator.bridge"], {
+    timeout: 10000, maxBuffer: 64 * 1024, encoding: "utf8", signal,
+  });
+}
 
 export const definitionSchema = z.object({
   identity: z.object({ udid: udidSchema, name: z.string(), model: z.string() }),
@@ -36,10 +49,13 @@ export class Baguette {
   private diagnostics = "";
   private disposed = false;
   private readonly embedded: boolean;
+  private readonly kickstartBridge: (udid: string, signal: AbortSignal) => Promise<void>;
+  private readonly bridgeRestarts = new Map<string, number>();
 
-  constructor(baseUrl?: string) {
+  constructor(baseUrl?: string, kickstartBridge = kickstartAccessibilityBridge) {
     this.embedded = baseUrl == null;
     this.baseUrl = parseBaseUrl(baseUrl ?? "http://127.0.0.1:0");
+    this.kickstartBridge = kickstartBridge;
   }
 
   async json(path: string, options: RequestInit = {}, timeout = 10000): Promise<unknown> {
@@ -63,6 +79,37 @@ export class Baguette {
     return payload;
   }
 
+  async describeUi(udid: string): Promise<DescribedUi> {
+    const path = `/simulators/${udidSchema.parse(udid)}/describe-ui.json`;
+    try { return await this.json(path) as DescribedUi; }
+    catch (error) {
+      if (!(error instanceof Error) || error.message !== NO_ACCESSIBILITY_DATA || !await this.restartAccessibilityBridge(udid)) throw error;
+    }
+    try {
+      const result = await this.json(path) as DescribedUi;
+      recordAccessibilityBridgeRestart("recovered");
+      return result;
+    } catch (error) {
+      recordAccessibilityBridgeRestart("unavailable");
+      throw error;
+    }
+  }
+
+  // A foreground app with no data usually means a stale bridge. Restart it at most every 10 s per device.
+  private async restartAccessibilityBridge(udid: string): Promise<boolean> {
+    if (Date.now() - (this.bridgeRestarts.get(udid) ?? -Infinity) < BRIDGE_RESTART_INTERVAL) return false;
+    this.bridgeRestarts.set(udid, Date.now());
+    try {
+      await this.kickstartBridge(udid, this.lifecycle.signal);
+      return true;
+    } catch (error) {
+      if (this.lifecycle.signal.aborted) throw error;
+      recordAccessibilityBridgeRestart("failed");
+      captureServerError(error, "simulator.accessibility_bridge");
+      return false;
+    }
+  }
+
   async status(signal?: AbortSignal): Promise<Status> {
     if (this.embedded && !this.child) {
       return { connected: false, managed: false, baseUrl: this.baseUrl.origin, devices: [], error: "The bundled simulator backend has not started." };