Skip to content

fix(ios): size the sole display from the system app, not the runner's UIScreen - #4157

Open
thymikee wants to merge 17 commits into
mainfrom
maint/4091-sole-screen-display-size
Open

thymikee wants to merge 17 commits into
mainfrom
maint/4091-sole-screen-display-size

Conversation

@thymikee

@thymikee thymikee commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes #4091. Synthesized landscape presses landed a fixed offset away from the requested point and still reported ok. soleDisplayFrame read UIScreen.main in the runner process. The generated AgentDeviceRunnerUITests-Runner.app has no launch screen, so that read can come back in compatibility mode (768x1024 on an iPad).

  • Display source: the AXRuntime system application frame, normalized to portrait. It is read once per command (~0.6–0.9 ms, measured), and only for turns that use the box (landscape and upside-down).
  • Unreadable display: portrait needs no box, so a tap is sent as given, even with no window. In landscape, an app the sole-screen rule (RunnerResolveSoleScreen) admits uses its window. With no window, or a placed iPad window (iOS: XCTest runner publishes window-relative frames and shifted taps for an iPad app that does not fill the screen (Windowed Apps / Stage Manager) #3899), the gesture returns UNSUPPORTED_OPERATION before dispatch. Each case logs AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=rotation-basis.
  • Unchanged: multi-screen hosts keep the window route. No new private API is added; iPhone synthesis now also uses AXRuntime.

Rejected: a launch-screen key for the runner, which would mean patching Xcode's runner template and could not be proven here.

Validation

Base a42bc35282, head 00d36437ba.

  • The compatibility-mode test fails on base and passes on head.
  • AX-failure tests: portrait taps are dispatched, with or without a window. In landscape, iPhone dispatches about the window, a placed iPad window refuses, and a missing window refuses. Each test fails when its branch is removed.
  • The per-command cache test passes.
  • Live run with the branch runner on iOS 27.0: iPad 3/3 per orientation, iPhone 17 3/3 per landscape side.
  • Last local receipt: check:affected --run --head bf0b1ca48d passed all runnable checks, exactCommit=true (run-jWH5SD). The head 00d36437ba changes only two Swift test comments, which the gate does not compile. Gate reruns on that head failed only in unrelated TS unit tests, hit by stray GET / traffic on the host. CI is green on 00d36437ba. Swift runner gates run in CI.

Unverified: the live compatibility-mode offset did not reproduce on these runtimes. Possible follow-up: a unit test that fakes a two-screen XCUIScreen.screens; the single-screen guard is enforced in RunnerResolveSoleScreen and synthesizedRotationFrame.

🤖 Generated with Claude Code

… UIScreen

The generated AgentDeviceRunnerUITests-Runner.app has no launch screen, so the
runner process can read UIScreen.main in compatibility mode (768x1024 on an
iPad, a letterbox on an iPhone). The synthesized rotation then turned landscape
touches about that box, and every press landed a fixed offset away while
reporting ok. Read the display from the AXRuntime system application frame,
the same screen the fills-screen proof already compares against.

Fixes #4091

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agent-device-website Ready Ready Preview Nov 2, 2026 4:14pm UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.66 MB 5.67 MB +5.1 kB
Package (unpacked) 5.66 MB 5.67 MB +5.1 kB
Package (download) 1.72 MB 1.73 MB +1.2 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 33.5 ms 32.9 ms -0.6 ms
CLI --help 69.8 ms 67.9 ms -1.9 ms

@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Apex Sentinel

Verdict: approve

Since the last review: 2 new · 3 still open · 2 no longer raised

5 finding(s): 2 minor, 3 nit. Verdict: approve.

🟡 Minor (2)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:585 Long-press refusal message renders "synthesized longpress"
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:24 Portrait drag and scroll now refuse on the sole-screen route, though a portrait turn needs no box
🔵 Nit (3)
Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:522 🆕 portraitUpsideDown and an unknown orientation get the window fallback when the display box cannot be read, though the shared helper refuses unknown
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:545 🆕 PointGestureResolution.refusal fabricates .performed on the wrong case
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:631 unmeasuredSoleDisplayOutcome takes a String where SynthesizedPointGesture exists

Reply under an inline comment with @apex-sentinel-staging note ... or @apex-sentinel-staging pivot ... to re-check that finding.
Comment @apex-sentinel-staging review on this conversation to run a full review.

Earlier reviews

Review of 5c470cbb · 02.11.2026 15:31 UTC

Apex Sentinel

Verdict: request_changes

Since the last review: 3 new · 2 still open · 6 no longer raised

5 finding(s): 1 major, 1 minor, 3 nit. Verdict: request_changes.

Required fixes

  • Sole-screen route now closes entirely when the AXRuntime display read fails, even for an app the admission rule proves full-screen

🟠 Major (1)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:509 Sole-screen route now closes entirely when the AXRuntime display read fails, even for an app the admission rule proves full-screen

🟡 Minor (1)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedInteractionTests.swift:286 🆕 Compatibility-mode test no longer exercises anything: production reads no UIScreen
🔵 Nit (3)
Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:539 🆕 Rotation basis comes from two different screen sources with no cross-check, contradicting the invariant the test documents
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:573 🆕 Gesture path has no log naming the unmeasured-display refusal
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:574 Gesture string literals passed to unmeasuredSoleDisplayOutcome are a hardcoded, drifting list
Review of 3e2b1777 · 02.11.2026 14:44 UTC

Apex Sentinel

Verdict: approve

Since the last review: 5 new · 3 still open · 1 no longer raised

8 finding(s): 3 minor, 5 nit. Verdict: approve.

🟡 Minor (3)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:507 🆕 Sole-screen route now dies in portrait when the AXRuntime display read fails, where the old UIScreen.main read always answered
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:536 🆕 Unreadable display now refuses portrait taps for apps that are not full-screen-proven
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:538 Portrait turns now silently rotate about a placed iPad window when the display size cannot be read
🔵 Nit (5)
Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerMainOwnedState.swift:12 🆕 Per-command reset lives in one call path while the cache is owned by shared state
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerXCTestEventBridge.m:348 ObjC comment documents the per-command display-size cache inside RunnerResolveSoleScreen, which performs no such read
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:559 Per-command display-size cache can serve a stale box when the orientation changes inside one command
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:571 🆕 A 115-character line and a reflowed guard style disagree with the file's own convention
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:576 🆕 Gesture name in the refusal message is a hand-written string, and one call site says "gesture"
Review of 4690ea7b · 02.11.2026 14:06 UTC

Apex Sentinel

Verdict: request_changes

Since the last review: 0 new · 4 still open · 6 no longer raised

4 finding(s): 1 major, 3 nit. Verdict: request_changes.

Required fixes

  • iPhone sole-screen route now dies when the AXRuntime display read fails; the old UIScreen read answered

🟠 Major (1)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:562 iPhone sole-screen route now dies when the AXRuntime display read fails; the old UIScreen read answered
🔵 Nit (3)
Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift:10 One AXRuntime system-application read per command on paths that never use its answer
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:508 soleScreenReferenceFrame(app:) lost the XCUIScreen.screens.count == 1 guard
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:537 Portrait synthesis silently switched its rotation basis from the display frame to the window fallback
Review of 409188ee · 02.11.2026 13:43 UTC

Apex Sentinel

Verdict: approve

Since the last review: 7 new · 3 still open · 2 no longer raised

10 finding(s): 4 minor, 6 nit. Verdict: approve.

🟡 Minor (4)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:509 Sole-screen route now dies in portrait when the AXRuntime size read fails, though portrait needs no display box
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:532 🆕 portraitUpsideDown refuses a synthesized gesture it does not need to
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedInteractionTests.swift:331 🆕 Per-command-cache assertion also passes when the size is merely not re-read
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:546 🆕 Windowed-iPad landscape gestures now refuse outright when the AXRuntime size read fails; the UIScreen.main source that served them is gone
🔵 Nit (6)
Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+CommandExecution.swift:13 Per-command display-size cache is cleared in only one of the runner's two command entry points
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:506 🆕 The single-screen condition now lives in one of the two sole-screen entry points
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:507 soleScreenReferenceFrame now needs the AXRuntime read twice per context, and closing admission is logged as resolved=none
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:521 🆕 appFillsSoleScreen discards the screen and display id the predecessor used
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:521 🆕 appFillsSoleScreen is the admission rule, not the fills-screen proof
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:537 🆕 New unguarded XCUIScreen.screens.count read on every landscape gesture, where the bridge wraps the same read in an exception catcher
Review of 7ea0bb30 · 02.11.2026 12:05 UTC

Apex Sentinel

Verdict: request_changes

Since the last review: 3 new · 2 still open · 4 no longer raised

5 finding(s): 1 major, 2 minor, 2 nit. Verdict: request_changes.

Required fixes

  • A single-screen tap now refuses when the display is unmeasurable, even in an orientation that never rotates

🟠 Major (1)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:528 A single-screen tap now refuses when the display is unmeasurable, even in an orientation that never rotates

🟡 Minor (2)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerMainOwnedState.swift:12 🆕 Cached display size is never invalidated and contradicts the sibling probe's stated resize invariant
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerXCTestEventBridge.m:348 🆕 Comment claims the display-size read is paid once, but it runs on the admission path before the read is cached
🔵 Nit (2)
Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:203 Gesture name in the refusal is a loose string, and 'long press' disagrees with the command family's spelling
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerXCTestEventBridge.m:254 🆕 RunnerSystemApplicationScreenSize is defined in .m without the TARGET_OS_IOS gate its header declaration has
Review of b81d3cc4 · 02.11.2026 11:35 UTC

Apex Sentinel

Verdict: request_changes

Since the last review: 4 new · 2 still open · 5 no longer raised

6 finding(s): 1 major, 5 nit. Verdict: request_changes.

Required fixes

  • Dropping the window fallback turns AX-read failures into refusals on single-screen devices (previously-working iPhone taps)

🟠 Major (1)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:528 🆕 Dropping the window fallback turns AX-read failures into refusals on single-screen devices (previously-working iPhone taps)
🔵 Nit (5)
Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedInteractionTests.swift:44 🆕 Refusal test depends on a side-effect pre-read having loaded AXRuntime before it swaps it
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:512 Every synthesized gesture now pays a redundant cross-process AXRuntime read on the hot path
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:542 New refusal paths bypass the AGENT_DEVICE_RUNNER_SYNTHESIZED_SOLE_SCREEN_ROUTE log the hint points at
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:543 🆕 Refusal message and hint restate the same cause
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:543 🆕 Gesture kind is a free-form string at the four new refusal call sites
Review of 0fcd7270 · time not recorded

Apex Sentinel

Verdict: approve

Since the last review: 6 new · 1 still open · 8 no longer raised

7 finding(s): 3 minor, 4 nit. Verdict: approve. reached max rounds (1).

🟡 Minor (3)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SynthesizedInteractionTests.swift:234 🆕 testSynthesizedRotationFrameIsTheDisplayNotAPlacedWindow launches the app without invalidating the cached target
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:525 🆕 An unknown orientation still falls back to the placed window, which the new comment says never happens
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:527 🆕 iPhone sole-screen synthesis now hard-depends on the AXRuntime system-app read, which the idiom route deliberately avoided
🔵 Nit (4)
Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:541 🆕 Hint points at a log line the failed read does not always emit
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:534 🆕 unmeasuredSoleDisplayOutcome is the only member of this cluster without an access modifier
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:550 🆕 The log tag is hand-copied into the user-facing hint, so a rename silently misleads users
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerXCTestEventBridge.m:254 Two new AXRuntime probes per gesture are undocumented and unmeasured

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 4 new findings on eb706bd. The verdict and the full list are in the summary comment.

Log AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=system-app when the
AXRuntime read fails, rename the reference-frame size to portraitSize (it
is points), resolve every UIScreen getter before swapping any, and skip the
display test unless the host app fills the screen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 5 new findings on f1e452f. The verdict and the full list are in the summary comment.

thymikee and others added 2 commits November 2, 2026 09:34
…l tolerance

The gesture-viewport test now expects the host app's frame instead of the
runner's UIScreen, and both display tests compare within the 0.5 pt the
fills-screen proof allows. The sole-screen frame doc names its unit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e measured

A sole display whose size AXRuntime would not report fell back to the
resolved window's box, which on a placed iPad window is the #3899 basis.
synthesizedRotationFrame now answers nil only for that case, and tap, long
press, drag and planned gestures refuse with UNSUPPORTED_OPERATION before
they log or dispatch. A multi-screen host keeps the window route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 6 new findings on 0fcd727. The verdict and the full list are in the summary comment.

…ey-free

The system-app size read now logs an unavailable AXRuntime class too, the
refusal hint points at the runner log without quoting its key, the rotation
frame doc says an unknown orientation keeps the window box (no turn reads
it), and the display test clears its cached target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

This needs physical-device evidence before merge. At c3deb5d the size fix looks right for the iPad Simulator case in #4091, but it adds a private AX read to every synthesized gesture on every single-screen iOS device, phones included.

synthesizedRotationFrame calls RunnerSystemApplicationScreenSize on every synthesized tap, long press, drag and planned gesture (RunnerTests+SynthesizedInteraction.swift:530), and soleScreenReferenceFrame(app:) (:513) makes the same read on the windowless route. runnerSynthesizesTap sends synthesized:true to physical iPhones and iPads (device.ts:275). Before this PR the phone idiom never made this read. Every run in the PR is on a Simulator. If a physical device answers CGSizeZero, scroll, sampled planned gestures, alert accept/dismiss and the windowless route all refuse, and coordinate taps drop to the slower XCTest fallback, in portrait too. Please record one run on a physical iPhone with the branch runner (pnpm build:xcuitest): scroll down, then press <x> <y> in landscape-left and landscape-right. The runner log must have no AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=system-app line. Each AGENT_DEVICE_RUNNER_SYNTHESIZED_DISPATCH point= must equal the native point from the portrait size: (y, W-x) for landscape-left and (H-y, x) for landscape-right, with W and H the long and short sides (874 and 402 on a 402x874 iPhone). Use point=, not reference=, because reference= logs the window and not the rotation box. Also attach the iPad (A16, 820x1180) Simulator landscape DISPATCH lines the body describes: press 120,300 in landscape-left must log point=(300.0,1060.0), and press 150,300 in landscape-right must log point=(520.0,150.0). A 768x1024 basis would log (300,904) and (468,150). The run also needs the pending check:affected --run result.

Not blocking, and you can take or leave these: CoordinateSpaceRotation.native reads the frame only for landscape-right, landscape-left and portrait-upside-down, so portrait and unknown orientations refuse or fall back on a size they never use, and the new doc sentence about unknown orientation is true only when the read succeeds; RunnerSystemApplicationScreenSize repeats the systemApplication read in RunnerProbeApplicationFillsScreen, and the windowless route now makes three separate reads per gesture that the body says "share one source"; the body and doc say the gesture "refuses before it dispatches", but the call-site policy in RunnerTests+SynthesizedGesturePolicy.swift turns that into an XCTest fallback for coordinate taps, and alert activation should be listed among the callers that refuse; and testTapRefusesBeforeDispatchWhenTheSoleDisplayCannotBeMeasured skips the policy layer, the XCTAssertNotEqual(screen, 768x1024) at :251 fails on a real 768x1024 iPad Simulator (an XCTSkip fits better), and no test asserts the dispatched point for a landscape tap.

Could this be simpler? One static RunnerReadSystemApplicationFrame(CGRect *) in RunnerXCTestEventBridge.m, logging stage=system-app on every failed branch, could serve both the probe and RunnerSystemApplicationScreenSize. A CoordinateSpaceRotation.usesFrame(interfaceOrientation:) next to native() could let synthesizedRotationFrame return fallback before any read when the orientation does not use the frame. On the windowless route, context.referenceFrame is already the display frame, so it could replace the second read. That gives one reader, a read only where rotation uses it, and a physical-device ask limited to landscape.

Of the earlier inline threads, the one on the AX read for every gesture still applies (#4157 (comment)); its UIScreen fallback is rightly rejected, and the points above answer it. These no longer apply and are already resolved: the test cleanup now calls invalidateCachedTarget in the defer (#4157 (comment)), the doc was narrowed (#4157 (comment)), the class-Nil branch now logs stage=system-app (#4157 (comment)), unmeasuredSoleDisplayOutcome needs internal access across files (#4157 (comment)), and the hint no longer quotes the log key (#4157 (comment)).

CI is still pending: Smoke Tests is queued, not failed, and it builds this runner and drives synthesized press and scroll, so a failure there would likely relate to this change. I did not run the runner or the new unit tests, so the read cost, whether the AX read answers on physical iPhones, and the landed points are unverified. The PR body's validation section still names 0fcd727.

…read

Admission on an iPhone still skips the fills-screen probe, but every device
now pays one AXRuntime read (~0.6-0.9 ms) when the caller sizes the display.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 3 new findings on b81d3cc. The verdict and the full list are in the summary comment.

A device's display size does not change while the runner runs, so the
first successful AXRuntime read (~0.6-0.9 ms) is kept on the main-owned
state and later gestures reuse it; a failed read is not kept, so the
refusal stays reachable and the next gesture reads again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 new findings on 7ea0bb3. The verdict and the full list are in the summary comment.

thymikee and others added 2 commits November 2, 2026 13:22
…s life

Display scaling can change the system application frame while the runner
runs, so a size kept for the whole runner could turn later gestures about a
stale box. Each command clears it and reads again; a command's gestures
still share one successful read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… is unreadable

A failed AXRuntime display read refused every synthesized gesture on a
single-screen device, including iPhone taps that worked through the window
before. When the sole-screen rule (RunnerResolveSoleScreen) proves the app
fills the screen, its window is the display and stays the rotation basis;
otherwise the gesture still refuses with UNSUPPORTED_OPERATION. Both routes
log AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=rotation-basis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .github/workflows/xctest-nightly.yml Resolved
thymikee and others added 2 commits November 2, 2026 14:21
Portrait and an unknown orientation never read the rotation box, so an
unreadable display no longer refuses their gestures and they skip the
AXRuntime read entirely. The AX-failure tap tests now run in landscape,
where the box is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The display-box test is stub-based and runs on the iPhone full suite, so
the iPad leg keeps the refusal test, which only an iPad can exercise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

… setup

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 3 new findings on 409188e. The verdict and the full list are in the summary comment.

… read the box

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Replies to the four findings Apex Sentinel still lists at 4690ea7ba3 (none has an open inline thread):

  • Major, the iPhone sole-screen route fails when the AXRuntime display read fails. Kept by design, as agreed for the Cubic P2 in the first round. The window-less route runs only when XCTest cannot resolve a window, a tree it cannot serialize. If AXRuntime also fails there, no source of the display size is left that is not the runner's own UIScreen, and iOS Simulator (iPad): press lands a fixed offset away in landscape (0,+52 left; +156,0 right) and reports success #4091 shows that one is wrong on an iPhone too. The command refuses with UNSUPPORTED_OPERATION and logs stage=system-app. Whenever a window resolves, an admitted app (every iPhone) still dispatches about its window (a42f844).
  • Nit, one AX read per command on paths that never use it. No read happens there. executeOnMainPrepared only clears the kept size. The read is lazy, made the first time a landscape or upside-down turn needs the box.
  • Nit, soleScreenReferenceFrame(app:) lost the screens.count == 1 guard. RunnerResolveSoleScreen, which that function calls first, refuses screenCount != 1, so the condition is still enforced on that path.
  • Nit, portrait silently switched its basis to the window. Nothing reads it: CoordinateSpaceRotation.native returns the point unchanged for portrait, so the basis never enters the dispatched point. Skipping the read there is what keeps portrait gestures from refusing when the display is unreadable.

…eadable

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 new findings on 3e2b177. The verdict and the full list are in the summary comment.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Thanks for the update. At 4690ea7 the code mostly addresses the earlier review: portrait and unknown orientations make no read, a windowed iPhone gesture falls back to its window when the read fails, and the windowless iPhone route now refuses when the read fails. One thing is still unproven, and it needs a physical device.

Every landscape synthesized gesture on a physical iPhone or iPad now takes its rotation basis from the AXRuntime system-app frame instead of the base's UIScreen.main (:547, :562-568). A read that comes back wrong but non-zero would land presses off target and still report ok. A Simulator cannot rule that out. Physical and remote iPhones are the main target, and the PR body gives only a pass tally from Simulators. The required runs from the earlier review are still not posted. Please run the branch runner (pnpm build:xcuitest) on one physical iPhone and post the runner log for press <x> <y> in landscape-left and landscape-right, plus scroll down in landscape. The log must have no AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=system-app or stage=rotation-basis line. Each AGENT_DEVICE_RUNNER_SYNTHESIZED_DISPATCH point= must equal (y, W-x) for landscape-left and (H-y, x) for landscape-right, where W and H are the long and short portrait sides of the device. Please also post the iPad (A16, 820x1180) Simulator dispatch lines: press 120,300 in landscape-left should log point=(300.0,1060.0), and press 150,300 in landscape-right should log point=(520.0,150.0).

Not blocking, and fine to take or leave: the orientation switch in synthesizedRotationFrame repeats which cases of CoordinateSpaceRotation.native read the frame, in a second module, so a new case there would silently get the window fallback. Could CoordinateSpaceRotation.readsFrame(interfaceOrientation:) live beside native() and be called from here?

No open P1 or P2 thread still applies, and no lower-priority thread applies either. The cubic P3 and P2 threads on the bridge comment and the nightly job (r4442029047, r4442815286) are fixed at this head, as is the rename in r4443101235. These are already resolved, as addressed or by design: the XCTUnwrap is kept on purpose (r4441884109); the hint wording is fine (r4441884132); nothing parses the gesture strings (r4441884138); there is no second entry into executeOnMainPrepared (r4442162292); the bridge definition is already inside the TARGET_OS_IOS block (r4442162327); the test-comment wording is fine (r4442922131); the portrait dependency is now documented, and the duplicate case set is the note above (r4443099065); the read is lazy, so non-landscape and multi-screen gestures return before it (r4443099755); RunnerResolveSoleScreen already refuses screenCount != 1 (r4443100160); the windowless refusal is by design (r4443100626); the test fails without the clear (r4443101215); this path needs only the admission answer (r4443101229).

The only check that is not passing is a review bot check (ACTION_REQUIRED). It reflects review findings and runs no build or test, so it looks unrelated to the code. The other 18 checks pass, including Smoke Tests, which builds this runner and runs the changed unit tests. I ran no runner, unit test or device. The mutation results, the read cost and the check:affected receipt are the author's reports, which I could not inspect. Before merge, please post the physical iPhone landscape-left and landscape-right dispatch lines (with no stage=system-app or stage=rotation-basis line) and the two iPad Simulator lines above.

This review covers 4690ea7. The newer head 3e2b177 only adds a test pinning that a portrait tap dispatches when the display is unreadable, so the device runs above still apply.

…press as the CLI does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

I re-read 5c470cb. The new commits add a portrait-tap test and doc wording, and the code findings from the earlier reviews are addressed. The one open item is unchanged: the physical iPhone landscape run and the two iPad Simulator dispatch lines from my 4690ea7 review (#4157 (comment)). Every landscape synthesized gesture on a physical device now takes its rotation basis from the system-app frame, and a Simulator cannot show that the read is right there. Smoke Tests is still running at this head.

Not blocking, take or leave: testPortraitTapDispatchesWhenTheSoleDisplayCannotBeMeasured (test) stays green if the portrait early return is removed, so it could count system-app reads and assert zero, or be dropped since the refusal test already pins portrait. The long-press refusal at :207 says "synthesized longpress" while :189 and :195 say "synthesized coordinate long press".

…s unreadable

With the window query rejected and the AXRuntime display read failing there
is no box at all. A portrait tap or long press keeps its point, so on an
admitted sole screen it is now dispatched as given; any other turn still
refuses with UNSUPPORTED_OPERATION. Both log stage=rotation-basis window=none.
Tap and long press share one point-gesture resolver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift">

<violation number="1" location="apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift:592">
P3: `referenceFrame: .null` is a sentinel that the dispatch log prints as `reference=(inf,inf,0.0,0.0)` on the new window-less path. Log an explicit marker for the no-window case, or pass a meaningful frame, so the diagnostic stays readable.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on 00d3643. The verdict and the full list are in the summary comment.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

The delta to 00d3643 does not yet make this ready: one earlier finding is unchanged, and there is still no device evidence. CI is pending (Smoke Tests is still running at 00d3643), so there is no failure to attribute, and there are no conflicts.

The earlier review (#4157 (comment)) asked for device runs. They are still missing. Every landscape and upside-down synthesized gesture on a single-screen device takes its rotation basis from the AXRuntime system-app frame (https://github.com/callstack/agent-device/blob/00d3643/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedInteraction.swift#L627). A wrong but non-zero size passes the >0 guard and rotates every press about the wrong box while the command reports ok. Physical and remote iPhones are the main target, and a Simulator cannot show that this read is right on hardware. Please post a branch-runner (pnpm build:xcuitest) log from one physical iPhone for press <x> <y> in landscape-left and landscape-right and for scroll down in landscape. The log should show no AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED stage=system-app or stage=rotation-basis line. Each AGENT_DEVICE_RUNNER_SYNTHESIZED_DISPATCH point= should equal (y, W-x) for landscape-left and (H-y, x) for landscape-right, where W and H are the long and short portrait sides. Please also post the iPad (A16, 820x1180) Simulator lines: press 120,300 in landscape-left should give point=(300.0,1060.0), and press 150,300 in landscape-right should give point=(520.0,150.0). I did not run the Swift unit tests, and I could not confirm that the new window-less portrait test reaches the nil-context path on an iPhone in CI.

Is the window-less portrait branch needed at all? The smallest change is to drop it (PointGestureResolution, SynthesizedPointGesture, the second arm of pointGestureTarget, unrotatedSoleScreenPoint) and keep the uniform refusal from 5c470cb. That refusal was accepted as by design, and the state this branch serves (iPhone, rejected window query, failing AXRuntime read, portrait) has not been observed. If you keep the dispatch, please make it one decision for every gesture: a single readsFrame(interfaceOrientation:) predicate beside CoordinateSpaceRotation.native, used by synthesizedRotationFrame and by the sole-screen branch of synthesizedCoordinateContext. Then tap, long press, drag, scroll and sampled gestures share it, and the per-gesture wrapper enum goes away. Today only tap and long press get it, while drag (:24), the sampled path (:814) and the scroll route (CommandExecution.swift:282) still refuse with "could not resolve an app window" in the same state. Nothing else needs to change first, and no ADR is needed.

Not blocking, and you can take or leave these: unrotatedSoleScreenPoint (:614) tests interfaceOrientation == portrait, while synthesizedRotationFrame (:524) uses a switch where portrait and unknown both skip the box, so one predicate should own that rule; and PointGestureResolution.refusal (:557) returns .performed for a .target, and unresolvedWindowName only keeps two refusal texts apart, so switching on the resolution at the two call sites and deleting .refusal would remove that hidden contract.

On the open inline threads: the one on #4157 (comment) (refusal on .target) still applies as a non-blocking nit, and the one on #4157 (comment) (referenceFrame: .null) no longer applies, because the author declined it and the stage=rotation-basis line already names the route, and it is already resolved.

Before merge, the physical iPhone landscape log and the two iPad Simulator dispatch lines need to be posted, and the window-less portrait branch needs a decision.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Final replies to Apex Sentinel's summary at 00d36437ba (verdict approve). Every item is settled design; none is a behavior bug, so there is no push.

The rule when the AXRuntime display read fails on a single-screen device:

Case Result Pinned by
Portrait Point passes through unchanged; no box is read or used testPortraitTapDispatchesWhenTheSoleDisplayCannotBeMeasured (placed window, iPad and iPhone), testWindowlessPortraitTapDispatchesWhenTheSoleDisplayCannotBeMeasured
Landscape or upside-down, app admitted by RunnerResolveSoleScreen The window is the box testTapTurnsAboutTheWindowWhenTheSoleDisplayCannotBeMeasuredOnAFullScreenApp
Landscape or upside-down, not admitted (placed or windowed iPad), or no window UNSUPPORTED_OPERATION before dispatch testTapRefusesBeforeDispatchWhenTheSoleDisplayCannotBeMeasured, testWindowlessLandscapeTapRefusesWhenTheSoleDisplayCannotBeMeasured

The old UIScreen.main source is the one #4091's reports show to be wrong in compatibility mode (768x1024 on an iPad), so a typed refusal replaces a silently wrong tap. testSynthesizedRotationFrameIsTheDisplayNotAPlacedWindow stubs UIScreen to those values: it fails on base and passes on head, which is what proves the source switch.

Replies to the individual items:

  • "synthesized longpress" in the long-press refusal: this is the CLI's command spelling (agent-device longpress), adopted on request in 5c470cb. The host keys on UNSUPPORTED_OPERATION, not on this text.
  • Portrait drag and scroll refuse on the window-less route: only when there is no window and the display is unreadable. Drag and scroll plan their path in the reference viewport (RunnerScrollViewportPolicy, keyboard avoidance). Without a window or a measured display there is no viewport to plan in, so they refuse rather than plan inside a guessed box. With any resolved window, portrait drag and scroll proceed, because the rotation box is never read.
  • Upside-down and unknown take the window fallback while the window-less helper refuses unknown: upside-down reads the box (native mirrors with width and height). When an admitted route uses the window, that window is the display, so the fallback is correct. Unknown keeps the axes, so the fallback is never read. The window-less helper admits only portrait, conservatively. An unknown orientation there keeps main's behavior: the sole-screen frame was nil for it before this PR too.
  • refusal returns .performed on the wrong case, and unmeasuredSoleDisplayOutcome(gesture:) takes a String: answered in their threads. The .performed branch is unreachable (it is read only in the guard case .target else), and an optional would collide with tap's nil = deadline-expired. The string is message wording only, never parsed.
  • Cache-reset location: executeOnMainPrepared is the single command boundary every synthesizing command passes through. testSoleDisplaySizeIsReadOncePerCommandAndAFailedReadIsNotKept runs a real command through it and fails if the clear is removed.
  • ObjC comment placement: the comment above the admission check says admission and the display-size read are separate. The read happens after admission, once per command, when a turn needs the box.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Thanks, that settles the design question from my 00d3643 review. Drag and scroll plan their path in a viewport, so refusing them when there is neither a window nor a measured display is the right call, and the table makes the rule clear. The window-less portrait branch can stay.

The one open item is the device evidence: the physical iPhone landscape press and scroll down log (no stage=system-app or stage=rotation-basis line, and SYNTHESIZED_DISPATCH point= equal to (y, W-x) and (H-y, x)), plus the two iPad (A16) Simulator lines, point=(300.0,1060.0) and point=(520.0,150.0).

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Device evidence for the 16:45 / 17:31 review, branch runner built from head 00d36437ba (production code identical to bf0b1ca48d).

iPad (A16) Simulator, 820x1180, iOS 27.0. Both requested lines appear. Note that the side names are mirrored: the agent-device CLI's orientation landscape-left sets UIDeviceOrientation.landscapeLeft, which is UIKit interface orientation 3 (landscapeRight), and vice versa.

CLI landscape-left  (interface 3), press 150 300:
AGENT_DEVICE_RUNNER_SYNTHESIZED_DISPATCH kind=tap point=(520.0,150.0) reference=(0.0,0.0,1180.0,820.0) orientation=3
CLI landscape-right (interface 4), press 120 300:
AGENT_DEVICE_RUNNER_SYNTHESIZED_DISPATCH kind=tap point=(300.0,1060.0) reference=(0.0,0.0,1180.0,820.0) orientation=4

The other two presses from the same run: press 120 300 on CLI landscape-left → point=(520.0,120.0) orientation=3, and press 150 300 on CLI landscape-right → point=(300.0,1030.0) orientation=4.

  • Interface 3 gives (H−y, x) and interface 4 gives (y, W−x), with W=1180 and H=820.
  • The run logged no AGENT_DEVICE_RUNNER_APP_SCREEN_UNRESOLVED line (neither stage=system-app nor stage=rotation-basis).
  • In the earlier live runs, these mappings selected the intended Settings sidebar row 9/9 across portrait and both landscape sides.

Physical iPhone (thymikee-iphone, iPhone 17 Pro): blocked, not run. The branch runner built and signed for the device, but xcodebuild test-without-building stopped at run-destination preflight with:

Error Domain=com.apple.dt.deviceprep Code=-3 "Unlock thymikee-iphone to Continue"
Xcode cannot launch AgentDeviceRunnerUITests on thymikee-iphone because the device is locked.

devicectl device info lockState reported passcodeRequired: true. No press or scroll reached the phone. The physical press (both landscape sides) and scroll down log needs the phone unlocked, ideally with Auto-Lock set to Never for the run.

For reference only, not a hardware substitute: the iPhone 17 Simulator (874x402) runs logged landscape dispatches with reference=(0.0,0.0,874.0,402.0), for example point=(243.0,437.0) orientation=3, and no SCREEN_UNRESOLVED line.

@thymikee

thymikee commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Thanks. The iPad (A16) Simulator lines match exactly: point=(520.0,150.0) and point=(300.0,1060.0) against a 1180x820 reference, with no SCREEN_UNRESOLVED line, and the mirrored side naming is clear. That settles the Simulator half.

The only open item is the physical iPhone run, which did not reach the phone because it was locked. Once it is unlocked (Auto-Lock off for the run), the landscape press on both sides and scroll down log with no stage=system-app or stage=rotation-basis line is all that remains.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

iOS Simulator (iPad): press lands a fixed offset away in landscape (0,+52 left; +156,0 right) and reports success

1 participant