Repository navigation
Conversation
Integrate Xcode's native MCP interaction sessions and simctl with the shared device driver, standalone CLI, and embedded SDK.\n\nPreserve Android defaults, native accessibility hit points, and cancellation-safe device leases. Add read-only setup checks, usage documentation, and hermetic regression coverage.
Capture via xcrun simctl io recordVideo anchored to the first-frame marker, seal segments on demand for live analyzer clips, roll on rotation/duration/crash, and finalize VFR .mov segments to CFR MP4 with a version-2 manifest so iOS recordings flow through the same DataEngine, extraction, and replay pipeline as Android scrcpy. Video tools now auto-detect per platform (simctl+ffmpeg on iOS) and the CLI/SDK no longer disable recording for iOS tasks. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
simctl io screenshot does not stream to stdout on Xcode 27, so the dimension probe always failed and segments finalized on the 1080x1920 fallback canvas. Parse the framebuffer IOSurface port size instead - it is the exact surface recordVideo encodes, excludes external scene displays, and swaps on rotation. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Extend the native Xcode 27 iOS support to every Artemis surface so iOS reaches feature parity with Android where the platform permits: - Shared simctl discovery (artemis/drivers/ios/discovery.py) and an IosDevicePool producing platform-tagged DeviceStatus entries, with explicit-UDID validation, Booted/Shutdown admission, and ambiguous "booted" rejection. - iOS locks scoped under "ios" (ios__<UDID>) so a simulator UDID can never collide with an Android serial; queue items carry platform, ios_workspace, and the pinned UDID, and workers spawn with --platform ios --ios-workspace without leaking ADB_DEVICE_SERIAL. - Web API accepts platform="ios", validates UDIDs, skips the Android screen-lock probe, rejects locked_app_package, and lists simulators alongside Android devices in /api/devices. - Live stream follows the active iOS lock owner and captures frames via simctl io screenshot; /api/stream/device-state reports platform. - Replay preserves mobile_platform from session device_info, rebuilds iOS contexts with DevicePlatform.IOS, honors UDID overrides, and lists simulators as replay targets. - MCP mobile_run_task accepts a platform argument (daemon + standalone paths), mobile_get_device_state observes iOS through UnifiedMobileController + XcodeSimulatorDriver, and mobile_diagnose adjusts verdicts/probes for iOS targets. - An ios_simulators readiness probe reports Xcode/simulator/MCP availability without becoming a blocker for Android-only hosts. - artemis run and artemis batch forward platform, device_serial, and ios_workspace through the daemon. Android remains the default platform and its behavior is unchanged. Android-only operations (shell, logcat, UIAutomator, AVD launch, accessibility helper) fail explicitly for iOS. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The run command still carried a pre-queue guard that rejected iOS workers and forced every iOS invocation into standalone mode, so daemon-submitted iOS tasks died at worker startup with "The daemon and device queue support Android only". Drop the stale checks; iOS now daemon-routes like Android (--standalone still forces local), and the worker path reaches execute_task with platform/ios_workspace intact. Verified live: /api/run platform=ios now spawns the iOS worker and reaches agent init, stopping only at the missing GOOGLE_API_KEY. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The console frontend was Android-only even though the backend already listed iOS simulators and accepted platform=ios runs: - connectedDevices merges ios_simulators probe metadata (udid/name/ state/runtime) into the device list, tagged platform=ios - Device chips render phone_iphone + an iOS badge; the hero card shows the selected simulator with its runtime - /api/system/devices/select accepts platform; iOS validates the UDID via the iOS device pool instead of retargeting the ADB probe - iOS selection is tracked client-side and satisfies the run button device gate; runTask sends platform=ios + device_serial when a simulator is selected - Sessions/queue items carry platform for an iOS tag in the session list Android selection/readiness/submission paths are unchanged. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The connected-device panel only rendered when the ADB probe passed, so on hosts without an Android device the iOS simulators could never be selected — and selecting one was what made the panel appear: - Show an "iOS Simulators" chip panel inside the no-device guide state whenever the ios_simulators probe reports simulators, independent of ADB state - Selecting a sim flips the device step to ready and shows the hero card - Retitle step 3 "Device & Emulator Connection" (Android via ADB or iOS Simulator via Xcode 27+) Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Verified the live console with headless Chrome: the diagnostics tab now
lists all simulators and clicking one selects it via
POST /api/system/devices/select {serial, platform:"ios"}. But the
launcher composer gave no hint which device a task would run on:
- Add a target-device-chip to the composer action bar showing the
selected device with its platform tag; clicking it jumps to the
setup guide to change the target
- Make the prerequisites warning platform-neutral ("device" not
"Android device")
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The device guide card mixed Android-only controls (Restart ADB, connection methods, AVD management) with the iOS simulator picker, and the merged chip list made the run target ambiguous. - Add platform tabs at the top of the card; auto-select the tab that has devices (or the platform of the currently selected simulator) - Scope Restart ADB, Change Connection, Android hero/switcher, connection methods, AVD list, and locked/unauthorized states to the Android tab - Give iOS its own panel: selected-simulator hero, simulator chip picker, and an empty state explaining the Xcode 27+ requirement - Selecting a chip also switches the active tab so the chosen target stays visible; ready banner text is platform-neutral Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Each click on a simulator POSTed to /devices/select, which re-ran "xcrun simctl list devices" inline to validate the UDID — 4+ seconds on a busy host while the readiness probe spawned back-to-back simctl calls of its own. The chip only marked itself selected after the response, so clicks appeared to do nothing. - Cache successful simctl enumerations for 10s in drivers/ios/ discovery.py (shared by the readiness probe, device pool, and HTTP endpoints); failures are never cached, force_refresh bypasses, and the driver clears the cache after booting a simulator - Apply iOS selection optimistically in the console since it is client-side state; revert only when the server rejects the pick Select endpoint: ~4.3s -> ~0.1s. Chip shows Active instantly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
TS6 requires rootDir to be explicit whenever outDir is set; the compiler was inferring ./src anyway, so this only silences the diagnostic without changing emit layout. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Runtime now renders as "iOS 27.0" instead of "iOS 27 0" - Hero card reports the real simulator state: green "Booted" for running sims, amber "Shutdown — boots on run" otherwise (previously always claimed "Connected"), with a matching status dot instead of the hardcoded green pulse - Drop the redundant per-chip "iOS" badge (the tab and panel already scope the list) and the cramped "(Shutdown)" suffix; state is now a colored dot — green Booted, grey Shutdown, amber Booting — with a tooltip for the raw simctl state - iPad simulators use the tablet_mac icon in chips and the hero card Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The feature section and roadmap entry predated the web/daemon/MCP integration commits and still described iOS as standalone-only; only physical devices remain planned. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Extend the iOS driver family to paired physical devices. Device routing keys off the requested UDID: serials resolving to a CoreDevice physical entry get PhysicalIosDriver, while simulators and unknown serials keep the existing XcodeSimulatorDriver path. - discovery.py: devicectl list devices JSON parsing (current and deprecated property shapes) with the same fail-closed caching contract as simctl - PhysicalIosDriver subclasses the simulator driver to inherit the Xcode DeviceInteraction session (screenshots, hierarchy, taps, swipes, text), overriding only lifecycle: devicectl install/launch/terminate/openURL/apps and pairing/connectivity checks with actionable errors - connect() is decomposed into seams (_require_ios_host, _prepare_device, _start_interaction_session, _validate_session_device) so the physical driver asserts deviceIsSimulator=false without duplicating session setup - PhysicalIosRecorder polls devicectl screenshots into timestamped frames and assembles MP4 segments with the ffconcat demuxer; rotation and seal boundaries roll segments, matching the simulator manifest contract - IosDevicePool enumerates physical devices for the console/queue, validates explicit serials (paired + connected), and never auto-selects hardware - CLI help, ios.md, and both READMEs updated for physical requirements and recording limitations Physical automation requires Xcode 27+, a paired/trusted device with Developer Mode enabled, and a device-signed .app/.ipa for installs. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Agent.init picks the driver via ios_driver_class so embedded-SDK iOS configs resolve physical UDIDs to PhysicalIosDriver - device stream service captures physical frames via devicectl screenshot (simctl io screenshot does not exist on hardware) - replay device picker lists paired physical devices for iOS retargeting - smoke-test repair hints cover pairing/trust/Developer Mode and offline physical devices Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Xcode's DeviceInteraction* MCP tools accept simulators only — verified live against a paired iPhone (the eligible-device list contains simulators exclusively), so physical observation and input move to WebDriverAgent: - wda.py: stdlib HTTP client (no new dependency) covering /status, /session, /source?format=json, /screenshot, /window/size, W3C pointer actions for tap/long-press/swipe, /wda/keys for text, /wda/homescreen and /wda/pressButton for keys, /wda/activeAppInfo for the foreground bundle. - Endpoint resolution probes ARTEMIS_IOS_WDA_URL, ARTEMIS_IOS_WDA_HOST, the CoreDevice tunnel address from devicectl info details, and 127.0.0.1:8100 for iproxy/pymobiledevice3 forwards. - connect() locates an installed *WebDriverAgent* runner, launches it via devicectl, and optionally hosts a build-for-testing .xctestrun through xcodebuild test-without-building (ARTEMIS_IOS_WDA_XCTESTRUN); disconnect tears both down. - Hierarchy flattens the WDA JSON tree into the same ui_elements shape (text/resource_id/class/parsed_bounds/hit_point) so controllers, element lookup, and tap_element are unchanged. - devicectl JSON for info subcommands now goes through a scratch file — '--json-output -' stdout is polluted by the human-readable table on hardware — and terminate resolves PIDs by matching the app's install URL prefix against running executables. - docs/ios.md and READMEs describe the WDA prerequisite, signing options, endpoint overrides, and the UI-Automation passcode gate. 47 physical unit tests pass; the simulator driver and Android paths are untouched. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ing hardening - ios_device_pool: never auto-select physical hardware; validators match by UDID or name and fail open when either enumeration is indeterminate - sdk/agent: pick the iOS driver class in a worker thread so simctl/ devicectl subprocesses never block the event loop during init - physical_driver: tunnel-IP reads every observed devicectl JSON shape; exclusive runner/xctestrun hosting; broad connect cleanup; --kill terminate with stale-PID fallback; normalize file:// executables; IPA bundle id read from Payload Info.plist; scale sanity guard - wda: map http.client.HTTPException, propagate transport errors from press_button, IPv6-safe URL normalization, bracket only v6 candidates - physical_recording: monotonic frame indices across segment rolls, shielded stop, broad poll-loop failure accounting, watchdog notices a dead poller, ffmpeg timeout, atomic .part cleanup - discovery: null-safe property reads, iPadOS classification - diagnose/probe: physical serials reachable via mobile_diagnose; pool auto-pick stays simulator-only; platform-aware smoke labels - Sweep stale 'simulator' wording across MCP tools, routers, schemas, daemon client, setup script, and docs; document XCTESTRUN + UI Automation consent; tests: generic fixtures, WDA env pinning, deterministic recorder seams Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…cels - Mock _require_ios_host on the driver fixture so _resolve_device tests never spawn real xcodebuild or fail on non-macOS CI - Synthetic UDID/name/URL fixtures (no real-looking hardware identifiers) - Await the cancelled poll task on recording start failure - Base the session poll/watchdog fields as instance defaults Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- list_apps: simctl emits OpenStep plists that plistlib cannot read; route through plutil -convert json (plistlib fast-path kept) - connect(): reconnect when the MCP bridge retired but a stale session key remained; null-safe deviceUUID; widen connect-cleanup catch - device_lock: annotate_active_owner preserves lock_scope and resolves scoped lock files; cleanup_stale_locks glob matches scoped names - readiness: SKIPPED probes no longer force a degraded verdict on non-macOS hosts - recording: kill+reap display-probe on timeout/cancel; bound crash-loop respawns (spawn success no longer resets the counter; only a segment surviving a healthy interval does); ffmpeg/probe timeouts; bound concurrent conversions; guard stale seal through_time - bridge: scoped child env (no API keys into mcpbridge), start() lock, close() never masks caller errors - adb_server: iOS controller no longer aliases into the Android global - cli: batch --ios-workspace path validation; run no longer misclassifies iOS prerequisite errors as missing API keys - stream: reap cancelled simctl child; drop stale frames on target switch; annotate active_tasks with platform; queue items surface platform in the UI mapper - env scrub pops ARTEMIS_DEVICE_ID for iOS workers (daemon + MCP) - device_smoke: platform-aware error wording for iOS - docs: fix broken SDK builder example; mobile_run_task documents platform/ios_workspace; platform_guidance no longer claims video analysis is unavailable - diagnose validates platform values; visualization resolves 0-1000 coords on small screenshots - pyright-core covers the new iOS runtime/actuator modules Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Operator/schema key enum gains POWER, VOLUME_UP, VOLUME_DOWN — the iOS driver and platform guidance already support them; both platforms map them natively (regenerated action-surface fixtures) - action_names translation now keeps power/volume keys bare so iOS drivers receive supported names - Diagnoser drops the logcat-only analyze_logs tool on iOS - mobile_run_task app_path doc covers iOS .app directories - Regression test pins the iOS key set to the operator gate Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Combines feat/ios-physical (devicectl discovery, PhysicalIosDriver with WDA backend, physical recording) with the simulator branch so a single PR delivers full iOS support: simulators + paired physical devices. Conflict resolution keeps both sides' fixes: plutil listapps parsing, dead-bridge reconnect, scoped lock metadata, SKIPPED-neutral readiness, MCP env scrubbing, and physical driver/recording internals. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…elpers - Shared physical_ios_ready() predicate across pool, probe, and driver paths (paired + connected-or-absent tunnelState) - Sync and async validate_explicit_serial now share matching/state rules (UDID or device name) - normalize_device_platform/device_pool_for/target_for_platform in adb_endpoint; adopted at CLI, admin, MCP surfaces - BOOTED_SIMULATOR_ID and DEFAULT_MAX_DURATION_SECONDS replace magic literals across iOS code - Shared helpers: reap_process, devicectl_screenshot, pixel_element, device_matches_identifier - MCP worker env now reuses IosTarget.apply_to_environment; IOS_LOCK_SCOPE replaces 'ios' literals - WDA-aware smoke hints and hierarchy_backend='wda' for physical devices - Removed dead code: duplicate replay /api/devices route, ReplayManager.list_devices/_init_device, fetchDevices JS, no-op driver overrides, unused imports - for_ios_device() is the canonical SDK method; for_ios_simulator kept as alias; IosDeviceProbe with stable probe id and class alias - Docstring/wording sweep: 'iOS device' instead of simulator-only phrasing where hardware is supported Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Lease and close one-shot iOS observations; drain cancelled native and WDA requests; reap incomplete simulator recording starts and bound recovery loops. Preserve modern CoreDevice schema, unambiguous targets, per-platform queue identity, and physical-device selection without changing Android driver code. Add remote SDK iOS platform/workspace forwarding with capabilities preflight to reject legacy hosts before submission. Keep Android payloads unchanged. Add regressions and resolve protected-core typing diagnostics without changing quality thresholds. Verification: 2900 deterministic tests passed, 11 skipped; protected-core pyright reports 0 errors; Ruff and quality ratchets pass. Live simulator acceptance is blocked by Xcode agent consent; physical/Android hardware and the compatible frontend Node runtime are unavailable. No production-readiness claim.
Live acceptance on iPhone 15 Pro surfaced two defects that blocked every
normally-named physical device:
- WDA device_info reports the product family name ("iPhone"), not the
personalized devicectl name ("Jane's iPhone"). The identity guard now
rejects only a conflicting *specific* name instead of any inequality.
- A driver-launched WDA runner auto-creates a session on startup; the
foreign-session refusal then deadlocked connect(). open_session gains
adopt_existing, enabled only when the driver owns the runner process;
discovered/user-provisioned endpoints keep the refusal.
Verified live: WDA session, 319-element hierarchy, tap/swipe/home, WDA and
devicectl screenshots, pid-verified terminate, MP4 recording, clean
disconnect with zero orphaned processes.
Generated with [Devin](https://devin.ai)
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
8 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: add native iOS support for simulators and physical devices
What Changes
Add one opt-in iOS feature covering both simulators and paired physical
iPhones/iPads across the CLI, embedded and remote SDKs, daemon, MCP, and console.
simctl.polled recording use
devicectl.stream targets, and recorded-session replay preserve the selected target.
workspace explicitly, and refuses legacy hosts before submitting iOS tasks.
Developer Mode, WDA provisioning, and supported/unsupported operations.
Android Compatibility
Android remains the default platform. The Android driver implementation is not
replaced. Shared routing changes are platform-gated, and Android task/lock/
observation/video regressions are included in verification.
Hardening
The submission also addresses observed integration risks: cancellation-safe
native/HTTP child ownership, leased diagnostic sessions, scope-aware queue
deduplication, actual CoreDevice JSON compatibility, unambiguous identifiers,
and physical-device selection in the console.
Verification
Verified head:
65c3412onfeat/ios-support. Results below are from thatlineage (code commit
19e04cc; later commit is docs-only).pytest tests/unit tests/tools packages/artemis-client/tests -q:2900 passed, 11 skipped, 8 deselected, exit 0.
ruff format --check .andruff check .: exit 0.quality_ratchet.py: passed with the baseline unchanged.pyright --project pyright-core.json: 0 errors, exit 0.git diff --checkand local Markdown file links: clean.install/list/launch/terminate, native recordVideo to valid MP4, and cleanup
pass; with Xcode agent consent granted, the full MCP path also passes —
real DeviceInteraction session, 248-element hierarchy, tap/swipe input,
and clean session teardown. Without consent it refuses cleanly
(XcodeApprovalRequiredError) and releases all resources.
ng buildpasses; karma suite 139/139 SUCCESS(Chrome Headless, Node 22.23.3).
uv buildproduces wheels containing every iOS module and thefrontend bundle; fresh-venv install passes;
artemis/artemis-admin/mcpentry points run;
--platform, iOS UDID, and--ios-workspaceCLI surfaceverified.
AndroidAdbDriversmoke on an Android 36 emulator (emulator-5554) — connect, real 1080x2400 screenshot, Settings launch, tap, swipe, clean disconnect (screenshot below). No physical Android handset is attached.model or a driver smoke test.
Evidence
iOS 27.0 simulator screen captured through the live Xcode MCP
DeviceInteractionpipeline (XcodeSimulatorDriver.get_screen_data, 248parsed hierarchy elements):
Android 36 emulator screen through the unchanged
AndroidAdbDriver:Live physical acceptance on iPhone 15 Pro (Xcode 27, LAN tunnel): WDA
session via driver-launched runner, 319-element hierarchy at 1178x2556,
tap/swipe/home on-device, WDA and devicectl screenshots, pid-verified app
terminate, physical recording to valid MP4, clean disconnect with zero
orphans. Two defects found and fixed by this acceptance: WDA product-family
vs personalized device names, and driver-launched runner auto-session
adoption.
The deterministic suite uses a dummy validation-only provider key for legacy
configuration checks, not a real credential. No provider calls or live hardware
acceptance are claimed by that result. The Android driver source is unchanged;
shared Android/iOS regression tests pass.
Platform Limits
Requires macOS and Xcode 27+. Xcode binds agent approval to the client
binary's code signature: unsigned/adhoc interpreters (uv/Homebrew Python)
receive ~24-hour grants requiring periodic re-approval, while signed agents
can hold persistent grants — documented in
docs/ios.md. Hardware needs auser-provisioned, reachable WDA runner and the user's device consent; WDA's
HTTP server cannot start while the device is locked. Physical recording is
lower-frame-rate screenshot polling; no iOS audio recording is added. Android
shell/logcat/app locking, unsupported keys, and cloud execution are not
advertised as iOS features.
Closes #174