feat: native iOS support for simulators and physical devices

README.md

@@ -104,6 +104,21 @@ cd artemis
 
 > **Tip**: Opens `http://localhost:8000` in your default browser with a device connection wizard, live screen mirroring, prompt sandbox, and execution replays. You can also run directly from CLI: `uv run artemis run "Open Settings, find Battery and tell me current level" --profile flash`.
 
+### iOS Devices (macOS, Xcode 27+)
+
+CLI, embedded Python SDK, web console, and daemon tasks can target iOS
+simulators through Xcode's native MCP device-interaction tools, and paired
+physical iPhones/iPads through `devicectl` plus a WebDriverAgent runner
+installed on the device. See the
+[iOS setup and usage guide](./docs/ios.md) for prerequisites, Xcode access
+approval, WDA setup, and current limits:
+
+```bash
+bash scripts/setup_ios_env.sh
+uv run artemis run "Open Settings and inspect the General screen" \
+  --platform ios --standalone --device-serial <SIMULATOR-UDID> --profile flash
+```
+
 <a id="mcp-setup"></a>
 <a id="mcp"></a>
 <details>
@@ -300,7 +315,7 @@ ARTEMIS supports two execution profiles tailored for different automation requir
 ## Roadmap
 
 - [ ] **Android Studio Integration**: Native IDE plugin and workflow integration to enable in-editor debugging, test recording, and automated device control directly within Android Studio.
-- [ ] **iOS Platform Expansion**: Extending multimodal perception and mobile automation to iOS devices and simulators.
+- [x] **iOS Platform Expansion**: [iOS support](./docs/ios.md) is available with Xcode 27+ across the CLI, SDK, web console, daemon, MCP, replay, and streaming, covering simulators and paired physical devices.
 - [ ] **On-Device Lightweight VLMs**: Local execution with lightweight edge vision models for low-latency, privacy-first automation.
 - [ ] **Real-time Duplex Voice Interaction**: Voice-driven task dispatch with real-time conversational control and interruption handling.
 

README_CN.md

@@ -106,6 +106,17 @@ cd artemis
 
 > **提示**：启动后将自动在默认浏览器中打开 Web 控制台（`http://localhost:8000`），提供设备连接向导、实时投屏、任务演练与状态回放面板。你也可以通过命令行直接运行：`uv run artemis run "打开系统设置，找到电池选项并告诉我当前电量" --profile flash`。
 
+### iOS 设备（macOS，Xcode 27+）
+
+CLI、嵌入式 Python SDK、Web 控制台与守护进程任务都可以通过 Xcode 原生 MCP 设备交互工具控制 iOS 模拟器；真机（iPhone/iPad）则通过 `devicectl` 管理生命周期、并在设备上安装 WebDriverAgent 负责界面操作。
+安装要求、Xcode 权限审批、WDA 配置和当前限制见 [iOS 使用指南（英文）](./docs/ios.md)：
+
+```bash
+bash scripts/setup_ios_env.sh
+uv run artemis run "打开系统设置并查看通用页面" \
+  --platform ios --standalone --device-serial <SIMULATOR-UDID> --profile flash
+```
+
 <a id="mcp-setup"></a>
 <a id="mcp"></a>
 <details>
@@ -298,7 +309,7 @@ ARTEMIS 提供两种运行模式以适应不同的自动化需求：
 ## 路线图
 
 - [ ] **Android Studio 深度集成**：推出官方 IDE 插件与协同工作流，支持在 Android Studio 内直接进行自动化测试、设备交互与断点调试。
-- [ ] **iOS 跨平台支持**：将视觉感知与自动化执行引擎拓展至 iOS 真机与模拟器。
+- [x] **iOS 跨平台支持**：[iOS 支持](./docs/ios.md)已支持 Xcode 27+，覆盖 CLI、SDK、Web 控制台、守护进程、MCP、回放与投屏，含模拟器与已配对真机。
 - [ ] **端侧轻量化模型**：支持离线运行的轻量级 Edge VLM，实现低延迟与隐私安全的本地自动化。
 - [ ] **实时语音双工交互**：支持自然语音下发任务与实时打断（Barge-in）控制。
 

apps/admin_console/index.html

@@ -1442,29 +1442,6 @@ <h3 style="font-size: 0.95rem; margin-bottom: 3px; margin-top: 0; white-space: n
         let activeReplayNodeDetails = null;
         const replayInProgressSteps = new Set();
 
-        async function fetchDevices() {
-            try {
-                const res = await fetch(`/api/devices?_t=${Date.now()}`);
-                const devices = await res.json();
-                const select = document.getElementById('replay-device-select');
-                select.innerHTML = '';
-
-                if (devices.length === 0) {
-                    select.innerHTML = '<option value="">No devices detected</option>';
-                    return;
-                }
-
-                devices.forEach(d => {
-                    const opt = document.createElement('option');
-                    opt.value = d.serial;
-                    opt.innerText = `${d.serial} (${d.status})`;
-                    select.appendChild(opt);
-                });
-            } catch (err) {
-                console.error("Failed to fetch devices:", err);
-            }
-        }
-
         let replayTools = [];
 
         async function fetchReplayTools() {

apps/admin_console/replay_manager.py

@@ -113,21 +113,15 @@ class ReplayManager:
     def __init__(
         self,
         workspace_root: Path = WORKSPACE_ROOT,
-        device_id: str = None,
         original_db_path: Path = None,
-        init_device: bool = False,
     ):
-        """Initializes the ReplayManager and autodetects the connected Android device.
+        """Initializes the ReplayManager.
 
         Args:
             workspace_root (Path, optional): The absolute path to the workspace
               root. Defaults to WORKSPACE_ROOT.
-            device_id (str, optional): The serial ID of the target Android
-              device. If not specified, autodetects the first connected device.
             original_db_path (Path, optional): Path to the master SQLite
               database. Defaults to DB_PATH.
-            init_device (bool, optional): Whether to initialize the ADB device
-              connection on startup.
         """
         self.workspace_root = workspace_root
         self.traces_path = TRACES_PATH
@@ -137,62 +131,6 @@ def __init__(
         self.db_path = Path(original_db_path) if original_db_path else DB_PATH
         self.images_dir = IMAGES_DIR
 
-        # Device connection and autodetection
-        self.device_id = None
-        self.w = None
-        self.h = None
-        self.adb = None
-        self.ui_client = None
-
-        if init_device:
-            try:
-                self._init_device(device_id)
-            except Exception as e:
-                print(
-                    "Warning: Connected device could not be initialized during"
-                    f" ReplayManager startup: {e}"
-                )
-
-    def _init_device(self, device_id: str = None):
-        """Initializes connection to the target device and queries screen metrics."""
-        from adbutils import AdbClient
-
-        try:
-            from third_party.mobile_use.clients.ui_automator_client import UIAutomatorClient
-        except ImportError:
-            raise ImportError(
-                "Failed to import UIAutomatorClient. Ensure artemis package is installed in path."
-            )
-
-        self.adb = AdbClient(host="localhost", port=5037)
-        try:
-            devices = self.adb.device_list()
-        except Exception as adb_err:
-            raise ConnectionError(
-                f"Failed to query device list from ADB server: {adb_err}"
-            ) from adb_err
-
-        if not devices:
-            raise ConnectionError(
-                "No active ADB devices connected. Please connect an Android device via ADB."
-            )
-
-        if device_id:
-            matched_device = next((d.serial for d in devices if d.serial == device_id), None)
-            if not matched_device:
-                raise ConnectionError(
-                    f"Requested device '{device_id}' is not connected."
-                    f" Connected devices: {[d.serial for d in devices]}"
-                )
-            self.device_id = matched_device
-        else:
-            self.device_id = devices[0].serial
-
-        self.ui_client = UIAutomatorClient(device_id=self.device_id)
-        ui_data = self.ui_client.get_screen_data()
-        self.w, self.h = ui_data.width, ui_data.height
-        print(f"Connected to device: {self.device_id} ({self.w}x{self.h})")
-
     def chunk_session_traces(self, session_id: str, output_dir: Path = None) -> Path:
         """Chunks traces, screenshots, and metadata for a specific execution session.
 
@@ -396,18 +334,6 @@ def chunk_session_traces(self, session_id: str, output_dir: Path = None) -> Path
         print("Chunking trace completion status: success")
         return output_dir
 
-    def list_devices(self) -> list[dict]:
-        """Dynamically queries the ADB server for connected Android devices."""
-        try:
-            from adbutils import AdbClient
-
-            adb = AdbClient(host="localhost", port=5037)
-            devices = adb.device_list()
-            return [{"serial": d.serial, "status": "online"} for d in devices]
-        except Exception as e:
-            print(f"Warning: Failed to query device list from ADB: {e}")
-            return []
-
     def load_session_goal(
         self, session_id: str, step_dir: Path, original_db_path: str = None
     ) -> str:
@@ -1604,12 +1530,69 @@ def _preemptive_clip_video(
         except Exception as e:
             print(f"Warning: Failed to update sandbox SQLite database: {e}")
 
+    def _session_device_info(self, session_id: str) -> dict:
+        """The recorded session's ``device_info`` blob (``{}`` when absent)."""
+        import sqlite3
+
+        try:
+            conn = sqlite3.connect(self.db_path)
+            try:
+                row = conn.execute(
+                    "SELECT device_info FROM sessions WHERE session_id = ?",
+                    (session_id,),
+                ).fetchone()
+            finally:
+                conn.close()
+        except sqlite3.Error:
+            return {}
+        if not row or not row[0]:
+            return {}
+        try:
+            info = json.loads(row[0])
+        except (ValueError, TypeError):
+            # Malformed device_info: simulate with the defaults.
+            return {}
+        return info if isinstance(info, dict) else {}
+
+    @staticmethod
+    def _replay_device_context(device_info: dict, override_device_id: str | None):
+        """Build the simulated ``DeviceContext`` for a replayed session.
+
+        The recorded session decides the platform: an iOS session replays
+        through the native iOS driver, never through ADB. The frontend's
+        device pick only retargets iOS replays (the picker was always
+        decorative for Android and stays that way); the driver still
+        validates the UDID — simctl for simulators, devicectl for physical
+        devices — at connect time.
+        """
+        from artemis.context import DeviceContext, DevicePlatform
+
+        sim_platform = str(
+            device_info.get("mobile_platform") or DevicePlatform.ANDROID.value
+        ).lower()
+        mobile_platform = (
+            DevicePlatform.IOS
+            if sim_platform == DevicePlatform.IOS.value
+            else DevicePlatform.ANDROID
+        )
+        sim_device_id = device_info.get("device_id", "replay-device")
+        if mobile_platform == DevicePlatform.IOS and override_device_id:
+            sim_device_id = override_device_id
+        return DeviceContext(
+            host_platform=("DARWIN" if mobile_platform == DevicePlatform.IOS else "LINUX"),
+            mobile_platform=mobile_platform,
+            device_id=sim_device_id,
+            device_width=device_info.get("device_width", 1080),
+            device_height=device_info.get("device_height", 2400),
+        )
+
     def create_ctx(
         self,
         session_id: str,
         step_number: int,
         agent_name: str = "explorer",
         replay_id: str = None,
+        override_device_id: str = None,
     ) -> ArtemisContext:
         """Creates a sandboxed ArtemisContext and DataEngine for the given session and step."""
         self._ensure_session_chunked(session_id)
@@ -1682,37 +1665,13 @@ def create_ctx(
         except ImportError as import_err:
             raise ImportError(f"Failed to import Artemis core modules: {import_err}")
 
-        import sqlite3
-
-        conn = sqlite3.connect(self.db_path)
-        cursor = conn.cursor()
-        cursor.execute(
-            "SELECT device_info FROM sessions WHERE session_id = ?",
-            (session_id,),
-        )
-        row = cursor.fetchone()
-        conn.close()
-
-        device_info = {}
-        if row and row[0]:
-            try:
-                device_info = json.loads(row[0])
-            except (ValueError, TypeError):
-                # Malformed device_info: simulate with the defaults below.
-                pass
-
-        sim_device_id = device_info.get("device_id", "replay-device")
-        sim_w = device_info.get("device_width", 1080)
-        sim_h = device_info.get("device_height", 2400)
-
-        device_context = DeviceContext(
-            host_platform="LINUX",
-            mobile_platform=DevicePlatform.ANDROID,
-            device_id=sim_device_id,
-            device_width=sim_w,
-            device_height=sim_h,
+        device_info = self._session_device_info(session_id)
+        device_context = self._replay_device_context(device_info, override_device_id)
+        sim_device_id = device_context.device_id
+        print(
+            f"Replay simulated device: {sim_device_id} "
+            f"({device_context.device_width}x{device_context.device_height})"
         )
-        print(f"Replay simulated device: {sim_device_id} ({sim_w}x{sim_h})")
 
         from artemis.config import get_default_llm_config
 
@@ -1765,6 +1724,7 @@ def instantiate_state(
                 pre_image_meta = json.load(f)
 
         initial_goal = self.load_session_goal(session_id, step_dir, str(self.db_path))
+        device_info = self._session_device_info(session_id)
 
         # State has no fields for the app info or device date (extra="forbid").
         ui_hier, decisions, _app_info, _dev_date = self.extract_state_prepopulation_data(
@@ -1781,7 +1741,10 @@ def instantiate_state(
             latest_screenshot=str(step_dir / "pre.jpg")
             if (step_dir / "pre.jpg").exists()
             else str(step_dir / "post.jpg"),
-            operator_raw_data={"width": self.w, "height": self.h},
+            operator_raw_data={
+                "width": device_info.get("device_width"),
+                "height": device_info.get("device_height"),
+            },
             current_step_id=step_data["step_id"],
             subagent_calls=step_data.get("subagent_calls") or [],
             latest_ui_hierarchy=ui_hier,
@@ -1872,6 +1835,7 @@ async def replay_step_tool(
             step_number=step_number,
             agent_name=agent_name,
             replay_id=replay_id,
+            override_device_id=override_device_id,
         )
 
         state = self.instantiate_state(

apps/admin_console/routers/replay.py

@@ -30,12 +30,6 @@
 router = APIRouter(tags=["replay"])
 
 
-@router.get("/api/devices")
-async def list_devices():
-    """Dynamically queries the ADB server for connected Android devices."""
-    return replay_manager.list_devices()
-
-
 @router.get("/api/replay/tools")
 async def get_replay_tools():
     """Returns the list of registered tools available for replay."""

apps/admin_console/routers/stream.py

@@ -45,12 +45,13 @@ async def stream_device_live():
 
 @router.get("/api/stream/device-state")
 async def get_device_stream_state():
-    """Returns whether an ADB device is connected and live streaming is available."""
-    serial = await device_stream_service.get_device_serial()
+    """Returns whether a device is connected and live streaming is available."""
+    target = await device_stream_service.get_stream_target()
     return JSONResponse(
         {
-            "connected": serial is not None,
-            "serial": serial,
-            "live_stream_url": "/api/stream/device-live" if serial else None,
+            "connected": target is not None,
+            "serial": target["serial"] if target else None,
+            "platform": target["platform"] if target else None,
+            "live_stream_url": "/api/stream/device-live" if target else None,
         }
     )