diff --git a/CHANGELOG.md b/CHANGELOG.md index 76e84f5..c33855c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ Notable changes to Desktop Tools, newest first. Each release's GitHub release no - The app updates itself from GitHub Releases. **Update Automatically**, on by default, checks at launch and daily, downloads a newer release, and installs it once agents have been idle for 10 minutes, after the calls already running finish, then relaunches. **Check for Updates…** checks at once and asks before installing. Only a release notarized and signed with Developer ID by the Desktop Tools team, with the app's bundle ID, is installed, so permissions and approved agents are kept. The development app never updates itself. See [Updates](docs/app.md#updates). +### Changed + +- The agent cursor and window outline are per agent and colored by tool: Claude is orange, Codex blue, and any other agent gets a color fixed by its name. Each agent's arrow carries a name tag, peels off the user's pointer on its first input, travels on a slight arc from where it last acted, and returns to merge with the pointer after a few seconds idle, using motion tuned in the cursor lab. The first move waits at most about half a second, later ones 0.2 to 0.4 seconds by distance. Agents working at once each get their own arrow; agents sharing a window get nested outlines with labels side by side. + ### Removed - The Peekaboo stack, which the app stopped shipping in 0.17.0: `native/` with its vendored Peekaboo, its client and runtime, the release workflow's `stack` input, and wire 1, the Node tool logic behind `createDesktop` without `wire: 2`. `desktop-tools-build`, `desktop-tools-app` and `npm run dev-app` build only the Swift host in `native2/`, and accept `--native2` without effect; `createDesktop` and `dist/mcp.js` with options always forward to its `desktop-tools-client`, and accept `wire: 2` and `--wire 2` without effect, while `--wire 1` is refused. `dist/mcp.js` with no options still hands over to the app's bundled MCP server. An embedder whose app still embeds `native/`'s runtime must stay on 0.18.0 or move to `native2/`'s. The npm package no longer ships `native/`, and `npm run peekaboo-test` is gone. ([#233](https://github.com/githubnext/desktop-tools/pull/233)) diff --git a/docs/embed.md b/docs/embed.md index 070a6f8..4f1acfc 100644 --- a/docs/embed.md +++ b/docs/embed.md @@ -21,6 +21,6 @@ | `void desktop_tools_log(const char *message)` | Append one line of the app's own to that log, such as when the user pauses input | | `void desktop_tools_free(void *)` | Release a status or activity string | -macOS grants Accessibility, Screen Recording, and event synthesis to the embedding app. Each embedding app needs its own grants on each machine. Clients must be signed by the host's team with the exact allowlisted identifier. In turn, a client signed with a team sends nothing, not even its hello, to a socket served by a process that isn't the same user's and signed by that team, and fails with `HOST_UNTRUSTED`; this covers `desktop-tools-client` and the app's `desktop-tools-mcp`. Both checks read the signature of the exact process at the other end of the socket, by its audit token. An unsigned build, such as `swift build` makes for development, skips them, as the host checks its client only when it is signed itself. The agent cursor and window outline draw click-through panels from the GUI process, so turn them on only from an app with a running AppKit main loop. The outline's label names the agent from `DESKTOP_TOOLS_AGENT` in the requesting client's environment, which `createDesktop` sets from its `agent` option; without it the label says "Agent". For agents outside an embedding app, use the [Desktop Tools app](app.md). +macOS grants Accessibility, Screen Recording, and event synthesis to the embedding app. Each embedding app needs its own grants on each machine. Clients must be signed by the host's team with the exact allowlisted identifier. In turn, a client signed with a team sends nothing, not even its hello, to a socket served by a process that isn't the same user's and signed by that team, and fails with `HOST_UNTRUSTED`; this covers `desktop-tools-client` and the app's `desktop-tools-mcp`. Both checks read the signature of the exact process at the other end of the socket, by its audit token. An unsigned build, such as `swift build` makes for development, skips them, as the host checks its client only when it is signed itself. The agent cursor and window outline draw click-through panels from the GUI process, so turn them on only from an app with a running AppKit main loop. The outline's label names the agent from `DESKTOP_TOOLS_AGENT` in the requesting client's environment, which `createDesktop` sets from its `agent` option; without it the label names the program that runs the client, as approvals do. Claude is orange, Codex blue, and any other agent gets a color fixed by its name; each agent has its own cursor and outline. For agents outside an embedding app, use the [Desktop Tools app](app.md). The caller is the client's nearest ancestor that isn't `node`, `npx`, `npm`, a shell, `login` or `env`, such as Claude Code or the terminal app an agent runs in. Its key is the team ID and signing identifier of its running code when that signature validates, so an update that moves it keeps its approval; otherwise it is its path, plus its identifier when it has one. Its name is its app's name, else the last part of a team-signed identifier (`claude-code`), else its file name. The process can't choose either one through its arguments or environment. diff --git a/docs/tools.md b/docs/tools.md index d48c197..d446e71 100644 --- a/docs/tools.md +++ b/docs/tools.md @@ -140,9 +140,9 @@ A status item is pressed through Accessibility in its owner process without movi | `desktop_dialog_answer` | Press one dialog button by its exact title, replace one dialog text field's value, or choose a path in an Open or Save panel | | `desktop_clipboard_write` | Replace the clipboard with text, an image, or file references | -Pointer input is delivered to the exact window without moving the physical pointer, except for `desktop_hover` and a `desktop_drag` with `to_snapshot` or `modifiers`. Hover effects follow the real pointer, so it moves the user's pointer and leaves it there. Before moving, it checks that the snapshot's window is still at the same place and frontmost at the point, and refuses otherwise. Afterward, it checks that the pointer arrived and the window is still frontmost there; if either can't be confirmed, the result is `unknown`. A drag into another window or app can't stay inside one window, so it also uses the real pointer, and pressing in the source may bring its app forward. Pass the destination's snapshot as `to_snapshot`. Both points must be uncovered at their snapshot positions, or the drag is refused before input. The pointer rests on the start point, then presses, pauses, moves, rests over the destination and releases. The result is always `unknown`; inspect both windows to see whether the drop happened. A drag with `modifiers`, such as Option to copy or Command to move in Finder, or Shift to constrain, holds the real keys system-wide from before the press until after the release, so it takes the same foreground route even within one window, and its result is `unknown` too. Don't drag a window's title bar or edge to move or resize it: such a drag may be delivered without moving the window, and nothing reads the bounds back. Use `desktop_move` and `desktop_resize`, which set the window's bounds directly. A `path` adds up to 16 points visited in order in straight segments; it works in both routes within one window but not with `to_snapshot`. `desktop_press` holds the button through the same exact-window route as a drag, and the host always releases it, even when the call is interrupted or the window changes. When the host turns on the agent cursor, an orange arrow first glides to each click, scroll, drag, hover or press point, about a third of a second before the input is sent. It then shows a ripple for clicks, chevrons for scrolls and a trail for drags. For a press, it stays squeezed until the button is released, then ripples. The cursor fades after a few seconds idle. The cursor never takes clicks or focus and is left out of `desktop_inspect` and `desktop_screen` captures. It is decoration only: if it fails or is slow, input proceeds unchanged after at most 600 ms. Keyboard input, `desktop_type` and `desktop_select` don't show it. +Pointer input is delivered to the exact window without moving the physical pointer, except for `desktop_hover` and a `desktop_drag` with `to_snapshot` or `modifiers`. Hover effects follow the real pointer, so it moves the user's pointer and leaves it there. Before moving, it checks that the snapshot's window is still at the same place and frontmost at the point, and refuses otherwise. Afterward, it checks that the pointer arrived and the window is still frontmost there; if either can't be confirmed, the result is `unknown`. A drag into another window or app can't stay inside one window, so it also uses the real pointer, and pressing in the source may bring its app forward. Pass the destination's snapshot as `to_snapshot`. Both points must be uncovered at their snapshot positions, or the drag is refused before input. The pointer rests on the start point, then presses, pauses, moves, rests over the destination and releases. The result is always `unknown`; inspect both windows to see whether the drop happened. A drag with `modifiers`, such as Option to copy or Command to move in Finder, or Shift to constrain, holds the real keys system-wide from before the press until after the release, so it takes the same foreground route even within one window, and its result is `unknown` too. Don't drag a window's title bar or edge to move or resize it: such a drag may be delivered without moving the window, and nothing reads the bounds back. Use `desktop_move` and `desktop_resize`, which set the window's bounds directly. A `path` adds up to 16 points visited in order in straight segments; it works in both routes within one window but not with `to_snapshot`. `desktop_press` holds the button through the same exact-window route as a drag, and the host always releases it, even when the call is interrupted or the window changes. When the host turns on the agent cursor, each agent gets its own arrow with a name tag, in its color: orange for Claude, blue for Codex, and a color fixed by its name for any other agent. An agent's first input peels its arrow off the user's pointer; after that the arrow glides from where it last acted, along a slight arc, to each click, scroll, drag, hover or press point. Input waits at most about half a second for the first move and 0.2 to 0.4 seconds, by distance, for later ones. It then shows a ripple for clicks, chevrons for scrolls and a trail for drags. For a press, it stays squeezed until the button is released, then ripples. After a few seconds idle the arrow returns to the user's pointer and merges into it. Several agents' arrows move independently, so they can work in different windows at once. The cursor never takes clicks or focus and is left out of `desktop_inspect` and `desktop_screen` captures. It is decoration only: if it fails or is slow, input proceeds unchanged after at most 600 ms. Keyboard input, `desktop_type` and `desktop_select` don't show it. -When the host turns on the window highlight, the window an inspection or action targets gets an orange outline and a label such as "Claude is using TextEdit". It shows from the moment the request reaches the host, moves to a new window when the target changes, and fades about four seconds after the last request. The outline follows the window as it moves or resizes and stays directly above it in the window order, so windows in front of the target also cover the outline. It hides while the window is minimized, off screen or on another Space. Like the cursor, it never takes clicks or focus and is left out of captures. Requests without one exact window, such as `desktop_screen`, menus or the clipboard, don't show it. +When the host turns on the window highlight, the window an inspection or action targets gets an outline in the agent's color and a label such as "Claude is using TextEdit". Each agent has its own outline; when several use one window, their outlines nest and their labels sit side by side. It shows from the moment the request reaches the host, moves to a new window when the target changes, and fades about four seconds after that agent's last request. The outline follows the window as it moves or resizes and stays directly above it in the window order, so windows in front of the target also cover the outline. It hides while the window is minimized, off screen or on another Space. Like the cursor, it never takes clicks or focus and is left out of captures. Requests without one exact window, such as `desktop_screen`, menus or the clipboard, don't show it. `desktop_key` goes to the control the snapshot reports as focused. When there is none, as in Finder lists, outlines, tables, canvases and many web areas, it goes to whatever has keyboard focus in that window; the result's `receiver` is `observed_control` or `window_focus`. Either way the key is sent in the background, and is refused unless the app's focused element is still inside the window, and the window is still the app's key window, before each event. diff --git a/native2/Sources/DesktopHost/Hooks.swift b/native2/Sources/DesktopHost/Hooks.swift index 7f1e154..b6f44b8 100644 --- a/native2/Sources/DesktopHost/Hooks.swift +++ b/native2/Sources/DesktopHost/Hooks.swift @@ -18,7 +18,8 @@ public enum Hooks { /// Shows pointer input before it is posted; the input waits until it returns or `pointerLimit` passes. Delivery /// calls `showPointer` once per input, after its window check and before its first event: desktop_press shows /// `.hold` before pressing, and its release, however the hold ends, a single `.click` at the same point. - nonisolated(unsafe) public static var pointer: (@Sendable (PointerFeedback) async -> Void)? + /// The second argument is the requesting client, which tells agents' cursors apart; it is nil outside a call. + nonisolated(unsafe) public static var pointer: (@Sendable (PointerFeedback, pid_t?) async -> Void)? /// Decides each secure text field fill, as Peekaboo's handleSecureTextFill (MIT) asked its embedder. /// desktop_fill_password calls it from its delivery, after the snapshot and window checks, refusing first: @@ -145,9 +146,10 @@ public enum Hooks { /// Shows `feedback` from a delivery thread, which waits for it at most `pointerLimit`; a late one plays on alone. public static func showPointer(_ feedback: PointerFeedback) { guard let pointer else { return } + let client = call?.client let done = DispatchSemaphore(value: 0) Task { - await pointer(feedback) + await pointer(feedback, client) done.signal() } _ = done.wait(timeout: .now() + pointerLimit) diff --git a/native2/Sources/DesktopHost/Inspect.swift b/native2/Sources/DesktopHost/Inspect.swift index 6ba28a7..e058b91 100644 --- a/native2/Sources/DesktopHost/Inspect.swift +++ b/native2/Sources/DesktopHost/Inspect.swift @@ -300,9 +300,13 @@ extension Tools { } } -/// Runs blocking Accessibility work on its own thread so it never holds a cooperative pool thread. +/// Runs blocking Accessibility work on its own thread so it never holds a cooperative pool thread. The running call +/// goes along, so delivery's pointer feedback knows its client. func run(_ work: @escaping @Sendable () throws -> T) async throws -> T { - try await withCheckedThrowingContinuation { continuation in - Thread.detachNewThread { continuation.resume(with: Result { try work() }) } + let call = Hooks.call + return try await withCheckedThrowingContinuation { continuation in + Thread.detachNewThread { + Hooks.$call.withValue(call) { continuation.resume(with: Result { try work() }) } + } } } diff --git a/native2/Sources/DesktopTools/Agents.swift b/native2/Sources/DesktopTools/Agents.swift new file mode 100644 index 0000000..7a94146 --- /dev/null +++ b/native2/Sources/DesktopTools/Agents.swift @@ -0,0 +1,68 @@ +import AppKit +import DesktopCore +import DesktopHost +import Foundation + +/// Who an agent is on screen: the name on its cursor and outline, its color, and the key that gives each agent its own +/// cursor and outline. Claude is orange and Codex blue; any other agent gets a color from its key, the same every time. +struct AgentIdentity: Sendable, Equatable { + struct Tint: Sendable, Equatable { + let red: CGFloat, green: CGFloat, blue: CGFloat + + var color: CGColor { NSColor(srgbRed: red, green: green, blue: blue, alpha: 1).cgColor } + /// Darkened, to edge the label. + var rim: CGColor { NSColor(srgbRed: red * 0.65, green: green * 0.58, blue: blue * 0.55, alpha: 1).cgColor } + } + + static let claude = Tint(red: 0.85, green: 0.47, blue: 0.34) + static let codex = Tint(red: 0.30, green: 0.55, blue: 0.98) + /// For every other agent, picked by key: none is close to Claude's orange or Codex's blue. + static let others = [ + Tint(red: 0.30, green: 0.72, blue: 0.45), Tint(red: 0.60, green: 0.42, blue: 0.95), Tint(red: 0.92, green: 0.38, blue: 0.62), + Tint(red: 0.18, green: 0.68, blue: 0.72), Tint(red: 0.86, green: 0.66, blue: 0.18), Tint(red: 0.84, green: 0.30, blue: 0.30), + ] + + /// Input outside any call, which no agent asked for. + static let unknown = AgentIdentity(key: "agent", name: "Agent", tint: claude) + + let key: String + let name: String + let tint: Tint + + /// The agent behind `client`, the connected MCP binary: its `DESKTOP_TOOLS_AGENT` name when the server passed one, + /// else the program that runs it, as approvals name it. Resolved off the main thread while the client still waits + /// for its reply, and kept for a minute, since one client sends many calls. + static func of(client: pid_t?) -> AgentIdentity { + guard let client else { return unknown } + lock.lock() + if let (identity, at) = cache[client], Date().timeIntervalSince(at) < 60 { + lock.unlock() + return identity + } + lock.unlock() + let named = agentName(of: client) + let caller = Callers.shared.resolve(client: client)?.caller + let key = (caller?.key ?? "client:\(client)") + (named.map { " as:\($0)" } ?? "") + let identity = make(key: key, named: named, words: [caller?.identifier, caller?.name, caller?.path], caller: caller?.name) + lock.lock() + cache[client] = (identity, Date()) + lock.unlock() + return identity + } + + /// Claude's orange when its name or program says Claude, Codex's blue when it says Codex, else a color from `key`. + static func make(key: String, named: String?, words: [String?], caller: String?) -> AgentIdentity { + let text = ([named] + words).compactMap { $0?.lowercased() }.joined(separator: " ") + if text.contains("claude") { return AgentIdentity(key: key, name: named ?? "Claude", tint: claude) } + if text.contains("codex") { return AgentIdentity(key: key, name: named ?? "Codex", tint: codex) } + return AgentIdentity(key: key, name: named ?? caller ?? "Agent", tint: others[Int(fnv(key) % UInt64(others.count))]) + } + + private static let lock = NSLock() + nonisolated(unsafe) private static var cache: [pid_t: (AgentIdentity, Date)] = [:] + + /// FNV-1a, which unlike `hashValue` is the same in every process. + static func fnv(_ text: String) -> UInt64 { + text.utf8.reduce(0xcbf2_9ce4_8422_2325) { ($0 ^ UInt64($1)) &* 0x100_0000_01b3 } + } +} diff --git a/native2/Sources/DesktopTools/Cursor.swift b/native2/Sources/DesktopTools/Cursor.swift index 6e7ea5d..8a03696 100644 --- a/native2/Sources/DesktopTools/Cursor.swift +++ b/native2/Sources/DesktopTools/Cursor.swift @@ -1,251 +1,454 @@ -// Ported from native/Sources/runtime's cursor.swift; Peekaboo's feedback client calls became Hooks.pointer. +// Ported from native/Sources/runtime's cursor.swift; Peekaboo's feedback client calls became Hooks.pointer. The motion +// follows the cursor lab (native2/fixture/cursorlab), with the values tuned there. import AppKit import DesktopHost import DesktopMac import QuartzCore -/// Draws the agent's own pointer, since background input never moves the real one. +/// Draws each agent's own pointer, in its color, since background input never moves the real one. /// -/// Hooks.pointer awaits `show` before the input is posted, so it returns once the cursor arrives and the click, -/// scroll, or drag effect then plays alongside the delivered input. Points are global Core Graphics coordinates: -/// top-left origin on the primary display, in points. +/// An agent's cursor peels out of the real pointer for its first input, travels along an arc, clicks, and once its +/// agent goes quiet springs back to the live pointer and merges into it. Hooks.pointer awaits `show` before the input +/// is posted, so it returns once the cursor arrives and the click, scroll, or drag effect then plays alongside the +/// delivered input; a first input waits at most `firstWait` for the breakout and travel together, well inside +/// Hooks.pointerLimit. Points are global Core Graphics coordinates: top-left origin on the primary display, in points. @MainActor -final class AgentCursor { +final class AgentCursor: NSObject { static let shared = AgentCursor() - private static let tint = NSColor(srgbRed: 0.85, green: 0.47, blue: 0.34, alpha: 1).cgColor - private static let secondary = NSColor(srgbRed: 0.38, green: 0.62, blue: 1, alpha: 1).cgColor - private static let idle: Duration = .seconds(3) - /// The longest a held button can last, which is the Bridge's own limit for a hold's expiry. - private static let longestHold: TimeInterval = 30 - /// Off until an embedding app turns it on with `desktop_tools_set_cursor`. var enabled = false { didSet { if !enabled { reset() } } } - /// The exact window of the latest request, which Hooks.targetWindow and each feedback report before dispatch. - var window: CGWindowID? + private var cursors: [String: Cursor] = [:] + private var link: CADisplayLink? + private var last: CFTimeInterval? - private var overlays: [Overlay] = [] - /// The window the cursor is over, which it stays directly above so windows covering it cover the cursor too. - private var anchor: CGWindowID? - private var tracking: Timer? - private var position: CGPoint? - private var fade: Task? - private var holding = false - - private init() { + override private init() { + super.init() NotificationCenter.default.addObserver( forName: NSApplication.didChangeScreenParametersNotification, object: nil, queue: .main ) { _ in - MainActor.assumeIsolated { AgentCursor.shared.discard() } + MainActor.assumeIsolated { AgentCursor.shared.reset() } } } - /// Plays one pointer input's feedback, returning once the cursor has arrived where the input lands. + /// The exact window of an agent's latest request, which Hooks.targetWindow reports before dispatch. + func target(_ agent: AgentIdentity, window: CGWindowID) { + guard enabled else { return } + cursor(for: agent).window = window + } + + /// Plays one pointer input's feedback, returning once the agent's cursor has arrived where the input lands. + func show(_ feedback: Hooks.PointerFeedback, agent: AgentIdentity) async { + guard enabled else { return } + await cursor(for: agent).show(feedback) + } + + private func cursor(for agent: AgentIdentity) -> Cursor { + if let cursor = cursors[agent.key] { return cursor } + let cursor = Cursor(agent, wake: { [weak self] in self?.wake() }) + cursors[agent.key] = cursor + return cursor + } + + /// Hides every cursor at once. + private func reset() { + for cursor in cursors.values { cursor.discard() } + cursors = [:] + link?.isPaused = true + } + + // MARK: Clock + + private func wake() { + if link == nil { + link = NSScreen.screens.first?.displayLink(target: self, selector: #selector(frame(_:))) + link?.add(to: .main, forMode: .common) + } + link?.isPaused = false + } + + @objc private func frame(_ link: CADisplayLink) { + let dt = min(last.map { link.timestamp - $0 } ?? 1.0 / 60, 1.0 / 30) + last = link.timestamp + CATransaction.begin() + CATransaction.setDisableActions(true) + for cursor in cursors.values { cursor.step(now: CACurrentMediaTime(), dt: dt) } + CATransaction.commit() + // A clock left running burns CPU with nothing on screen. + if cursors.values.allSatisfy({ !$0.isVisible }) { + link.isPaused = true + last = nil + } + } +} + +/// The values tuned in the cursor lab, in seconds, points and degrees. +private enum Tuning { + static let windup = 0.06 + static let dip = 0.12 + static let peel = 0.18 + static let peelDistance = 24.0 + static let travelShort = 0.205 + static let travelLong = 0.4 + /// The longest a first input waits for the wind-up, peel and travel together. + static let firstWait = 0.54 + static let bend = 0.09 + static let bendMost = 54.0 + static let overshoot = 0.6 + static let stretchGain = 0.12 + static let stretchMost = 0.35 + static let tiltGain = 6.0 + static let tiltMost = 14.0 + static let tagStiffness = 600.0 + static let tagDamping = 0.7 + static let tagLag = 18.0 + static let press = 0.16 + static let pressScale = 0.82 + static let ripple = 0.4 + static let pop = 1.6 + static let homeStiffness = 170.0 + static let homeDamping = 0.75 + static let merge = 0.18 + /// How long an agent stays quiet before its cursor goes home. + static let idle = 3.0 + /// The longest a held button can last, which is the Bridge's own limit for a hold's expiry. + static let longestHold = 30.0 +} + +private enum Ease { + static func out(_ u: Double) -> Double { 1 - pow(1 - u, 3) } + static func inOut(_ u: Double) -> Double { u < 0.5 ? 4 * u * u * u : 1 - pow(-2 * u + 2, 3) / 2 } + /// Slow in, then past the end by an amount `back` sets, and back. + static func overshoot(_ u: Double, back: Double) -> Double { + let s = u * u * (3 - 2 * u), t = s - 1 + return 1 + (back + 1) * t * t * t + back * t * t + } + /// Down quickly, held, then released: a button press over one duration. + static func press(_ u: Double) -> Double { u < 0.25 ? out(u / 0.25) : u < 0.75 ? 1 : 1 - inOut((u - 0.75) / 0.25) } +} + +private extension CGPoint { + static func + (a: CGPoint, b: CGVector) -> CGPoint { CGPoint(x: a.x + b.dx, y: a.y + b.dy) } + static func - (a: CGPoint, b: CGPoint) -> CGVector { CGVector(dx: a.x - b.x, dy: a.y - b.y) } +} + +private extension CGVector { + static func * (v: CGVector, k: Double) -> CGVector { CGVector(dx: v.dx * k, dy: v.dy * k) } + static func + (a: CGVector, b: CGVector) -> CGVector { CGVector(dx: a.dx + b.dx, dy: a.dy + b.dy) } + var length: Double { hypot(dx, dy) } + var unit: CGVector { length < 0.001 ? CGVector(dx: 1, dy: 0) : self * (1 / length) } +} + +/// The real pointer in global Core Graphics points. +@MainActor +private func realPointer() -> CGPoint? { + guard let primary = NSScreen.screens.first else { return nil } + let location = NSEvent.mouseLocation + return CGPoint(x: location.x, y: primary.frame.maxY - location.y) +} + +private func windowInfo(_ window: CGWindowID) -> (bounds: CGRect, onScreen: Bool, level: Int)? { + guard let info = (CGWindowListCopyWindowInfo([.optionIncludingWindow], window) as? [[String: Any]])?.first, + let frame = info[kCGWindowBounds as String] as? NSDictionary, + let bounds = CGRect(dictionaryRepresentation: frame) + else { return nil } + return (bounds, info[kCGWindowIsOnscreen as String] as? Bool ?? false, info[kCGWindowLayer as String] as? Int ?? 0) +} + +// MARK: One agent's cursor + +@MainActor +private final class Cursor { + let agent: AgentIdentity + /// The exact window of the latest request. + var window: CGWindowID? + private let wake: () -> Void + private var overlays: [Overlay] = [] + /// The window the cursor is over, which it stays directly above so windows covering it cover the cursor too; nil + /// keeps it above everything, as on its way out of and back into the real pointer. + private var anchor: CGWindowID? + + private var position = CGPoint.zero + private var scale = 1.0 + private var opacity = 0.0 + /// Rotation a phase sets, such as the peel, on top of the tilt. + private var spin = 0.0 + /// Smoothed velocity in points per second, for stretch and tilt. + private var velocity = CGVector.zero + private var tilt = 0.0 + private var tag = CGPoint.zero + private var tagVelocity = CGVector.zero + private var last: CGPoint? + private var pressStart: Double? + private var holding = false + /// Whether a newer input took over from the one running. + private var generation = 0 + /// The running phase: it moves the cursor each frame and returns false once it ends. + private var phase: ((Double, Double) -> Bool)? + private var done: CheckedContinuation? + private var rest: Task? + + init(_ agent: AgentIdentity, wake: @escaping () -> Void) { + self.agent = agent + self.wake = wake + } + + var isVisible: Bool { !overlays.isEmpty } + func show(_ feedback: Hooks.PointerFeedback) async { + generation += 1 + let run = generation + rest?.cancel() switch feedback { case let .click(point, kind, window): self.window = CGWindowID(exactly: window) ?? self.window - await click(at: point, kind: kind, held: false) + guard await arrive(at: point, run: run) else { return } + if holding { + holding = false + } else { + pressStart = CACurrentMediaTime() + } + let count = switch kind { + case .double: 2 + case .triple: 3 + default: 1 + } + for index in 0.. Bool { - guard enabled else { return false } - fade?.cancel() - if overlays.isEmpty { overlays = NSScreen.screens.map(Overlay.init) } - let start = currentPosition() ?? realPointer() ?? point - position = point - anchor(at: point) - let distance = hypot(point.x - start.x, point.y - start.y) - if distance < 1, overlays.first?.isVisible == true { - // Already there, as when a held button is released: arrive at once so the input isn't delayed. - for overlay in overlays { - overlay.show(above: anchor) - overlay.place(at: point) - } - return true + /// Runs a phase until it ends or a newer input takes over. + private func run(_ step: @escaping (_ now: Double, _ dt: Double) -> Bool) async { + finish() + // Nothing steps a discarded cursor, so a phase would never end. + guard !overlays.isEmpty else { return } + wake() + await withCheckedContinuation { continuation in + done = continuation + phase = step } - let duration = 0.25 + min(distance / 1600, 1) * 0.15 - let path = Self.curve(from: start, to: point) - for overlay in overlays { - overlay.show(above: anchor) - let motion = CAKeyframeAnimation(keyPath: "position") - motion.path = path.copy(using: [overlay.transform]) - motion.duration = duration - motion.timingFunction = CAMediaTimingFunction(name: .easeInEaseOut) - motion.calculationMode = .paced - overlay.place(at: point) - overlay.cursor.add(motion, forKey: "travel") - } - try? await Task.sleep(for: .seconds(duration)) - return true } - /// Anchors the cursor to the request's window when `point` is inside it, or keeps it above everything for - /// input outside any one window, such as the Dock or the menu bar, and follows the window's order meanwhile. - private func anchor(at point: CGPoint) { - anchor = window.flatMap { window in - guard let info = Self.info(window), info.onScreen, info.bounds.contains(point) else { return nil } - return window + private func finish() { + phase = nil + let done = done + self.done = nil + done?.resume() + } + + /// Brings the cursor to `point`: out of the real pointer when it isn't up, else from wherever it is now, even on + /// its way home. False when a newer input took over. + private func arrive(at point: CGPoint, run: Int) async -> Bool { + var budget = Tuning.travelLong + if overlays.isEmpty { + overlays = NSScreen.screens.map { Overlay($0, agent: agent) } + anchor = nil + for overlay in overlays { overlay.show(above: nil) } + let start = realPointer() ?? point + position = start + tag = start + last = nil + velocity = .zero + await breakout(from: start, toward: point) + guard run == generation else { return false } + budget = Tuning.firstWait - Tuning.windup - Tuning.peel } - guard anchor != nil, tracking == nil else { return } - let timer = Timer(timeInterval: 1.0 / 60, repeats: true) { _ in - MainActor.assumeIsolated { AgentCursor.shared.follow() } + if (point - position).length < 1 { + // Already there, as when a held button is released: arrive at once so the input isn't delayed. + finish() + position = point + } else { + await travel(to: point, longest: budget) } - RunLoop.main.add(timer, forMode: .common) - tracking = timer + guard run == generation else { return false } + anchor(at: point) + return true } - /// Keeps every overlay directly above the anchor window, hidden while the window is off screen. - private func follow() { - guard let anchor, overlays.first?.isVisible == true else { - tracking?.invalidate() - tracking = nil - return + /// A wind-up at the real pointer, with a halo dipping around it, then a peel away toward the first input. + private func breakout(from origin: CGPoint, toward target: CGPoint) async { + let direction = (target - origin).unit + let side = CGVector(dx: -direction.dy, dy: direction.dx) + opacity = 1 + halo(dip: true, length: Tuning.windup + Tuning.peel * 0.5) + var start = CACurrentMediaTime() + await run { [self] now, _ in + let u = min((now - start) / Tuning.windup, 1) + position = origin + scale = 1 - Tuning.dip * Ease.out(u) + return u < 1 } - let info = Self.info(anchor) - for overlay in overlays { - if let info, info.onScreen { overlay.order(above: anchor, level: info.level) } else { overlay.conceal() } + start = CACurrentMediaTime() + let dipped = 1 - Tuning.dip + await run { [self] now, _ in + let u = min((now - start) / Tuning.peel, 1) + let out = Ease.out(u) + position = origin + direction * (Tuning.peelDistance * out) + side * (Tuning.peelDistance * 0.6 * sin(.pi * u)) + spin = -0.5 * (1 - out) + scale = dipped + (1 - dipped) * out + return u < 1 + } + spin = 0 + scale = 1 + } + + /// Peekaboo's path: a quadratic Bézier bent to one side, taking 205–400 ms by distance, with a slight overshoot. + private func travel(to target: CGPoint, longest: Double) async { + let from = position + let delta = target - from + let distance = max(delta.length, 1) + let length = min(Tuning.travelShort + min(distance / 1600, 1) * (Tuning.travelLong - Tuning.travelShort), longest) + let bend = min(distance * Tuning.bend, Tuning.bendMost) + let mid = CGPoint(x: (from.x + target.x) / 2, y: (from.y + target.y) / 2) + let normal = CGVector(dx: -delta.dy / distance * bend, dy: delta.dx / distance * bend) + let control = CGPoint(x: mid.x + normal.dx, y: mid.y + normal.dy) + let start = CACurrentMediaTime() + await run { [self] now, _ in + let u: Double = min((now - start) / length, 1) + let p: CGFloat = Ease.overshoot(u, back: Tuning.overshoot) + let q: CGFloat = 1 - p + // A quadratic Bézier's weights for its start, control and end points. + let a: CGFloat = q * q, b: CGFloat = 2 * q * p, c: CGFloat = p * p + let x: CGFloat = a * from.x + b * control.x + c * target.x + let y: CGFloat = a * from.y + b * control.y + c * target.y + position = CGPoint(x: x, y: y) + return u < 1 } } - private static func info(_ window: CGWindowID) -> (bounds: CGRect, onScreen: Bool, level: Int)? { - guard let info = (CGWindowListCopyWindowInfo([.optionIncludingWindow], window) as? [[String: Any]])?.first, - let frame = info[kCGWindowBounds as String] as? NSDictionary, - let bounds = CGRect(dictionaryRepresentation: frame) - else { return nil } - return (bounds, info[kCGWindowIsOnscreen as String] as? Bool ?? false, info[kCGWindowLayer as String] as? Int ?? 0) + /// Once the agent goes quiet, springs back to the live pointer, so it never waits for the hand to stop, then merges. + private func goHome(after seconds: Double) { + rest?.cancel() + let run = generation + rest = Task { [weak self] in + try? await Task.sleep(for: .seconds(seconds)) + guard !Task.isCancelled, let self, run == self.generation else { return } + await self.home(run: run) + } } - /// Peekaboo's agent-cursor path: a quadratic Bézier bent to one side by up to 54 points. - private static func curve(from: CGPoint, to: CGPoint) -> CGPath { - let dx = to.x - from.x, dy = to.y - from.y - let distance = max(hypot(dx, dy), 1) - let bend = min(max(distance * 0.09, 12), 54) - let control = CGPoint(x: (from.x + to.x) / 2 - dy / distance * bend, y: (from.y + to.y) / 2 + dx / distance * bend) - let path = CGMutablePath() - path.move(to: from) - path.addQuadCurve(to: to, control: control) - return path + private func home(run: Int) async { + holding = false + anchor = nil + for overlay in overlays { overlay.show(above: nil) } + var springVelocity = velocity + let k = Tuning.homeStiffness, c = 2 * Tuning.homeDamping * sqrt(k) + await self.run { [self] _, dt in + guard let pointer = realPointer() else { return false } + let gap = pointer - position + springVelocity = springVelocity + (gap * k + springVelocity * -c) * dt + position = position + springVelocity * dt + return !(gap.length < 4 && springVelocity.length < 120) + } + guard run == generation, let pointer = realPointer() else { return } + ripple(at: pointer, dashed: false, delay: 0) + halo(dip: false, length: Tuning.merge * 2) + let start = CACurrentMediaTime() + await self.run { [self] now, _ in + let u = min((now - start) / Tuning.merge, 1) + position = realPointer() ?? position + scale = 1 - 0.5 * Ease.out(u) + opacity = 1 - u + return u < 1 + } + guard run == generation else { return } + scale = 1 + opacity = 0 + discard() } - /// Where the cursor is drawn now, mid-flight included, or where it stopped before fading. - private func currentPosition() -> CGPoint? { - guard let position else { return nil } - guard let overlay = overlays.first, overlay.isVisible, let shown = overlay.cursor.presentation()?.position - else { return position } - return shown.applying(overlay.transform.inverted()) + /// Anchors the cursor to the request's window when `point` is inside it, or keeps it above everything for input + /// outside any one window, such as the Dock or the menu bar. + private func anchor(at point: CGPoint) { + anchor = window.flatMap { window in + guard let info = windowInfo(window), info.onScreen, info.bounds.contains(point) else { return nil } + return window + } + for overlay in overlays { overlay.show(above: anchor) } } - private func realPointer() -> CGPoint? { - guard let primary = NSScreen.screens.first else { return nil } - let location = NSEvent.mouseLocation - return CGPoint(x: location.x, y: primary.frame.maxY - location.y) - } + // MARK: Frames - /// Fades the cursor out once no input has arrived for a few seconds. - private func rest(after extra: TimeInterval = 0) { - fade?.cancel() - fade = Task { [weak self] in - try? await Task.sleep(for: Self.idle + .seconds(extra)) - guard !Task.isCancelled, let self else { return } - for overlay in self.overlays { overlay.fadeOut() } + func step(now: Double, dt: Double) { + guard !overlays.isEmpty else { return } + if let phase, !phase(now, dt) { finish() } + if let anchor { + // Directly above the anchor window, hidden while it is off screen. + let info = windowInfo(anchor) + for overlay in overlays { + if let info, info.onScreen { overlay.order(above: anchor, level: info.level) } else { overlay.conceal() } + } } - } - - private func discard() { - for overlay in overlays { overlay.close() } - overlays = [] + let raw = last.map { (position - $0) * (1 / dt) } ?? .zero + last = position + let smoothing = 1 - exp(-dt * 30) + velocity = velocity * (1 - smoothing) + raw * smoothing + let tiltTarget = max(-1, min(1, velocity.dx / 1000 * Tuning.tiltGain / Tuning.tiltMost)) * Tuning.tiltMost * .pi / 180 + tilt += (tiltTarget - tilt) * (1 - exp(-dt * 12)) + // The tag trails on a spring, capped so it stays beside the cursor at travel speed. + let anchorPoint = position + CGVector(dx: 14, dy: 24) + let k = Tuning.tagStiffness, c = 2 * Tuning.tagDamping * sqrt(k) + tagVelocity = tagVelocity + ((anchorPoint - tag) * k + tagVelocity * -c) * dt + tag = tag + tagVelocity * dt + let lag = tag - anchorPoint + if lag.length > Tuning.tagLag { tag = anchorPoint + lag.unit * Tuning.tagLag } + + var press = 1.0 + if holding { + press = Tuning.pressScale + } else if let pressStart { + let u = (now - pressStart) / Tuning.press + if u >= 1 { self.pressStart = nil } else { press = 1 - (1 - Tuning.pressScale) * Ease.press(max(u, 0)) } + } + var transform = CGAffineTransform.identity + let stretch = min(velocity.length / 1000 * Tuning.stretchGain, Tuning.stretchMost) + if stretch > 0.001 { + // Longer along the motion and thinner across it, keeping its area. + let angle = atan2(velocity.dy, velocity.dx), k = 1 + stretch + transform = transform.rotated(by: angle).scaledBy(x: k, y: 1 / k).rotated(by: -angle) + } + let size = scale * press + transform = transform.rotated(by: tilt + spin).scaledBy(x: size, y: size) + for overlay in overlays { overlay.draw(at: position, transform: transform, opacity: opacity, tag: tag) } } // MARK: Effects - /// Squeezes the cursor and keeps it squeezed, for at most the longest hold. - private func hold() { - holding = true - let squeeze = CAKeyframeAnimation(keyPath: "transform.scale") - squeeze.values = [1, 0.82, 0.82, 1] - squeeze.keyTimes = [0, NSNumber(value: 0.04 / Self.longestHold), 0.999, 1] - squeeze.duration = Self.longestHold - for overlay in overlays { overlay.cursor.add(squeeze, forKey: "press") } - } - - /// Releases a held squeeze. - private func lift() { - holding = false - let release = CABasicAnimation(keyPath: "transform.scale") - release.fromValue = 0.82 - release.toValue = 1 - release.duration = 0.12 - for overlay in overlays { overlay.cursor.add(release, forKey: "press") } - } - - private func press(for duration: TimeInterval) { - holding = false - let squeeze = CAKeyframeAnimation(keyPath: "transform.scale") - squeeze.values = [1, 0.82, 0.82, 1] - squeeze.keyTimes = [0, 0.25, 0.75, 1] - squeeze.duration = max(duration, 0.16) - for overlay in overlays { overlay.cursor.add(squeeze, forKey: "press") } - } - - private func ripple(at point: CGPoint, color: CGColor, dashed: Bool, delay: TimeInterval) { + private func ripple(at point: CGPoint, dashed: Bool, delay: TimeInterval) { + let color = agent.tint.color for overlay in overlays { let ring = CAShapeLayer() ring.path = CGPath(ellipseIn: CGRect(x: -18, y: -18, width: 36, height: 36), transform: nil) @@ -257,16 +460,42 @@ final class AgentCursor { overlay.effect(ring, at: point) let grow = CABasicAnimation(keyPath: "transform.scale") grow.fromValue = 0.35 - grow.toValue = 1.6 + grow.toValue = Tuning.pop let vanish = CABasicAnimation(keyPath: "opacity") vanish.fromValue = 0.95 vanish.toValue = 0 let group = CAAnimationGroup() group.animations = [grow, vanish] - group.duration = 0.5 + group.duration = Tuning.ripple group.beginTime = CACurrentMediaTime() + delay group.timingFunction = CAMediaTimingFunction(name: .easeOut) - overlay.expire(ring, after: 0.5 + delay, animation: group) + overlay.expire(ring, after: Tuning.ripple + delay, animation: group) + } + } + + /// A ring around the real pointer, standing in for the dip and pop macOS won't let us draw on the pointer itself. + private func halo(dip: Bool, length: TimeInterval) { + guard let pointer = realPointer() else { return } + for overlay in overlays { + let ring = CAShapeLayer() + ring.path = CGPath(ellipseIn: CGRect(x: -16, y: -16, width: 32, height: 32), transform: nil) + ring.fillColor = nil + ring.strokeColor = NSColor.white.withAlphaComponent(0.9).cgColor + ring.lineWidth = 1.5 + ring.shadowColor = .black + ring.shadowOpacity = 0.4 + ring.shadowRadius = 2 + ring.shadowOffset = .zero + ring.opacity = 0 + overlay.effect(ring, at: pointer + CGVector(dx: 5, dy: 8)) + let size = CAKeyframeAnimation(keyPath: "transform.scale") + size.values = dip ? [1, 1 - Tuning.dip * 2, 1] : [0.6, 1.4] + let fade = CAKeyframeAnimation(keyPath: "opacity") + fade.values = dip ? [0, 0.8, 0] : [0.8, 0] + let group = CAAnimationGroup() + group.animations = [size, fade] + group.duration = length + overlay.expire(ring, after: length, animation: group) } } @@ -286,15 +515,16 @@ final class AgentCursor { } let angle = atan2(dy, dx) - .pi / 2 let anchor = CGPoint(x: point.x + dx * 30 + 14, y: point.y + dy * 30 + 10) + let color = agent.tint.color for overlay in overlays { let layer = CAShapeLayer() layer.path = glyph layer.fillColor = nil - layer.strokeColor = Self.tint + layer.strokeColor = color layer.lineWidth = 3 layer.lineCap = .round layer.lineJoin = .round - layer.shadowColor = Self.tint + layer.shadowColor = color layer.shadowOpacity = 0.8 layer.shadowRadius = 4 layer.shadowOffset = .zero @@ -314,21 +544,21 @@ final class AgentCursor { } } + /// Holds the button down along a straight line, leaving a dotted trail; the input doesn't wait for it. private func drag(from: CGPoint, to: CGPoint, duration: TimeInterval) { let line = CGMutablePath() line.move(to: from) line.addLine(to: to) - position = to - press(for: duration + 0.1) + let color = agent.tint.color for overlay in overlays { let trail = CAShapeLayer() trail.path = line.copy(using: [overlay.transform]) trail.fillColor = nil - trail.strokeColor = Self.tint.copy(alpha: 0.85) + trail.strokeColor = color.copy(alpha: 0.85) trail.lineWidth = 3 trail.lineCap = .round trail.lineDashPattern = [1, 7] - trail.shadowColor = Self.tint + trail.shadowColor = color trail.shadowOpacity = 0.6 trail.shadowRadius = 3 trail.shadowOffset = .zero @@ -344,19 +574,23 @@ final class AgentCursor { group.animations = [draw, linger] group.duration = duration + 0.8 overlay.expire(trail, after: duration + 0.8, animation: group) - - let motion = CABasicAnimation(keyPath: "position") - motion.fromValue = from.applying(overlay.transform) - motion.toValue = to.applying(overlay.transform) - motion.duration = duration - overlay.place(at: to) - overlay.cursor.add(motion, forKey: "travel") } - rest(after: duration) + holding = true + let start = CACurrentMediaTime() + finish() + phase = { [self] now, _ in + let u = duration > 0 ? min((now - start) / duration, 1) : 1 + position = CGPoint(x: from.x + (to.x - from.x) * u, y: from.y + (to.y - from.y) * u) + if u >= 1 { holding = false } + return u < 1 + } + wake() } } -/// One click-through panel per display; each draws the same scene clipped to its screen. +// MARK: Overlay + +/// One click-through panel per display for one agent; each draws the same scene clipped to its screen. @MainActor private final class Overlay { private final class Panel: NSPanel { @@ -364,19 +598,18 @@ private final class Overlay { override var canBecomeMain: Bool { false } } - let cursor = CALayer() /// Maps global Core Graphics points into this panel's top-left-origin layer space. let transform: CGAffineTransform private let panel: Panel private let root = CALayer() private let effects = CALayer() + private let cursor = CALayer() + private let tag = CALayer() - init(_ screen: NSScreen) { + init(_ screen: NSScreen, agent: AgentIdentity) { let primary = NSScreen.screens.first?.frame ?? screen.frame - transform = CGAffineTransform( - translationX: -screen.frame.minX, y: -(primary.maxY - screen.frame.maxY)) - panel = Panel(contentRect: screen.frame, styleMask: [.borderless, .nonactivatingPanel], - backing: .buffered, defer: false) + transform = CGAffineTransform(translationX: -screen.frame.minX, y: -(primary.maxY - screen.frame.maxY)) + panel = Panel(contentRect: screen.frame, styleMask: [.borderless, .nonactivatingPanel], backing: .buffered, defer: false) panel.level = .screenSaver panel.collectionBehavior = [.canJoinAllSpaces, .stationary, .fullScreenAuxiliary, .ignoresCycle] panel.ignoresMouseEvents = true @@ -397,16 +630,15 @@ private final class Overlay { view.layer?.addSublayer(root) effects.frame = root.bounds root.addSublayer(effects) + root.addSublayer(tag) root.addSublayer(cursor) panel.contentView = view - Self.draw(cursor, scale: screen.backingScaleFactor) + Self.drawArrow(cursor, color: agent.tint.color, scale: screen.backingScaleFactor) + Self.drawTag(tag, name: agent.name, tint: agent.tint, scale: screen.backingScaleFactor) } - /// Whether the cursor is up, even while hidden with its window off screen. - var isVisible: Bool { cursor.opacity > 0 } - - /// The arrow's tip is the layer's anchor, so scaling presses toward the hotspot. - private static func draw(_ cursor: CALayer, scale: CGFloat) { + /// The arrow's tip is the layer's anchor, so scaling and turning pivot at the hotspot. + private static func drawArrow(_ cursor: CALayer, color: CGColor, scale: CGFloat) { let arrow = CGMutablePath() let points: [(CGFloat, CGFloat)] = [ (0, 0), (0, 17.5), (4.2, 13.6), (7, 19.8), (9.8, 18.6), (7, 12.5), (12.6, 12.5), @@ -418,33 +650,52 @@ private final class Overlay { cursor.opacity = 0 let shape = CAShapeLayer() shape.path = arrow - shape.fillColor = NSColor(srgbRed: 0.85, green: 0.47, blue: 0.34, alpha: 1).cgColor + shape.fillColor = color shape.strokeColor = .white shape.lineWidth = 1.5 shape.lineJoin = .round shape.contentsScale = scale - shape.shadowColor = shape.fillColor + shape.shadowColor = color shape.shadowOpacity = 0.9 shape.shadowRadius = 7 shape.shadowOffset = .zero cursor.addSublayer(shape) } - /// Shows the cursor directly above `window`, or above everything without one. + private static func drawTag(_ pill: CALayer, name: String, tint: AgentIdentity.Tint, scale: CGFloat) { + let font = NSFont.systemFont(ofSize: 11, weight: .semibold) + let width = ceil((name as NSString).size(withAttributes: [.font: font]).width) + 14 + pill.bounds = CGRect(x: 0, y: 0, width: width, height: 18) + pill.anchorPoint = .zero + pill.backgroundColor = tint.color + pill.borderColor = tint.rim + pill.borderWidth = 1 + pill.cornerRadius = 9 + pill.opacity = 0 + pill.shadowColor = .black + pill.shadowOpacity = 0.25 + pill.shadowRadius = 3 + pill.shadowOffset = .zero + let text = CATextLayer() + text.string = name + text.font = font + text.fontSize = 11 + text.foregroundColor = .white + text.alignmentMode = .center + text.truncationMode = .end + text.contentsScale = scale + text.frame = pill.bounds.insetBy(dx: 7, dy: 2) + pill.addSublayer(text) + } + + /// Shows the panel directly above `window`, or above everything without one. func show(above window: CGWindowID?) { if let window { - let level = (CGWindowListCopyWindowInfo([.optionIncludingWindow], window) as? [[String: Any]])? - .first?[kCGWindowLayer as String] as? Int ?? 0 - order(above: window, level: level) + order(above: window, level: windowInfo(window)?.level ?? 0) } else { - panel.level = .screenSaver + if panel.level != .screenSaver { panel.level = .screenSaver } panel.orderFrontRegardless() } - CATransaction.begin() - CATransaction.setDisableActions(true) - cursor.removeAnimation(forKey: "fade") - cursor.opacity = 1 - CATransaction.commit() } func order(above window: CGWindowID, level: Int) { @@ -458,11 +709,12 @@ private final class Overlay { if panel.isVisible { panel.orderOut(nil) } } - func place(at point: CGPoint) { - CATransaction.begin() - CATransaction.setDisableActions(true) + func draw(at point: CGPoint, transform shape: CGAffineTransform, opacity: Double, tag at: CGPoint) { cursor.position = point.applying(transform) - CATransaction.commit() + cursor.setAffineTransform(shape) + cursor.opacity = Float(opacity) + tag.position = at.applying(transform) + tag.opacity = Float(opacity) } func effect(_ layer: CALayer, at point: CGPoint) { @@ -490,23 +742,6 @@ private final class Overlay { } } - func fadeOut() { - let fade = CABasicAnimation(keyPath: "opacity") - fade.fromValue = cursor.presentation()?.opacity ?? 1 - fade.toValue = 0 - fade.duration = 0.4 - CATransaction.begin() - CATransaction.setDisableActions(true) - CATransaction.setCompletionBlock { [panel, cursor] in - MainActor.assumeIsolated { - if cursor.opacity == 0 { panel.orderOut(nil) } - } - } - cursor.opacity = 0 - cursor.add(fade, forKey: "fade") - CATransaction.commit() - } - func close() { panel.orderOut(nil) panel.close() diff --git a/native2/Sources/DesktopTools/Runtime.swift b/native2/Sources/DesktopTools/Runtime.swift index fa17eb4..4edbdf9 100644 --- a/native2/Sources/DesktopTools/Runtime.swift +++ b/native2/Sources/DesktopTools/Runtime.swift @@ -92,15 +92,18 @@ private final class Desktop { private init() { Hooks.targetWindow = { target in // The requesting client is still waiting for its reply, so its environment is readable now. - let agent = agentName(of: target.client) + let agent = AgentIdentity.of(client: target.client) DispatchQueue.main.async { - AgentCursor.shared.window = target.window + AgentCursor.shared.target(agent, window: target.window) AgentWindow.shared.target(window: target.window, process: target.process, agent: agent) } } Hooks.operation = recordActivity Hooks.secureTextFill = { await PasswordFill.shared.authorize($0) } - Hooks.pointer = { feedback in await AgentCursor.shared.show(feedback) } + Hooks.pointer = { feedback, client in + let agent = AgentIdentity.of(client: client) + await AgentCursor.shared.show(feedback, agent: agent) + } } func start(socket: String, client: String, after pause: UInt64 = 0) { diff --git a/native2/Sources/DesktopTools/Window.swift b/native2/Sources/DesktopTools/Window.swift index cdfbe6c..faf373d 100644 --- a/native2/Sources/DesktopTools/Window.swift +++ b/native2/Sources/DesktopTools/Window.swift @@ -4,66 +4,89 @@ import CoreGraphics import Darwin import QuartzCore -/// Outlines the window the agent is observing or acting on, with a label naming the agent and the app. +/// Outlines each window an agent is observing or acting on, in that agent's color, with a label naming the agent and +/// the app. /// -/// Hooks.targetWindow reports each call's exact window before dispatch. The outline follows that window's frame, -/// sits directly above it in the window order so anything covering the window covers the outline too, hides -/// while the window is minimized, off screen, or on another Space, and fades out once requests stop. +/// Hooks.targetWindow reports each call's exact window before dispatch. Each agent has one outline, which follows its +/// window's frame, sits directly above it in the window order so anything covering the window covers the outline too, +/// hides while the window is minimized, off screen, or on another Space, and fades out once that agent's requests stop. +/// Agents on the same window nest their outlines and set their labels side by side. @MainActor final class AgentWindow { static let shared = AgentWindow() - private static let tint = NSColor(srgbRed: 0.85, green: 0.47, blue: 0.34, alpha: 1).cgColor - /// The tint darkened, to edge the label. - private static let rim = NSColor(srgbRed: 0.55, green: 0.27, blue: 0.18, alpha: 1).cgColor - private static let labelFont = NSFont.systemFont(ofSize: 11, weight: .semibold) + fileprivate static let labelFont = NSFont.systemFont(ofSize: 11, weight: .semibold) private static let idle: Duration = .seconds(4) - /// Room around the window for the outline and the label that straddles its bottom edge. - private static let margin: CGFloat = 16 + /// Room around the window for the outlines and the labels that straddle its bottom edge. + fileprivate static let margin: CGFloat = 16 /// Off until an embedding app turns it on with `desktop_tools_set_highlight`. var enabled = false { didSet { if !enabled { reset() } } } - private var target: CGWindowID? - private var overlay: Overlay? + private final class Outline { + let overlay: Overlay + var target: CGWindowID + /// When it was first shown on `target`, which orders agents sharing a window. + var since = Date() + var fade: Task? + + init(overlay: Overlay, target: CGWindowID) { + self.overlay = overlay + self.target = target + } + } + + private var outlines: [String: Outline] = [:] private var tracking: Timer? - private var fade: Task? - /// Shows the outline on `window` and restarts the idle fade. - func target(window: CGWindowID, process: pid_t, agent: String?) { + /// Shows `agent`'s outline on `window` and restarts its idle fade. + func target(window: CGWindowID, process: pid_t, agent: AgentIdentity) { guard enabled else { return } let app = NSRunningApplication(processIdentifier: process)?.localizedName ?? "an app" - let label = "\(agent ?? "Agent") is using \(app)" - if target != window || overlay == nil { - overlay?.close() - overlay = Overlay(label: label) - target = window + let label = "\(agent.name) is using \(app)" + let outline: Outline + if let existing = outlines[agent.key], existing.target == window { + outline = existing + outline.overlay.label = label } else { - overlay?.label = label + outlines[agent.key]?.fade?.cancel() + outlines[agent.key]?.overlay.close() + outline = Outline(overlay: Overlay(label: label, tint: agent.tint), target: window) + outlines[agent.key] = outline } - overlay?.present() + outline.overlay.present() track() - fade?.cancel() - fade = Task { [weak self] in + outline.fade?.cancel() + outline.fade = Task { [weak self, weak outline] in try? await Task.sleep(for: Self.idle) - guard !Task.isCancelled, let self else { return } - self.overlay?.fadeOut { [weak self] in self?.reset() } + guard !Task.isCancelled, let outline else { return } + outline.overlay.fadeOut { [weak self] in self?.remove(agent.key, outline) } + } + } + + private func remove(_ key: String, _ outline: Outline) { + guard outlines[key] === outline else { return } + outline.overlay.close() + outlines[key] = nil + if outlines.isEmpty { + tracking?.invalidate() + tracking = nil } } private func reset() { - fade?.cancel() - fade = nil tracking?.invalidate() tracking = nil - overlay?.close() - overlay = nil - target = nil + for outline in outlines.values { + outline.fade?.cancel() + outline.overlay.close() + } + outlines = [:] } - /// Polls the window's frame and order at display rate while the outline is up. + /// Polls the windows' frames and order at display rate while an outline is up. private func track() { update() guard tracking == nil else { return } @@ -75,16 +98,32 @@ final class AgentWindow { } private func update() { - guard let target, let overlay else { return } + var shared: [CGWindowID: [(key: String, outline: Outline)]] = [:] + for (key, outline) in outlines { shared[outline.target, default: []].append((key, outline)) } + for (_, group) in shared { + let ordered = group.sorted { ($0.outline.since, $0.key) < ($1.outline.since, $1.key) } + for (slot, entry) in ordered.enumerated() { + entry.outline.overlay.slot = (slot, ordered.count) + if !update(entry.outline) { remove(entry.key, entry.outline) } + } + } + } + + /// False once the window is gone. + private func update(_ outline: Outline) -> Bool { + let target = outline.target, overlay = outline.overlay guard let info = (CGWindowListCopyWindowInfo([.optionIncludingWindow], target) as? [[String: Any]])?.first - else { return reset() } + else { return false } var bounds = CGRect.null if let frame = info[kCGWindowBounds as String] as? NSDictionary { bounds = CGRect(dictionaryRepresentation: frame) ?? .null } let onScreen = info[kCGWindowIsOnscreen as String] as? Bool ?? false let visible = NSScreen.screens.contains { Self.flipped($0.frame).intersects(bounds) } - guard onScreen, visible, !bounds.isEmpty else { return overlay.conceal() } + guard onScreen, visible, !bounds.isEmpty else { + overlay.conceal() + return true + } // The display showing most of the window keeps the outline and label on it when the window fills it. let screen = NSScreen.screens.max { area(of: $0) < area(of: $1) }?.frame ?? .null func area(of screen: NSScreen) -> CGFloat { @@ -96,6 +135,7 @@ final class AgentWindow { .map { $0.offsetBy(dx: -frame.minX, dy: -frame.minY) } overlay.follow(target, frame: Self.flipped(frame), screen: screen, level: info[kCGWindowLayer as String] as? Int ?? 0, covers: covers) + return true } /// The bounds of the window's own sheets and panels stacked directly above it, such as an Open panel on a @@ -144,12 +184,17 @@ final class AgentWindow { /// The window's own sheets and panels above it, in the panel's flipped coordinates. private var covers: [CGRect] = [] + /// This agent's place among the agents on the same window. + var slot = (index: 0, count: 1) { + didSet { if slot != oldValue { layout() } } + } + var label: String { get { text.string as? String ?? "" } set { text.string = newValue; layout() } } - init(label: String) { + init(label: String, tint: AgentIdentity.Tint) { panel = Panel(contentRect: .zero, styleMask: [.borderless, .nonactivatingPanel], backing: .buffered, defer: false) panel.collectionBehavior = [.canJoinAllSpaces, .fullScreenAuxiliary, .ignoresCycle, .transient] @@ -172,17 +217,17 @@ final class AgentWindow { panel.contentView = view outline.fillColor = nil - outline.strokeColor = AgentWindow.tint + outline.strokeColor = tint.color outline.lineWidth = 3 - outline.shadowColor = AgentWindow.tint + outline.shadowColor = tint.color outline.shadowOpacity = 0.7 outline.shadowRadius = 6 outline.shadowOffset = .zero root.addSublayer(outline) - pill.backgroundColor = AgentWindow.tint + pill.backgroundColor = tint.color pill.cornerRadius = 9 - pill.borderColor = AgentWindow.rim + pill.borderColor = tint.rim pill.borderWidth = 1 pill.shadowColor = .black pill.shadowOpacity = 0.25 @@ -202,6 +247,15 @@ final class AgentWindow { func present() { root.removeAnimation(forKey: "fade") guard root.opacity < 1 else { return } + if root.presentation()?.opacity ?? 0 == 0 { + // A new outline draws itself around the window as the cursor arrives. + let draw = CABasicAnimation(keyPath: "strokeEnd") + draw.fromValue = 0 + draw.toValue = 1 + draw.duration = 0.16 + draw.timingFunction = CAMediaTimingFunction(name: .easeInEaseOut) + outline.add(draw, forKey: "draw") + } let fade = CABasicAnimation(keyPath: "opacity") fade.fromValue = root.presentation()?.opacity ?? 0 fade.toValue = 1 @@ -263,7 +317,8 @@ final class AgentWindow { let display = screen.isNull ? root.bounds : CGRect(x: screen.minX - frame.minX, y: frame.maxY - screen.maxY, width: screen.width, height: screen.height) .insetBy(dx: 2, dy: 2) - let outside = window.insetBy(dx: -2, dy: -2) + // Each further agent on the window nests its outline 5 points out. + let outside = window.insetBy(dx: -2 - CGFloat(slot.index) * 5, dy: -2 - CGFloat(slot.index) * 5) let clamped = outside.intersection(display) let edge = clamped.isNull || clamped.isEmpty ? outside : clamped outline.path = CGPath(roundedRect: edge, cornerWidth: 14, cornerHeight: 14, transform: nil) @@ -271,7 +326,8 @@ final class AgentWindow { let width = min(ceil(measured.width) + 16, max(window.width - 32, 60)) // The label straddles the bottom edge, or sits just inside it when the window reaches the display's bottom. let top = clamped.isNull || clamped.isEmpty ? edge.maxY - 9 : min(edge.maxY - 9, display.maxY - 20) - pill.frame = CGRect(x: window.midX - width / 2, y: top, width: width, height: 18) + let spread = (CGFloat(slot.index) - CGFloat(slot.count - 1) / 2) * (width + 8) + pill.frame = CGRect(x: window.midX - width / 2 + spread, y: top, width: width, height: 18) text.frame = pill.bounds.insetBy(dx: 8, dy: 2) text.contentsScale = panel.backingScaleFactor if covers.isEmpty { diff --git a/native2/Tests/RuntimeTests/AgentIdentityTests.swift b/native2/Tests/RuntimeTests/AgentIdentityTests.swift new file mode 100644 index 0000000..12d443d --- /dev/null +++ b/native2/Tests/RuntimeTests/AgentIdentityTests.swift @@ -0,0 +1,29 @@ +@testable import DesktopTools +import Testing + +struct AgentIdentityTests { + @Test + func `Claude is orange and Codex blue, by name or by the program that runs them`() { + let claude = AgentIdentity.make(key: "a", named: nil, words: ["com.anthropic.claudefordesktop", nil, nil], caller: "Claude") + #expect(claude.tint == AgentIdentity.claude) + #expect(claude.name == "Claude") + let codex = AgentIdentity.make(key: "b", named: nil, words: [nil, nil, "/opt/homebrew/bin/codex"], caller: "codex") + #expect(codex.tint == AgentIdentity.codex) + #expect(codex.name == "Codex") + let named = AgentIdentity.make(key: "c", named: "Codex", words: [nil, "Ghostty", nil], caller: "Ghostty") + #expect(named.tint == AgentIdentity.codex) + #expect(named.name == "Codex") + } + + @Test + func `any other agent gets a palette color that its key fixes`() { + let gemini = AgentIdentity.make(key: "team:X as:Gemini", named: "Gemini", words: [nil, "Ghostty", nil], caller: "Ghostty") + #expect(gemini.name == "Gemini") + #expect(AgentIdentity.others.contains(gemini.tint)) + #expect(AgentIdentity.make(key: "team:X as:Gemini", named: "Gemini", words: [], caller: nil).tint == gemini.tint) + let tints = Set((0..<40).map { AgentIdentity.make(key: "agent \($0)", named: nil, words: [], caller: nil).tint.red }) + #expect(tints.count > 3) + #expect(AgentIdentity.make(key: "k", named: nil, words: [], caller: nil).name == "Agent") + #expect(AgentIdentity.fnv("a") == 0xaf63_dc4c_8601_ec8c) + } +}