diff --git a/.changeset/desktop-driver-accessibility-actions.md b/.changeset/desktop-driver-accessibility-actions.md new file mode 100644 index 0000000000..6d74b3212e --- /dev/null +++ b/.changeset/desktop-driver-accessibility-actions.md @@ -0,0 +1,5 @@ +--- +'@fluentui-react-native/desktop-driver': patch +--- + +List the accessibility actions an element publishes through a new `accessibilityActions` element property, gated by the `accessibilityActions` entry in `furn:features`. The macOS provider reports authored custom actions such as `Toggle` and `Select` by name without exposing provider handles, while Windows and Win32 keep returning `unsupported operation`. The property only lists actions; it does not invoke them. diff --git a/packages/agentic/components/references/remaining-work.md b/packages/agentic/components/references/remaining-work.md index ae236e6d31..5258c6e0ef 100644 --- a/packages/agentic/components/references/remaining-work.md +++ b/packages/agentic/components/references/remaining-work.md @@ -25,7 +25,9 @@ is documented in the owning reference, component contract, or the repository foc - **Open gates:** - rerun the current `desktop-focus` cases on Windows Fabric and Win32 with real input and zero skips; - invoke UIA `TogglePattern.Toggle` and `SelectionItemPattern.Select`, and macOS AX custom actions, natively rather - than inferring them from clicks or state reads; + than inferring them from clicks or state reads (the Desktop Driver now lists macOS AX custom actions through + `getProperty('accessibilityActions')`: Fabric publishes `Toggle` and `Select`, while Paper publishes nothing until + the declaration carries a `label`, C-A11Y-007; invocation is still open); - qualify Narrator on both Windows endpoints and VoiceOver on macOS; - resolve or explicitly accept the native state announcement gaps: Win32 editors report UIA `IsEnabled` when noneditable (C-FOCUS-015), and RNmacOS Fabric does not project disabled or selected state without the diff --git a/packages/agentic/desktop-driver/native/PROTOCOL.md b/packages/agentic/desktop-driver/native/PROTOCOL.md index c88766582b..f051c6e7ec 100644 --- a/packages/agentic/desktop-driver/native/PROTOCOL.md +++ b/packages/agentic/desktop-driver/native/PROTOCOL.md @@ -181,6 +181,17 @@ destroyed. Helpers without the feature reject the command as `unsupported-operation`; Node never requests the log unless the feature is negotiated. +Helpers that advertise the `accessibilityActions` feature also accept an +`accessibilityActions` command with the helper-private `elementId` and return +`{ "actions": ["", ...] }`: the names of the actions the platform +accessibility tree publishes for the element, in provider order. Names are +authored text only; provider handles, targets, and selectors never cross FDR1. +The command is read-only and follows the liveness rules of other element +queries. An element that publishes no actions returns an empty list. Helpers +without the feature reject the command as `unsupported-operation`, which Node +reports as `unsupported operation` rather than an empty list. The feature is +additive and does not change the wire version. + ### `cancel` and `cancelled` When a command's `AbortSignal` fires, Node sends: diff --git a/packages/agentic/desktop-driver/native/macos/README.md b/packages/agentic/desktop-driver/native/macos/README.md index c09fb03571..e5b6c45361 100644 --- a/packages/agentic/desktop-driver/native/macos/README.md +++ b/packages/agentic/desktop-driver/native/macos/README.md @@ -227,6 +227,7 @@ The AX provider supports: - bounded tree/source generation; - snapshots with explicit supported/unsupported state; - active element, focus, hit testing, clear, and accessibility activation; +- the action names an element publishes (`accessibilityActions`), including authored custom actions; - primary and secondary application windows. AppKit omits `AXEnabled` from elements that have no disabled state, and React @@ -236,6 +237,17 @@ list does not contain `AXEnabled`, matching how assistive technologies treat it. An element that lists `AXEnabled` without a readable value still reports an unsupported enabled state. Fabric views always publish the attribute. +The `accessibilityActions` command reads `AXUIElementCopyActionNames` for a live +element; an element whose actions are unsupported yields an empty list. AppKit +lists an `NSAccessibilityCustomAction` as +`Name:\nTarget:
\nSelector:`, so the provider returns +only `` and no process address or selector leaves the helper. Standard +actions pass through unchanged. React Native macOS Fabric publishes each declared +action by `name` (`Toggle`, `Select`), whereas Paper publishes only actions that +declare a `label` and names them by it (`C-A11Y-007` in +`accessibility.action-transport-evidence`). The command lists actions only and +never performs one. + Quartz Event Services supplies physical pointer, keyboard, text, and wheel input. Physical input requires Accessibility/PostEvent authority for the responsible process and an unlocked interactive Aqua session. Missing authority @@ -385,7 +397,7 @@ FURN_DESKTOP_DRIVER_MACOS_SIGNING_IDENTITY="" \ The Swift self-test covers framing, JSON, input ledger and lock recovery, capture scaling, snapshot normalization, AX diagnostics, CoreGraphics fallback -classification, permission and login-session diagnostics, and hello identity. The Node native +classification, custom AX action name parsing, permission and login-session diagnostics, and hello identity. The Node native contract additionally builds, verifies, reuses, and directly resolves the app; starts the actual long-lived process; and exercises malformed-frame recovery. diff --git a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Accessibility.swift b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Accessibility.swift index eca47bf821..18226be683 100644 --- a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Accessibility.swift +++ b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Accessibility.swift @@ -332,6 +332,14 @@ final class AccessibilityEngine { } } + func accessibilityActions(_ elementID: String, token: CancellationToken) throws -> [String] { + try requireAccessibility("Accessibility action inspection") + try token.throwIfCancelled() + let record = try requireRecord(elementID) + try validateLive(record) + return try copyAXActionNames(record.element).map(accessibilityActionName) + } + func tryClearValue(_ elementID: String) throws -> Bool { let record = try requireRecord(elementID) try validateLive(record) diff --git a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Driver.swift b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Driver.swift index a28043d54d..3f52e55778 100644 --- a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Driver.swift +++ b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Driver.swift @@ -122,6 +122,9 @@ final class Driver { let elementID = try requireString(params, "elementId") let window = try windowForElement(elementID) return CommandResult(result: try accessibility.snapshot(elementID: elementID, window: window, token: token).json()) + case "accessibilityActions": + let actions = try accessibility.accessibilityActions(try requireString(params, "elementId"), token: token) + return CommandResult(result: ["actions": actions]) case "activeElement": let window = try applications.requireWindow(try requireString(params, "windowId")) let snapshot = try accessibility.activeElement(window: window, token: token) @@ -231,6 +234,7 @@ final class Driver { return [ "endpoint": optionalString(params, "endpoint") ?? "macos", "features": [ + "accessibilityActions": trusted, "accessibilityClick": trusted, "elementScreenshot": screenshot, "focus": trusted, diff --git a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Host.swift b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Host.swift index e3b8eaef9e..4ecd87f6c9 100644 --- a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Host.swift +++ b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Host.swift @@ -14,6 +14,7 @@ func createHello() -> JSONObject { "minor": 0, ], "features": [ + "accessibilityActions", "accessibilityClick", "activeElement", "attach", diff --git a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/SelfTest.swift b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/SelfTest.swift index 4de5615e2e..687cba2e74 100644 --- a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/SelfTest.swift +++ b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/SelfTest.swift @@ -27,6 +27,7 @@ private final class SelfTestRunner { testAXWindowDiagnostics() testCoreGraphicsWindowSelection() testReactNativeAccessibilityFallbacks() + testAccessibilityActionNames() testPermissionDiagnostic() testLoginSessionState() testWindowEventMonitor() @@ -661,6 +662,29 @@ private final class SelfTestRunner { "hello carries the generated build identity" ) check((hello["protocol"] as? JSONObject)?["major"] as? Int == 1, "hello reports the wire protocol") + check( + (hello["features"] as? [String])?.contains("accessibilityActions") == true, + "hello advertises accessibility action inspection" + ) + } + + private func testAccessibilityActionNames() { + check( + accessibilityActionName("Name:Toggle\nTarget:0x6190039a0880\nSelector:didActivateAccessibilityCustomAction:") == "Toggle" + && accessibilityActionName("Name:Select\nTarget:0x61900399fe80\nSelector:didActivateAccessibilityCustomAction:") == "Select", + "a custom action is listed by its authored name without the provider target or selector" + ) + check( + accessibilityActionName("Name:Open in tab\nTarget:0x1\nSelector:didActivateAccessibilityCustomAction:") == "Open in tab", + "a custom action name keeps its spaces" + ) + check( + accessibilityActionName("Name:a\nTarget:b\nTarget:0x1\nSelector:s:") == "a\nTarget:b", + "an authored name containing the target marker is not truncated" + ) + check(accessibilityActionName("Name:Toggle") == "Toggle", "a custom action without provider fields keeps its name") + check(accessibilityActionName("AXPress") == "AXPress", "a standard accessibility action is listed unchanged") + check(accessibilityActionName("") == "", "an empty action name stays empty") } } diff --git a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Support.swift b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Support.swift index 26e99890f0..6d61531132 100644 --- a/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Support.swift +++ b/packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost/Support.swift @@ -435,6 +435,20 @@ func copyAXActionNames(_ element: AXUIElement) throws -> [String] { return names as? [String] ?? [] } +// AppKit lists an NSAccessibilityCustomAction as "Name:\nTarget:
\nSelector:". The target +// and selector are process-local details, so only the authored name is exposed. Standard AX actions are unchanged. +func accessibilityActionName(_ raw: String) -> String { + let prefix = "Name:" + guard raw.hasPrefix(prefix) else { + return raw + } + let remainder = raw.dropFirst(prefix.count) + guard let target = remainder.range(of: "\nTarget:", options: .backwards) else { + return String(remainder) + } + return String(remainder[.. CGPoint? { guard let value, CFGetTypeID(value) == AXValueGetTypeID() else { return nil diff --git a/packages/agentic/desktop-driver/references/protocol.md b/packages/agentic/desktop-driver/references/protocol.md index 8c009f9ee2..0557b920fa 100644 --- a/packages/agentic/desktop-driver/references/protocol.md +++ b/packages/agentic/desktop-driver/references/protocol.md @@ -134,6 +134,19 @@ and Win32 read UI Automation's `HelpText` without provider defaults, preserving an explicitly empty string. Missing or unsupported help text, including on the current macOS provider, returns `unsupported operation` instead of an empty value. +The `accessibilityActions` property (also readable as an attribute) lists the +action names the native accessibility tree publishes for an element, in provider +order, when `furn:features` advertises `accessibilityActions`. The macOS provider +reads `AXUIElementCopyActionNames` and lists an authored custom action, such as +`Toggle` or `Select`, by name without any provider handle, alongside any standard +actions the element publishes. An element that publishes none returns `[]`; hosts +without the feature, including Windows and Win32, return `unsupported operation` +rather than `[]`. The property only lists actions; it does not invoke them. +What an element publishes depends on the renderer: React Native macOS Fabric +publishes every declared `accessibilityActions` entry by `name`, while Paper +publishes only entries that declare a `label` and lists them by that label +(`C-A11Y-007` in `accessibility.action-transport-evidence`). + ## Errors | Condition | WebDriver error | diff --git a/packages/agentic/desktop-driver/src/host/types.ts b/packages/agentic/desktop-driver/src/host/types.ts index f6d5e6cf58..bb7d2831fb 100644 --- a/packages/agentic/desktop-driver/src/host/types.ts +++ b/packages/agentic/desktop-driver/src/host/types.ts @@ -12,6 +12,8 @@ export type Rect = { export type SupportedValue = { supported: true; value: T } | { supported: false; reason: string }; export type DesktopHostFeatures = { + /** The host lists an element's accessibility actions, including authored custom actions such as `Select` and `Toggle`. */ + accessibilityActions?: boolean; accessibilityClick: boolean; elementScreenshot: boolean; focus: boolean; @@ -182,6 +184,12 @@ export interface DesktopHost { source(windowId: string, signal?: AbortSignal): Promise; tree(windowId: string, signal?: AbortSignal): Promise; windowEvents?(lease: ApplicationLease, signal?: AbortSignal): Promise; + /** + * Lists the names of the actions the native accessibility tree publishes for an element, in provider order. Custom + * action names are returned as authored and never include provider handles. Present only when the host advertises + * `features.accessibilityActions`. + */ + accessibilityActions?(elementId: string, signal?: AbortSignal): Promise; subscribe?(listener: (event: DesktopHostEvent) => void): () => void; dispose(signal?: AbortSignal): Promise; diff --git a/packages/agentic/desktop-driver/src/hosts/fake/FakeDesktopHost.ts b/packages/agentic/desktop-driver/src/hosts/fake/FakeDesktopHost.ts index c1b9ecd4bb..ae8e64475c 100644 --- a/packages/agentic/desktop-driver/src/hosts/fake/FakeDesktopHost.ts +++ b/packages/agentic/desktop-driver/src/hosts/fake/FakeDesktopHost.ts @@ -23,6 +23,8 @@ export type FakeDesktopElement = Omit< NativeElementSnapshot, 'checked' | 'enabled' | 'expanded' | 'focused' | 'required' | 'selected' | 'visible' > & { + /** Names reported by `accessibilityActions` when the fake host advertises that feature. */ + accessibilityActions?: readonly string[]; checked?: boolean | 'mixed'; enabled?: boolean; expanded?: boolean; @@ -214,6 +216,14 @@ export class FakeDesktopHost implements DesktopHost { }; } + async accessibilityActions(elementId: string): Promise { + if (!this.features.accessibilityActions) { + throw new HostUnsupportedError('The fake host is not configured to list accessibility actions.'); + } + const element = this.requireElement(elementId) as Pick; + return [...(element.accessibilityActions ?? [])]; + } + async activate(windowId: string): Promise { this.requireWindow(windowId); this.actions.push({ type: 'activate', windowId }); diff --git a/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts b/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts index d943790aa9..59d3a3e584 100644 --- a/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts +++ b/packages/agentic/desktop-driver/src/hosts/native/NativeDesktopHost.ts @@ -219,6 +219,11 @@ export class NativeDesktopHost implements DesktopHost { return this.request('windowEvents', { lease }, signal); } + async accessibilityActions(elementId: string, signal?: AbortSignal): Promise { + const result = await this.request<{ actions: string[] }>('accessibilityActions', { elementId }, signal); + return result.actions; + } + subscribe(listener: (event: DesktopHostEvent) => void): () => void { this.listeners.add(listener); void this.getProcess().catch(() => undefined); diff --git a/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts b/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts index 0f42e5e31b..cd7a8636f7 100644 --- a/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts +++ b/packages/agentic/desktop-driver/src/server/createDesktopDriverServer.ts @@ -533,6 +533,15 @@ async function handleElementCommand( if (method === 'GET' && operation[0] === 'displayed') { return requireSupported(snapshot.visible, 'displayed'); } + if (method === 'GET' && (operation[0] === 'attribute' || operation[0] === 'property') && operation[1] === 'accessibilityActions') { + const accessibilityActions = session.target.host.accessibilityActions?.bind(session.target.host); + if (!accessibilityActions) { + throw new WebDriverError('unsupported operation', `Target "${session.target.id}" does not list accessibility actions.`); + } + return hostCommand(session, `Reading accessibility actions of element "${record.id}"`, (signal) => + accessibilityActions(record.nativeId, signal), + ); + } if (method === 'GET' && operation[0] === 'attribute' && operation[1]) { return getElementValue(snapshot, operation[1]); } diff --git a/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts b/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts index bec52c67a8..b0b5ded14b 100644 --- a/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts +++ b/packages/agentic/desktop-driver/src/server/desktopDriver.test.ts @@ -390,6 +390,50 @@ describe('Desktop Driver W3C remote end', () => { } }); + test('lists published accessibility actions only for hosts that provide them', async () => { + const element = (id: string, accessibilityActions?: readonly string[]) => ({ + id, + automationId: id, + role: 'checkbox', + scope: 'application' as const, + windowId: 'main', + rect: { x: 0, y: 0, width: 100, height: 40 }, + accessibilityActions, + }); + const windows = [{ id: 'main', title: 'Action fixture', elements: [element('toggle', ['Toggle', 'AXPress']), element('plain', [])] }]; + const options = { endpoint: 'macos', platformName: 'macos', windows } as const; + + const supported = await createDesktopDriverTestHarness({ ...options, features: { accessibilityActions: true } }); + try { + const session = await createDesktopDriverClient({ url: supported.server.url }).newSession({ + alwaysMatch: { platformName: 'macos', 'furn:target': supported.target.id }, + }); + const toggle = await session.findElement('accessibility id', 'toggle'); + expect(await toggle.getProperty('accessibilityActions')).toEqual(['Toggle', 'AXPress']); + expect(await toggle.getAttribute('accessibilityActions')).toEqual(['Toggle', 'AXPress']); + expect(await (await session.findElement('accessibility id', 'plain')).getProperty('accessibilityActions')).toEqual([]); + await session.delete(); + } finally { + await supported.close(); + } + + const withoutFeature = await createDesktopDriverTestHarness(options); + const withoutMethod = await createDesktopDriverTestHarness({ ...options, features: { accessibilityActions: true } }); + Object.assign(withoutMethod.host, { accessibilityActions: undefined }); + for (const harness of [withoutFeature, withoutMethod]) { + try { + const session = await createDesktopDriverClient({ url: harness.server.url }).newSession({ + alwaysMatch: { platformName: 'macos', 'furn:target': harness.target.id }, + }); + const toggle = await session.findElement('accessibility id', 'toggle'); + await expect(toggle.getProperty('accessibilityActions')).rejects.toMatchObject({ code: 'unsupported operation' }); + await session.delete(); + } finally { + await harness.close(); + } + } + }); + test('routes Storybook commands and invalidates only preview element references', async () => { const manifest: DesktopStoryManifest = { endpoint: 'windows', diff --git a/packages/agentic/storybook-desktop/references/story-tests.md b/packages/agentic/storybook-desktop/references/story-tests.md index 39657c8bf2..1fd21d7e2e 100644 --- a/packages/agentic/storybook-desktop/references/story-tests.md +++ b/packages/agentic/storybook-desktop/references/story-tests.md @@ -58,7 +58,10 @@ Metro resolves their dependencies. Use native selectors and WebDriver operations; the app has no DOM or JavaScript execution surface. Read native properties through `getProperty()` rather than ordinary object matchers. Feature-check native actions and explicitly skip -unsupported cases; never turn unsupported state into false. +unsupported cases; never turn unsupported state into false. For example, +`getProperty('accessibilityActions')` lists the action names an element +publishes (such as `Toggle` or `Select`) when +`browser.capabilities['furn:features']?.accessibilityActions` is true. Tests in one story run in declaration order, each in its own Node process and native session. The preview is navigated/remounted before each case. A diff --git a/references/agents/accessibility/action-transport-evidence.md b/references/agents/accessibility/action-transport-evidence.md index 8662cb7a4a..c0a704af9c 100644 --- a/references/agents/accessibility/action-transport-evidence.md +++ b/references/agents/accessibility/action-transport-evidence.md @@ -31,3 +31,12 @@ paths are relative to each platform package's root. Recheck these claims when a - **C-A11Y-005:** The React Native macOS Fabric View does not project `accessibilityState.disabled` or `selected` to AXEnabled or AXSelected. Version: `react-native-macos` 0.81.9. Observed: 2026-09-17 (PR #4314). Limits: revisit when the renderer is upgraded. +- **C-A11Y-007:** The Desktop Driver's `accessibilityActions` property lists the action names macOS publishes through + `AXUIElementCopyActionNames`. AppKit presents a custom action as `Name:\nTarget:
\nSelector:`, + and the driver exposes only ``. On the Storybook test app, Fabric listed `["Toggle"]` for both Switch + instances and `["Select"]` for both Tab instances, with no `AXPress`. Paper listed `[]` for the same declarations, + and listed `["Flip it"]` for a Switch declaring `{ name: 'Toggle', label: 'Flip it' }`, because `React/Views/RCTView.m` + builds a custom action only for entries that have a `label` and names it by that label. Version: `react-native-macos` + 0.81.9 on macOS 27.2. Observed: 2026-10-29. Limits: listing only, as invoking custom actions is not implemented; + Windows and Win32 do not publish the feature. Paper components must declare a `label` before the actions are visible + to AX.