Skip to content

Desktop Driver: list published accessibility actions (macOS AX Select/Toggle) - #4426

Open
Jason Morse (JasonVMo) wants to merge 2 commits into
mainfrom
user/jasonvmo/desktop-driver-ax-actions
Open

Jason Morse (JasonVMo) wants to merge 2 commits into
mainfrom
user/jasonvmo/desktop-driver-ax-actions

Conversation

@JasonVMo

Copy link
Copy Markdown
Contributor

Platforms Impacted

  • iOS
  • macOS
  • win32 (Office)
  • windows
  • android

Desktop Driver tooling only: the platform-neutral server and host contract, the macOS native helper, and documentation. No component source changes. The Windows and Win32 helpers are unchanged and report the new property as unsupported.

Description of changes

The Desktop Driver can now list the accessibility actions an element publishes. A new element property, accessibilityActions (element.getProperty('accessibilityActions'); the attribute route serves the same list), returns the action names in provider order. Before this, tests had no way to see whether a component published the AX Toggle or Select custom actions: the attribute and property routes returned null for names they did not know. On macOS the helper reads AXUIElementCopyActionNames. AppKit lists a custom action as Name:<name>\nTarget:<address>\nSelector:<selector>, so the helper returns only the authored name and no process-local address or selector crosses the boundary. Standard actions such as AXPress pass through unchanged.

Design:

  • Optional feature, following windowEvents: an optional DesktopHost.accessibilityActions?(), a features.accessibilityActions flag that tests see as furn:features, and a hello feature string. The wire protocol stays at 1.0.
  • Unsupported never becomes an empty list. A host that cannot list actions fails the request with W3C unsupported operation. The Windows helper does not implement the command and rejects it as unsupported-operation (native/windows/src/driver.cpp), which the server surfaces the same way.
  • Listing only. Invoking a custom action needs input-lock, abort, and stale-element handling plus the raw action name, so it is left for a separate change.

macOS helper (packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost):

  • Support.swift: accessibilityActionName reduces the AppKit string to the authored name. It strips from the last \nTarget:, so a name that itself contains that text is kept whole.
  • Accessibility.swift: AccessibilityEngine.accessibilityActions requires Accessibility trust, honors cancellation, validates that the element is still live, and maps the names.
  • Driver.swift, Host.swift: dispatch the command and advertise it in the hello features and the probe's features.
  • SelfTest.swift: parser cases (Toggle and Select from raw strings, spaces kept, an authored \nTarget: kept, name only, AXPress, empty) and a hello feature check.

Node host and server (packages/agentic/desktop-driver/src):

  • host/types.ts, hosts/native/NativeDesktopHost.ts, hosts/fake/FakeDesktopHost.ts: the optional method and feature flag. The fake host takes an accessibilityActions list per element and throws HostUnsupportedError unless the feature is enabled.
  • server/createDesktopDriverServer.ts: GET of property/accessibilityActions or attribute/accessibilityActions returns the host's list, or unsupported operation when the host has no such method.
  • server/desktopDriver.test.ts: covers property and attribute reads, an element with no actions, and both unsupported paths (feature off, method absent).

Documentation and release:

  • native/PROTOCOL.md, references/protocol.md, native/macos/README.md, storybook-desktop/references/story-tests.md: the command, feature negotiation, and how tests should guard on furn:features.
  • references/agents/accessibility/action-transport-evidence.md: new claim C-A11Y-007 with the raw format and the Fabric and Paper results below. It is numbered 007 because 006 is reserved by pending macOS selection-test work that is not published yet.
  • components/references/remaining-work.md: the macOS AX custom-action gate now says listing exists and invocation is still open.
  • .changeset/desktop-driver-accessibility-actions.md: patch for @fluentui-react-native/desktop-driver.

Fabric and Paper publish different actions (react-native-macos 0.81.9). Fabric adds every declared action by name (RCTViewComponentView.mm, accessibilityCustomActions). Paper adds only declarations that have a label, and names them by it (RCTView.m). A Paper component that declares Toggle or Select without a label therefore publishes nothing to AX. That is a component gap the driver now reports accurately; this PR does not change any component.

Related: #4343 (not closed by this PR).

Verification

Automated, run on macOS on this branch, which was cut from main at b4ad0e8f3. main has since gained 15 commits that do not touch the Desktop Driver, and git merge-tree shows a clean merge. The package checks ran at 1742bda2a; the tip commit only renumbers a documentation claim ID, and the root checks ran at the tip.

In packages/agentic/desktop-driver:

  • yarn format: passed.
  • yarn lint: passed.
  • yarn build: passed.
  • yarn test: passed (74 passed; 3 opt-in native helper tests skipped, as by default). The new server test fails when the route is disabled.
  • FURN_NATIVE_DRIVER_TEST=1 yarn test --runInBand: passed (75 passed, 2 skipped). It builds the real macOS helper and runs its --self-test, which also passed 123/123 when run directly.

At the repository root:

  • yarn format:check: passed.
  • yarn check-agent-instructions: passed.
  • yarn lage test-links: passed.
  • yarn change:check: passed.
  • yarn clean-lage-cache && yarn lage buildci: passed (176/176 tasks).

Manual native check, macOS Fabric and Paper, apps/storybook:

  • yarn storybook bundle --macos, yarn storybook prep --macos, yarn storybook build --macos, then yarn storybook driver --macos and yarn storybook run --macos; the same sequence with --paper after switching renderers.
  • A temporary probe test (not committed) read getProperty('accessibilityActions') on the Storybook Switch and Tab elements. A temporary Switch declaring { name: 'Toggle', label: 'Flip it' } was also probed on Paper. Those edits were reverted.
Renderer Element accessibilityActions
Fabric Switch (both instances) ["Toggle"]
Fabric Tab (both instances) ["Select"]
Paper Switch and Tab as authored []
Paper Switch declaring { name: 'Toggle', label: 'Flip it' } ["Flip it"]

Not run: the Windows and Win32 native lanes, where the property reports unsupported operation. Disabled elements and the Radio, MenuItem, and Checkbox stories were not probed.

Before After
getProperty('accessibilityActions') returned null, so AX custom actions could not be listed. Fabric lists ["Toggle"] (Switch) and ["Select"] (Tab). Paper lists [] unless the declaration has a label. Windows and Win32 report unsupported operation.

Pull request checklist

This PR has considered (when applicable):

  • Automated Tests
  • Documentation and examples
  • Keyboard Accessibility
  • Voiceover
  • Internationalization and Right-to-left Layouts

Add an accessibilityActions element property that returns the names of the actions an element publishes in the native accessibility tree, gated by the accessibilityActions feature. The macOS helper reads AXUIElementCopyActionNames and reduces AppKit's custom-action strings (Name, Target and Selector fields) to the authored name so no process-local handle crosses the boundary. React Native macOS Fabric lists Toggle and Select by name; Paper lists only actions that declare a label, by that label. Hosts without the feature reject with unsupported operation.

Listing only. Invoking custom actions remains out of scope.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
C-A11Y-006 is already taken by the react-native-macos state projection claim on user/jasonvmo/validate-sriramas-tests, which this branch does not contain. Using the next free ID avoids two different claims sharing one ID when the branches merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Bundle size report

Tree-shaken, minified production esbuild bundles with React and React Native runtimes externalized.

Scenario Modules-Mac (Δ) Modules-Win (Δ) Size-Mac (Δ) Size-Win (Δ)
components-button 65 (+1) 66 (+1) 51.57k (+944b) 51.50k (+944b)
components-catalog 214 (+0) 215 (+0) 206.83k (+1.35k) 206.76k (+1.35k)
design-theme-state 34 (+0) 35 (+0) 30.15k (+0b) 30.08k (+0b)
design-themed-root 38 (+0) 39 (+0) 32.12k (+0b) 32.04k (+0b)
design-color-lib 6 (+0) 6 (+0) 6.33k (+0b) 6.33k (+0b)
design-theme-state-legacy-source 35 (+0) 36 (+0) 31.81k (+0b) 31.73k (+0b)
design-legacy-theme-from-flex 58 (+0) 59 (+0) 97.77k (+0b) 97.63k (+0b)

The job is advisory: size changes are reported but do not fail the pull request. Bundle or analysis errors still fail.


View bundle-size workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant