Repository navigation
Desktop Driver: list published accessibility actions (macOS AX Select/Toggle) - #4426
Open
Jason Morse (JasonVMo) wants to merge 2 commits into
Open
Jason Morse (JasonVMo) wants to merge 2 commits into
Jason Morse (JasonVMo) wants to merge 2 commits into
Conversation
Add an accessibilityActions element property that returns the names of the actions an element publishes in the native accessibility tree, gated by the accessibilityActions feature. The macOS helper reads AXUIElementCopyActionNames and reduces AppKit's custom-action strings (Name, Target and Selector fields) to the authored name so no process-local handle crosses the boundary. React Native macOS Fabric lists Toggle and Select by name; Paper lists only actions that declare a label, by that label. Hosts without the feature reject with unsupported operation. Listing only. Invoking custom actions remains out of scope. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
C-A11Y-006 is already taken by the react-native-macos state projection claim on user/jasonvmo/validate-sriramas-tests, which this branch does not contain. Using the next free ID avoids two different claims sharing one ID when the branches merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Bundle size reportTree-shaken, minified production esbuild bundles with React and React Native runtimes externalized.
The job is advisory: size changes are reported but do not fail the pull request. Bundle or analysis errors still fail. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Platforms Impacted
Desktop Driver tooling only: the platform-neutral server and host contract, the macOS native helper, and documentation. No component source changes. The Windows and Win32 helpers are unchanged and report the new property as unsupported.
Description of changes
The Desktop Driver can now list the accessibility actions an element publishes. A new element property,
accessibilityActions(element.getProperty('accessibilityActions'); theattributeroute serves the same list), returns the action names in provider order. Before this, tests had no way to see whether a component published the AXToggleorSelectcustom actions: the attribute and property routes returnednullfor names they did not know. On macOS the helper readsAXUIElementCopyActionNames. AppKit lists a custom action asName:<name>\nTarget:<address>\nSelector:<selector>, so the helper returns only the authored name and no process-local address or selector crosses the boundary. Standard actions such asAXPresspass through unchanged.Design:
windowEvents: an optionalDesktopHost.accessibilityActions?(), afeatures.accessibilityActionsflag that tests see asfurn:features, and a hello feature string. The wire protocol stays at 1.0.unsupported operation. The Windows helper does not implement the command and rejects it asunsupported-operation(native/windows/src/driver.cpp), which the server surfaces the same way.macOS helper (
packages/agentic/desktop-driver/native/macos/Sources/DesktopDriverHost):Support.swift:accessibilityActionNamereduces the AppKit string to the authored name. It strips from the last\nTarget:, so a name that itself contains that text is kept whole.Accessibility.swift:AccessibilityEngine.accessibilityActionsrequires Accessibility trust, honors cancellation, validates that the element is still live, and maps the names.Driver.swift,Host.swift: dispatch the command and advertise it in the hello features and the probe'sfeatures.SelfTest.swift: parser cases (Toggle and Select from raw strings, spaces kept, an authored\nTarget:kept, name only,AXPress, empty) and a hello feature check.Node host and server (
packages/agentic/desktop-driver/src):host/types.ts,hosts/native/NativeDesktopHost.ts,hosts/fake/FakeDesktopHost.ts: the optional method and feature flag. The fake host takes anaccessibilityActionslist per element and throwsHostUnsupportedErrorunless the feature is enabled.server/createDesktopDriverServer.ts:GETofproperty/accessibilityActionsorattribute/accessibilityActionsreturns the host's list, orunsupported operationwhen the host has no such method.server/desktopDriver.test.ts: covers property and attribute reads, an element with no actions, and both unsupported paths (feature off, method absent).Documentation and release:
native/PROTOCOL.md,references/protocol.md,native/macos/README.md,storybook-desktop/references/story-tests.md: the command, feature negotiation, and how tests should guard onfurn:features.references/agents/accessibility/action-transport-evidence.md: new claim C-A11Y-007 with the raw format and the Fabric and Paper results below. It is numbered 007 because 006 is reserved by pending macOS selection-test work that is not published yet.components/references/remaining-work.md: the macOS AX custom-action gate now says listing exists and invocation is still open..changeset/desktop-driver-accessibility-actions.md: patch for@fluentui-react-native/desktop-driver.Fabric and Paper publish different actions (
react-native-macos0.81.9). Fabric adds every declared action byname(RCTViewComponentView.mm,accessibilityCustomActions). Paper adds only declarations that have alabel, and names them by it (RCTView.m). A Paper component that declaresToggleorSelectwithout alabeltherefore publishes nothing to AX. That is a component gap the driver now reports accurately; this PR does not change any component.Related: #4343 (not closed by this PR).
Verification
Automated, run on macOS on this branch, which was cut from
mainatb4ad0e8f3.mainhas since gained 15 commits that do not touch the Desktop Driver, andgit merge-treeshows a clean merge. The package checks ran at1742bda2a; the tip commit only renumbers a documentation claim ID, and the root checks ran at the tip.In
packages/agentic/desktop-driver:yarn format: passed.yarn lint: passed.yarn build: passed.yarn test: passed (74 passed; 3 opt-in native helper tests skipped, as by default). The new server test fails when the route is disabled.FURN_NATIVE_DRIVER_TEST=1 yarn test --runInBand: passed (75 passed, 2 skipped). It builds the real macOS helper and runs its--self-test, which also passed 123/123 when run directly.At the repository root:
yarn format:check: passed.yarn check-agent-instructions: passed.yarn lage test-links: passed.yarn change:check: passed.yarn clean-lage-cache && yarn lage buildci: passed (176/176 tasks).Manual native check, macOS Fabric and Paper,
apps/storybook:yarn storybook bundle --macos,yarn storybook prep --macos,yarn storybook build --macos, thenyarn storybook driver --macosandyarn storybook run --macos; the same sequence with--paperafter switching renderers.getProperty('accessibilityActions')on the Storybook Switch and Tab elements. A temporary Switch declaring{ name: 'Toggle', label: 'Flip it' }was also probed on Paper. Those edits were reverted.accessibilityActions["Toggle"]["Select"][]{ name: 'Toggle', label: 'Flip it' }["Flip it"]Not run: the Windows and Win32 native lanes, where the property reports
unsupported operation. Disabled elements and the Radio, MenuItem, and Checkbox stories were not probed.getProperty('accessibilityActions')returnednull, so AX custom actions could not be listed.["Toggle"](Switch) and["Select"](Tab). Paper lists[]unless the declaration has alabel. Windows and Win32 reportunsupported operation.Pull request checklist
This PR has considered (when applicable):