[srividyaramas]

Platforms Impacted

• iOS
• macOS
• win32 (Office)
• windows
• android

Description of changes

Standalone Checkbox readiness-assessor pilot, ready for review, on its own branch. This is advisory evidence tooling, not a component behavior change, a new control test framework, or automatic maturity promotion.

Start the design review with plan.md. The implementation guide is scripts/CONTROL-ASSESSMENT.md.

• Reuse the existing component Jest invocation and existing Win32/Paper Storybook native reports. Collect attempt-specific artifacts and a job summary; the assessment job does not rerun tests.
• Bind evidence to source, run/attempt, report hashes, target and reviewed mappings. Preserve package-wide failures, skips, stale mappings, missing evidence and unmapped cases. Emit requirement-level JSON/Markdown without assigning a lifecycle stage.
• Add an optional trusted, default-branch publisher for only the managed Checkbox SPEC assessment block and generated summary. Publication remains opt-in: this PR does not configure an App, credentials or repository variables. The plan includes the exact five-step maintainer activation checklist.
• Document future mapping maintenance as one orchestrating agent with separate mapping and deterministic assessment modules. That automation is not implemented here; mapping proposals will require owner review and ordinary source CI.

Bounded initial coverage: 42 reviewed runtime cases, out of 67 current Checkbox runtime cases; the other 25 are reported as unmapped. One existing native callback is mapped: toggles native checked state in both directions. Hosted Win32 Storybook disables the physical-input capabilities it needs, so that callback remains skipped there. Compiler-result ingestion, complete native accessibility/focus/layout coverage, visual qualification, readiness policy and other controls are follow-ups.

No historical local Checkbox assessment, generated report or SPEC assessment snapshot is included. This PR adds assessor tests, not new Checkbox behavior tests. Collection can run in this PR's normal CI; automated branch publication needs reviewed tooling on main and separate maintainer setup.

Verification

Assessor fix commit: 6ca79067383ec3db75146705cb858bc38ee7bac1. The three findings in comment 6509427208 are addressed: valid partially skipped Jest suites, preserved/disqualified unit retries, and fingerprinting the reviewed assertion-bearing native helper. All 21 hosted checks passed for that commit, including the required PR check, CodeQL, all Storybook lanes and the assessor. Latest head feef01779965043d88767e37e1ba1848914385a0 only expands plan.md; all 21 fresh hosted checks also passed for this latest head. Conditional squash auto-merge remains enabled at the author's request; an approving review is still required.

The previous head 11e8e4fa351c5a178f9985fd199ad344aaa091ae passed all 21 hosted checks after the targeted unchanged-head macOS retry. Those results are historical, not validation of the new fix commit.

Check	Result
Immutable dependency restore	Passed; lockfile unchanged.
Scripts build, lint and formatting	Passed for the new evidence fixes.
Full local scripts tests	237 passed across 18 suites, with no failures or skips.
Focused feedback regressions	36 passed, covering real partially skipped suite serialization, invocation-only/reason-only retries, unrelated package retries, retained final failures, malformed metadata, helper-only fingerprint drift and normalized helper line endings. Earlier Markdown-escaping and executable-mode regressions remain in the full passing suite.
Existing component tests (retained historical run)	812 passed across 66 suites; 8 snapshots passed. Not rerun as part of these assessor-only fixes.
Current-parser replay of the actual full-package Jest JSON	Passed: 812 package passes, 67 Checkbox passes, 25 unmapped cases and zero observed retries. Without an execution sidecar, provenance correctly remains unverified and readiness insufficient.
Workflow Actionlint and instruction validation	Previously passed; workflow and instruction files are unchanged by this fix. Current diff checks passed.
Assessor-fix hosted checks (6ca790673)	All 21 passed, including JS/Desktop Driver tests, CodeQL, macOS Storybook (Fabric and Paper), Windows/Win32 Storybook, source validation, the assessor and the required PR check. See validated workflow.
Latest documentation-only head (feef01779)	Added explicit future-work checklists for affected-control/dependency selection, Windows/Fabric assessment and one generic multi-control job with optional later matrix scaling. Repository formatting and diff checks passed; all 21 fresh hosted checks passed, including the assessor, every Storybook lane, CodeQL and required PR. See latest PR workflow. No implementation or workflow changed.
Verified assessment artifact (6ca790673)	Assess Checkbox evidence passed and uploaded the report. Health: complete; outcome: insufficient-evidence; native/unit provenance matched. Checkbox runtime: 67 passes, 25 newer cases unmapped. Physical-input skips and remaining requirement gaps are preserved; no maturity stage is assigned. Independent replay of the original bundle reproduced assessment.json, ASSESSMENT.md and proposed SPEC.md exactly, with verified evidence hashes and no test rerun. This artifact describes the assessor-fix execution, not the later documentation-only head.
Previous-head hosted checks	All 21 passed at 11e8e4fa3, including JS/Desktop Driver tests, CodeQL, Win32 Storybook and advisory evidence collection. These results do not replace new-head CI. The original CodeQL conversation was resolved only after its fixed instance was verified.
Previous-head macOS retry	Passed without source or deadline changes in attempt 2: all 209 stories rendered; canonical run recorded 239 passed and 80 skipped, zero failures, incomplete work, retries or flaky cases. The initial five-minute first-story timeout did not recur; root cause remains unestablished. Skipped physical input is not qualification.
Latest local clean-cache yarn lage buildci	Not green locally: 108 tasks succeeded before one unchanged Desktop Driver CLI test timed out waiting for its loopback /status endpoint. Fail-fast left remaining work incomplete, not proven failed. The separate full 237-test scripts run passed, as did hosted JS/Desktop Driver tests for 6ca790673. Hosted success does not erase the local limitation.
Review and merge	Reviews requested from JasonVMo, Khalef1 and mganandraj. Squash auto-merge is enabled, but the effective rules still require the PR check, at least one approving review and resolved review conversations. Requesting three reviewers does not require all three approvals. Optional checks remain monitored but are not automatically merge blockers.
Optional same-PR publication	Not activated or trialled. Requires the separate maintainer setup in plan.md.

Earlier local Storybook tooling retries also had subprocess/startup timeouts. No StableText source, story selection, readiness checks or test deadlines were changed to suppress failures. Hosted success does not retroactively make failed local runs green.

These validation limitations are disclosed, not waived. Before: no requirement-level assessment of these existing reports. After: an advisory evidence report with explicit gaps; no claim that Checkbox is Production-ready or physically input-qualified. Squash auto-merge waits for the effective branch protections; no review or CI protections are bypassed.

Pull request checklist

This PR has considered (when applicable):

• Automated Tests
• Documentation and examples
• Keyboard Accessibility
• Voiceover
• Internationalization and Right-to-left Layouts

Native keyboard/accessibility qualification is a recorded coverage gap, not a completed checklist item.

[github-actions]

Bundle size report

Tree-shaken, minified production esbuild bundles with React and React Native runtimes externalized.

Scenario	Modules-Mac (Δ)	Modules-Win (Δ)	Size-Mac (Δ)	Size-Win (Δ)
components-button	65 (+1)	66 (+1)	51.57k (+944b)	51.50k (+944b)
components-catalog	214 (+0)	215 (+0)	206.83k (+1.35k)	206.76k (+1.35k)
design-theme-state	34 (+0)	35 (+0)	30.15k (+0b)	30.08k (+0b)
design-themed-root	38 (+0)	39 (+0)	32.12k (+0b)	32.04k (+0b)
design-color-lib	6 (+0)	6 (+0)	6.33k (+0b)	6.33k (+0b)
design-theme-state-legacy-source	35 (+0)	36 (+0)	31.81k (+0b)	31.73k (+0b)
design-legacy-theme-from-flex	58 (+0)	59 (+0)	97.77k (+0b)	97.63k (+0b)

The job is advisory: size changes are reported but do not fail the pull request. Bundle or analysis errors still fail.

---

View bundle-size workflow run

[srividyaramas]

Pushed test-only feedback fixes in 11e8e4f.

The JS CI failure came from the executable-mode fixture: its commit helper staged the file a second time, undoing the explicit index mode on Linux. The fixture now aligns physical and indexed executable mode, commits without restaging, and verifies that the committed entry is actually 100755 before testing digest invalidation. The CodeQL test assertion now covers lowercase, uppercase and mixed-case tags. No production, provenance, source-digest or publisher validation logic was relaxed.

Validation: scripts build/lint, formatting, focused regressions and the full 233-test/18-suite scripts run passed with no failures or skips. The required clean-cache local aggregate is still NOT green: 105 tasks succeeded before an unchanged Desktop Driver CLI test timed out waiting for loopback /status, leaving other tasks incomplete. No unrelated source or deadlines were changed. Fresh hosted CI is now pending; auto-merge remains subject to required CI, approval and conversation-resolution rules.

[srividyaramas]

Review findings at 11e8e4fa351c5a178f9985fd199ad344aaa091ae (ordinary comment because this is my own PR; not a self-approval):

[P2] Preserve valid partially skipped Jest suites

scripts/src/tasks/controlAssessmentUnit.ts:185-188

Jest 29.7 serializes a nonfailed suite containing pending tests as "focused", but this parser rejects that status. Because every package suite is parsed, even an unrelated component's skipped test can prevent Checkbox assessment generation rather than yielding advisory evidence and gaps. Accept/count this valid suite status while preserving the individual skips, as required by plan.md, Design steps 4-5. The existing skip fixture changes assertion statuses but leaves its synthetic enclosing suite "passed", so it does not exercise the real serialization.

Verification: executed the exact upstream Jest v29.7.0 formatTestResults.ts formatter; it produced "focused", which parseCheckboxJest rejected with Unsupported Jest suite status. Native tests were not run.

[P2] Preserve unit retry history before qualifying passes

scripts/src/tasks/controlAssessmentUnit.ts:205-210

This conversion discards Jest's invocations and retryReasons, making a retry-success result indistinguishable from a clean first-attempt pass when qualifying mapped requirements. invocations > 1 is evidence of repeated execution even if retry-reason logging was not enabled. Preserve retry/flaky observations and prevent silently treating them as clean qualifying evidence, as required by plan.md, Design step 4. Add a retry-success case to the existing unit-evidence tests.

Verification: a synthetic bound report with invocations: 2 and a prior failure in retryReasons yielded unitEvidence.outcome: "passed", a mapped evidenceOutcome: "passed" and partial CBX-001 coverage. This used manually populated retry metadata, not an actual Jest retry run.

[P2] Include the assertion-bearing native helper in the reviewed fingerprint

scripts/src/tasks/controlAssessment.ts:16-18

The mapped native callback delegates its checked-state assertions to packages/agentic/components/src/common/desktopFocus.wdio.ts, but the reviewed mapping fingerprint excludes that helper. Weakening or redirecting expectNativeState therefore leaves mappingCurrent: true, allowing fresh evidence from changed assertions to retain the old binary-transition coverage interpretation. Source-tree provenance authenticates the changed source; it does not establish that its assertions still match the reviewed mapping. Include the helper in the reviewed fingerprint, as required by plan.md, Design step 2, and cover a helper-only change in the fingerprint regression.

Verification: source inspection and an executed fingerprint probe showed the reader never requested the helper, and changing its supplied content left the digest/current flag unchanged. No native execution was performed.

[srividyaramas]

Addressed the three evidence-review findings in 6ca79067383ec3db75146705cb858bc38ee7bac1:

• Accept Jest 29.7's focused serialization for nonfailed partially skipped suites, including unrelated package suites. Individual pending assertions remain skipped, not passing coverage.
• Preserve optional invocations, retryReasons, retry counts and retry-success observations. Invocation-only and reason-only retry evidence both prevent the enclosing package's recorded passes from qualifying; actual final failures retain failure precedence.
• Include the assertion-bearing desktopFocus.wdio.ts helper in the reviewed native mapping fingerprint. The default callback and delegated checked-state assertions were inspected before deliberately updating that fingerprint. Helper-only drift now invalidates the mapping, and helper line endings are normalized consistently.

Validation: 237 scripts tests across 18 suites passed, with no failures or skips; 36 focused regressions, scripts build, lint, formatting and diff checks passed. Replaying the retained actual Jest JSON preserved 812 package passes, 67 Checkbox passes, 25 unmapped cases and zero retries; without a sidecar, provenance correctly remained unverified and readiness insufficient.

The required clean-cache local aggregate is not green: 108 tasks succeeded before an unchanged Desktop Driver CLI test timed out waiting for its loopback /status endpoint; remaining work was incomplete. Its source is unchanged from main. No deadlines, selection, evidence checks, or component behavior were relaxed. Current-head hosted CI is pending; auto-merge remains paused.