Skip to content

build(windows): default to static CRT and pinned Z3 packages - #5654

Open
shailendra-nv wants to merge 15 commits into
windowsfrom
shailendra-nv/windows-static-runtime
Open

shailendra-nv wants to merge 15 commits into
windowsfrom
shailendra-nv/windows-static-runtime

Conversation

@shailendra-nv

@shailendra-nv shailendra-nv commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Build Windows packages with static CRT in all four OpenShell executables and static Z3 5.1.0 in the gateway by default. Normal builds download and cache digest-pinned static libraries from OpenShell's Z3 GHCR package; source builds occur only when requested or the package is unavailable.

Related Issue

Closes #5653

The author has verified MAINTAIN access to NVIDIA/OpenShell; the issue does not have an acceptance label.

Changes

  • Publish x64 and ARM64 Release /MT Z3 packages through an explicit publish_z3 mode in the existing Windows MSVC workflow. This dependency build is separate from ordinary CI.
  • Preserve Z3 5.1.0 and its source checksum. Pin only the Z3 package producer to MSVC toolset 14.44 (compiler 19.44); the published packages used 14.44.35207. Disable LTO, audit solver PE imports, and check SAT/UNSAT before publication. Previously the repo had no explicit MSVC version pin, and normal OpenShell builds still select their installed MSVC toolchain. This producer baseline avoids newer C++ runtime helper references that N1x's 14.44 libraries cannot resolve.
  • Pin OCI manifests, ZIPs, and build manifests in tasks/windows-z3.json. Native PowerShell downloads packages without Docker, ORAS, or GitHub credentials, verifies their files/version/architecture/runtime/compiler baseline, and caches one extracted copy outside Cargo pruning.
  • Resolve each Cargo target independently. Use supplied static libraries when requested, and preserve optimized Ninja source builds as an explicit or unavailable-package fallback. Integrity failures stop the build.
  • Make static runtime the Windows task and workflow default. Preserve the PR lint/test workload and cache the solver separately, avoiding recurring C++ solver compilation.
  • Keep OPENSHELL_WINDOWS_STATIC_RUNTIME=0 as the explicit legacy shared mode, including shared DLL staging and -Z3DllPath compatibility. An inherited shared override in default static mode produces a migration warning and selects the pinned static package.
  • Audit architecture and normal/delayed Z3/redistributable CRT imports in all four release executables; remove stale staged Z3 DLLs. Update Windows guidance and the contributor skill.

The dependency archives are build inputs, not files added to distributed OpenShell ZIPs. Updating embedded CRT or Z3 code requires rebuilding the affected dependency package/executables. Existing MXC enforcement gaps and the separate executor's prerequisites remain as documented.

Testing

  • Repository pre-commit checks and Windows workflow actionlint.
  • Native x64/ARM64 package build and GHCR publication: pinned MSVC 14.44, static CRT import audits, solver SAT/UNSAT checks, and version verification.
  • Anonymous public downloads of both published archives; OCI, archive, manifest, and every packaged file checksum verified.
  • N1x-Test native PowerShell cold downloads/verification: ARM64 8.54 seconds, x64 7.69 seconds, no source compilation. Cached downloads require no registry request.
  • N1x MSVC 14.44 whole-archive C API link and SAT/UNSAT checks for both packages; PE architecture/static-runtime audits pass. ARM64 runs natively; the x64 executable smoke runs under emulation.
  • N1x: 36 dependency/runtime and 14 OCI cache, integrity, compiler-pin, and Cargo/Nextest routing regression checks.
  • Final default/prebuilt N1x ARM64 check, release build, fixtures, unsupported-driver matrix, and full Rust suite: 6,663 passed, 43 existing skips, no failures or retries. All four release PE audits passed.
  • N1x real MXC integration: Cargo reports 14 passed; one test internally skips the unavailable IsolationSession backend. All seven default real E2E scenarios plus https-ca passed with no E2E skips. Existing documented MXC enforcement gaps remain.
  • Exact external CI ZIP packager, extraction into empty directories, and all executable startup/version smokes passed. Same-toolchain ARM64 comparison against 28de5674: CLI ZIP 10,039,249 → 10,131,535 bytes (+0.92%); gateway bundle 44,568,145 → 44,541,442 bytes (−0.06%); combined +65,583 bytes (+0.12%). Total extracted payload, including the baseline Z3 DLL, decreased by 458,752 bytes (−0.34%).
  • Normal-default GitHub Windows x64/ARM64 lint and tests: 6,663 passed, 43 skips on each architecture, consuming the published packages.
  • Final GitHub release builds and all four executable PE/static CRT/Z3 import audits passed on both native architectures (same normal-default run), reusing cached Z3 packages.
  • Separate PR lint/test run: both native architectures passed lint and 6,663 tests each, 43 existing skips each, no flaky retries.
  • Final Branch Checks: all 26 non-skipped jobs passed; compatibility review completed with no unaddressed breaking changes. Code review completed for 79cb5d67 with no findings.
  • Final /windows-ci passed for 79cb5d67: check, release build, all four PE/static CRT/Z3 audits, 6,663 tests passed / 43 existing skips / no flaky retries, unsupported-driver contracts and all four feature combinations, plus both ZIP extraction/startup smokes. Build and GitHub-report jobs succeeded. The runner ignored its inherited shared override and downloaded/cached the pinned static Z3 5.1.0 package; no Z3 source compilation.

The earlier security-review retry returned “Codex review is not enabled for this repo”. No actionable security finding was returned.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Windows documentation and contributor skill updated

Signed-off-by: shailendras <shailendras@nvidia.com>
@shailendra-nv shailendra-nv added the test:windows Run native Windows x64 and ARM64 lint/tests on PR mirrors label Nov 17, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/windows-ci

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/ok to test bb8c4f9

@sam-india-007

sam-india-007 commented Nov 17, 2026 •

Copy link
Copy Markdown

Windows ARM64 CI: ✅ passed

Commit 79cb5d6, requested by @shailendra-nv, 1h 20m, full log

Step Result Duration
if ($env:PR_NUMBER) { Import-Module .\scripts\pr-ci\OpenShellPrCi.psm1; Assert-PrCiVariables -PrN... ✅ passed 0s
& .\scripts\setup-windows-arm64-runner.ps1 -VerifyOnly ✅ passed 1s
$env:PATH = "$(Split-Path -Parent $env:RUSTC_WRAPPER);$env:PATH"; $found = (Get-Command sccache -... ✅ passed 0s
New-Item -ItemType Directory -Force (Split-Path -Parent $env:SCCACHE_ERROR_LOG) | Out-Null; & $en... ✅ passed 2s
& $env:RUSTC_WRAPPER --zero-stats ✅ passed 0s
& .\scripts\checkout-openshell.ps1 -Ref $env:OPENSHELL_REF ✅ passed 4s
& .\scripts\set-openshell-version.ps1 -Version $env:OPENSHELL_VERSION ✅ passed 0s
Set-Location $env:OPENSHELL_CHECKOUT_DIR ✅ passed 0s
mise run --skip-tools windows:check:arm64 ✅ passed 6m 40s
mise run --skip-tools windows:build:arm64 ✅ passed 22m 46s
mise run --skip-tools windows:test:arm64 ✅ passed 24m 28s
mise run --skip-tools windows:test:unsupported:arm64 ✅ passed 24m 50s
mise run --skip-tools windows:artifacts ✅ passed 2s
& "$env:CI_PROJECT_DIR\scripts\package-windows-release.ps1" ✅ passed 48s

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

Unaddressed breaking changes

Windows remote-test invocations using -Z3DllPath now fail because the parameter was removed. Windows builds using a previously supported dynamic Z3 override now fail static-runtime verification.

1. cli — NOT ACKED

test-mxc-remote.ps1 -Z3DllPath was removed; the script no longer accepts that documented parameter.

  • Impact: Existing remote MXC test automation passing -Z3DllPath fails PowerShell parameter binding before testing starts.
  • Action: Retain -Z3DllPath as a deprecated compatibility parameter or restore its behavior until a break-allowing train provides migration guidance.
  • PR: build(windows): default to static CRT and pinned Z3 packages #5654

2. configuration — NOT ACKED

Z3_LIBRARY_PATH_OVERRIDE previously accepted a target-compatible Windows Z3 library but now requires a static libz3.lib built with /MT.

  • Impact: Existing windows:build:* configurations pointing to the documented prebuilt Z3 DLL import library fail the new static dependency check.
  • Action: Continue accepting the prior dynamic Z3 override or move the restriction to a break-allowing train with explicit static-library migration steps.
  • PR: build(windows): default to static CRT and pinned Z3 packages #5654
Review details
  • Execution: complete
  • Candidate: PR #5654 (8ee45773dfdae1893f95fd272d0c074d641e43b1)
  • Baseline: 28de5674feb1702290f746a70c3e6aba884a6f85 (28de5674feb1702290f746a70c3e6aba884a6f85)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

Signed-off-by: shailendras <shailendras@nvidia.com>
@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/ok to test ed4d29d

@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/windows-ci

Signed-off-by: shailendras <shailendras@nvidia.com>
@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/ok to test 7200fce

@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/windows-ci

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

Unaddressed breaking changes

Existing Windows MXC automation using -Z3DllPath now fails during parameter binding. Documented dynamic Z3 overrides are also ignored or rejected by the Windows build workflow.

1. cli — NOT ACKED

test-mxc-remote.ps1 -Z3DllPath was removed; the script no longer accepts the documented parameter.

2. configuration — NOT ACKED

Z3_LIBRARY_PATH_OVERRIDE previously accepted a target-compatible Windows Z3 import library but now requires static libz3.lib built with /MT.

  • Impact: Existing windows:build:* configurations using the documented dynamic Z3 override are ignored in favor of a source build or fail static-runtime verification.
  • Action: Continue accepting the previous dynamic Z3 override or defer the static-only requirement to a break-allowing train with explicit rebuild and configuration migration steps.
  • PR: build(windows): default to static CRT and pinned Z3 packages #5654
Review details
  • Execution: complete
  • Candidate: PR #5654 (eba95db233746c97309bc1077cfb2a39a4d8399f)
  • Baseline: 28de5674feb1702290f746a70c3e6aba884a6f85 (28de5674feb1702290f746a70c3e6aba884a6f85)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@shailendra-nv shailendra-nv linked an issue Nov 17, 2026 that may be closed by this pull request
5 tasks
Signed-off-by: shailendras <shailendras@nvidia.com>
@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/ok to test 27de545

@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/windows-ci

@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

Addressed the compatibility review in 27de545:

  • Restored -Z3DllPath as a deprecated, ignored parameter. Existing remote-test command lines still bind, with a native PowerShell regression covering the real parameter block (10 remote-helper cases pass).
  • Acknowledged the intentional build-configuration change in the PR description and added explicit migration settings to CONTRIBUTING.md. Windows packages now require static CRT/Z3 linkage. Documented prebuilt overrides containing libz3.dll remain accepted as inputs and fall back to checksum-pinned source with a warning. Offline builds can provide Z3 5.1.0 source; custom libraries can be rebuilt with static linkage and /MT. Import libraries without an adjacent DLL fail the release dependency check, as documented.

The static-only package contract implements the requested removal of the Z3 DLL and Visual C++ redistributable requirements. The real MXC suite passed all seven default scenarios plus HTTPS CA, with zero skips, before this argument/documentation-only follow-up. Windows CI is being refreshed for the new head.

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

Unaddressed breaking changes

The documented Windows Z3 override no longer selects shared Z3, so existing offline or CMake-free build automation can fail. SDKs, runtime APIs, CLI, Helm, policy, and persisted state are unchanged.

1. configuration — ACKED

Z3_LIBRARY_PATH_OVERRIDE changed from selecting a target-compatible shared Z3 installation to ignoring directories containing libz3.dll and building pinned static source.

  • Impact: Existing Windows builds using the documented shared-Z3 override can now require CMake and network access or fail when run offline.
  • Action: Restore shared-Z3 override support for this non-breaking train, or defer the static-only requirement to a train that permits breaks while retaining the supplied migration guidance.
  • PR: build(windows): default to static CRT and pinned Z3 packages #5654
Review details
  • Execution: complete
  • Candidate: PR #5654 (629b92ee364bc3c7b654c2d33718d56ebece0dd2)
  • Baseline: 28de5674feb1702290f746a70c3e6aba884a6f85 (28de5674feb1702290f746a70c3e6aba884a6f85)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

The latest review confirms that -Z3DllPath compatibility is restored and marks the static-Z3 migration ACKED. The remaining finding concerns the repository's Windows package-construction workflow, rather than installed OpenShell CLI/SDK behavior or user-authored gateway configuration.

Baseline context:

  • The Windows build lane is a contributor workflow in a skill explicitly marked metadata.internal: true: https://github.com/NVIDIA/OpenShell/blob/28de5674feb1702290f746a70c3e6aba884a6f85/.agents/skills/build-openshell-mxc-windows/SKILL.md
  • That baseline already lists Visual C++ CMake tools as a Windows build prerequisite. The default prebuilt-Z3 path already requires a first-use download; the affected special case is custom/offline shared-Z3 package construction.
  • The raw Cargo Z3 feature flags and z3-sys library override support are unchanged. This change makes the maintainer windows:* package-validation lane produce and verify the requested static artifacts.

The package-construction change is intentional: embedding CRT and Z3 removes the destination machine's redistributable and Z3-DLL requirements. CONTRIBUTING.md now documents local-source/offline builds and rebuilding custom static libraries with /MT. This does not change an installed SDK, CLI, TOML, policy, Helm, or persisted-state contract. The explicit migration notice is retained for contributor automation affected by the new package requirements.

Signed-off-by: shailendras <shailendras@nvidia.com>
Signed-off-by: shailendras <shailendras@nvidia.com>
@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/ok to test 10b5a55

Signed-off-by: shailendras <shailendras@nvidia.com>
@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/ok to test 5a3ff56

@shailendra-nv

Copy link
Copy Markdown
Collaborator Author

/windows-ci

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

Unaddressed breaking changes

Windows Z3_LIBRARY_PATH_OVERRIDE no longer selects a documented shared Z3 library, so existing custom and offline build automation can fail or silently build pinned source instead. The PR explicitly acknowledges the change and provides migration guidance, but this train allows no Stable breaks.

1. configuration — ACKED

Z3_LIBRARY_PATH_OVERRIDE previously selected a shared libz3.dll build, but shared overrides are now ignored and replaced with a static source build.

  • Impact: Windows automation using a custom shared Z3 may require newly installed CMake and C++ tools, lose its selected library, or fail in offline environments.
  • Action: Preserve shared Z3_LIBRARY_PATH_OVERRIDE support on this no-break train, or defer the static-only requirement to a train that permits the documented migration.
  • PR: build(windows): default to static CRT and pinned Z3 packages #5654
Review details
  • Execution: complete
  • Candidate: PR #5654 (d81b5d8661525fc762394b1b4cddd17df99cfc2f)
  • Baseline: 28de5674feb1702290f746a70c3e6aba884a6f85 (28de5674feb1702290f746a70c3e6aba884a6f85)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@github-actions

Copy link
Copy Markdown

Compatibility review (advisory)

Unaddressed breaking changes

Windows windows:* builds no longer accept documented shared-Z3 overrides and may now require CMake, Ninja, a C++ compiler, and source access. The break is explicitly acknowledged and documented, but the supplied train permits no Stable compatibility breaks.

1. configuration — ACKED

Z3_LIBRARY_PATH_OVERRIDE previously selected a documented shared Z3 library for windows:* tasks, but shared overrides are now ignored and only static /MT libraries or pinned source builds are accepted.

  • Impact: Existing Windows build automation using libz3.dll may fail offline or on hosts without CMake, Ninja, and a C++ compiler.
  • Action: Preserve shared-Z3 override behavior for this train or defer the static-only requirement to a break-allowing train while retaining the supplied static-library and Z3_SRC_SOURCE_DIR migration guidance.
  • PR: build(windows): default to static CRT and pinned Z3 packages #5654
Review details
  • Execution: complete
  • Candidate: PR #5654 (9ae70c222e80186c523a6b35199a1a8f7f3d6bf2)
  • Baseline: 28de5674feb1702290f746a70c3e6aba884a6f85 (28de5674feb1702290f746a70c3e6aba884a6f85)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@shailendra-nv
shailendra-nv marked this pull request as ready for review November 17, 2026 23:18
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T05:12:49.395955Z 79cb5d6 Draft marked ready
🔒 Security Review ❌ Failed 2026-11-18T05:08:22.360795Z
We couldn’t complete this review. Comment @codex security review to try again.
79cb5d6 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@shailendra-nv
shailendra-nv marked this pull request as draft November 17, 2026 23:56
Signed-off-by: shailendras <shailendras@nvidia.com>
@shailendra-nv shailendra-nv changed the title build(windows): link static CRT and Z3 in Windows packages build(windows): add opt-in static CRT and Z3 packages Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test b61b07a

Copy link
Copy Markdown
Collaborator Author

/windows-ci

shailendra-nv commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Updated for the current implementation in 79cb5d6: Windows tasks and CI now default to static CRT/Z3, using the published, digest-pinned Z3 5.1.0 /MT packages at ghcr.io/nvidia/openshell/z3. This replaces the earlier opt-in-static revision. Normal CI downloads/caches the prebuilt library; source compilation occurs only when explicitly requested or the package is unavailable. The Z3 package producer selects MSVC 14.44 for compatibility with N1x's existing toolchain. Explicit legacy shared builds remain supported with OPENSHELL_WINDOWS_STATIC_RUNTIME=0, preserving shared library overrides, DLL staging, and -Z3DllPath. All requested default/prebuilt checks passed for this commit: N1x builds/full Rust suite/contracts/real MXC/E2E/ZIP smokes; native GitHub Windows x64/ARM64 lint, full suites, release builds and all four PE audits; the separate PR Windows checks; and /windows-ci. Each full Rust suite passed 6,663 tests with 43 existing skips. Measured combined ARM64 ZIP size changed by +0.12%.

Compiler clarification: "GitHub default" referred specifically to the selected Visual Studio 2026 toolchain on the hosted windows-11-arm runner: toolset 14.52 / compiler 19.52. N1x has Visual Studio 2022 with toolset 14.44.35207 / compiler 19.44.35228. The repo previously did not pin MSVC. The new 14.44 pin applies only to producing the reusable Z3 libraries; regular OpenShell builds keep installed-toolchain selection. A 19.52-built static library referenced newer C++ runtime helpers absent from N1x's link libraries. The replacement 14.44 packages have passed whole-archive link and SAT/UNSAT checks on N1x (ARM64 native; x64 under emulation).

Signed-off-by: shailendras <shailendras@nvidia.com>

Copy link
Copy Markdown
Collaborator Author

/ok to test 5d2febf

Copy link
Copy Markdown
Collaborator Author

/windows-ci

@shailendra-nv
shailendra-nv marked this pull request as ready for review November 18, 2026 02:35

Copy link
Copy Markdown
Collaborator Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex review is not enabled for this repo. Please contact the admins of this repo to enable Codex.

@shailendra-nv
shailendra-nv marked this pull request as draft November 18, 2026 03:10
Signed-off-by: shailendras <shailendras@nvidia.com>
Signed-off-by: shailendras <shailendras@nvidia.com>
Comment thread .github/workflows/windows-msvc.yml Fixed
Signed-off-by: shailendras <shailendras@nvidia.com>
@shailendra-nv shailendra-nv changed the title build(windows): add opt-in static CRT and Z3 packages build(windows): default to static CRT and pinned Z3 packages Nov 18, 2026
Signed-off-by: shailendras <shailendras@nvidia.com>
Signed-off-by: shailendras <shailendras@nvidia.com>

Copy link
Copy Markdown
Collaborator Author

/ok to test 79cb5d6

Copy link
Copy Markdown
Collaborator Author

/windows-ci

@shailendra-nv
shailendra-nv marked this pull request as ready for review November 18, 2026 05:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:windows Run native Windows x64 and ARM64 lint/tests on PR mirrors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(windows): distribute static CRT and Z3 packages

3 participants