sp x86_64: separate lane selection from vector-register ownership

[kaleb-himes]

Description

Pairs with: https://github.com/wolfSSL/scripts/pull/677

CPUID alone picks the lane; the chosen lane saves the vector registers. This is a CPU step, not a fallback.
A refused save returns the error. The other lane never runs.
Every SP mulmod entry point saves for itself, both lanes (the non-AVX2 table lookups are SSE2). P-1024 base lane excepted, it has none.
FIPS v7 reads the CPU features once at power on. cpuid_select_flags / cpuid_set_flag / cpuid_clear_flag do nothing in a FIPS v7 build; --enable-fips=dev and dev-no-post keep the upstream behavior. A microcode update that changes CPU features needs a reboot (Linux Documentation/arch/x86/microcode.rst, late loading), which re-detects them.
ML-KEM: a decoded public or private key no longer keeps the matrix cached (WOLFSSL_MLKEM_CACHE_A) from the key it replaces.
ML-KEM: a failed public key decode leaves no public key, and software decapsulation refuses a key without one before it decrypts anything (FIPS 203 Algorithm 18 re-encrypts with the public key). Upstream returns an implicit-rejection secret with success in that case.
The SP x86_64 white-box test now fails if a refused save still lets a call run, so a regression is caught in tree.
Behavior change: on AVX2-capable x86_64, SP RSA/ECC entry points can return the vector-register save error instead of silently running the non-AVX2 lane. Userspace never refuses the save.

Testing

36 test cells, all passing.

Builds and gates

FIPS-ready, FIPS v7, and FIPS v7 with the SP assembly lanes on, full test suite on each.
A build with AVX2 compiled out entirely.
The fail-closed dev build under strict warnings.
sp_x86_64.c reproduces byte for byte from the generator, with a control proving the pre-change file still differs.
Windows: the whole file sits inside one feature guard, the Windows FIPS settings enable none of it, and with those settings it compiles to zero symbols. Windows gets no code from this change.
Refused saves must fail, never switch lanes

Forced save failures into ECC P-256, P-384, P-521 and RSA 2048 and 3072, in a non-FIPS build and again against the real FIPS v7 module. Every call fails; results after the injection stops match those from before it, so the fixed point cache is not left half built.
8 threads at once on one shared cache entry, half clean, a quarter always failing, a quarter alternating: 750 clean results and 450 refusals, every clean result matching the reference. Non-FIPS and FIPS.
CPU features read once at power on

Userspace, FIPS v7: the three setters compile to a bare return, and calling them leaves the flags, the lane and every self test state as they were. Control: the same build with the v7 guard removed, where all three setters write and the flags move.
In kernel: the same calls from process, softirq, hardirq and NMI context, under ECDH, AES-XTS and RNG load, leave the flags unchanged with no failures. Control: the module with the v7 guard removed moves them from process context.
ML-KEM key reuse

New API test: a made key reused for a decoded public key, then a decoded private key, must agree on the shared secret both ways. It fails without the fix on a cache-a build (CI's pq-all.json builds cache-a).
The same test decodes a bad public key into a full key and expects decapsulate to be refused. Without the fix it returns 0. Coverage shows the refused call no longer reaches the decryption step at all.
Real kernels

Local KVM on an Intel Core Ultra 9 285K, Linux 6.16.12: the FIPS module with the in-kernel crypto test, booted twice, once with AVX2 visible and once with it hidden so the non-AVX2 lanes run. Both pass the power-on self test, pass the in-kernel test, re-verify at unload and unload clean, with no kernel warnings.
Same guest with a save fuzzer: 8 loads, every one met a refused save, every one reported the error, no crash and no kernel warning.
AWS spot on real silicon against the stock Amazon Linux kernel 6.12.103: AMD EPYC and Intel Xeon, each running the FIPS test suite, the power-on only check, the forced failure check, and the FIPS kernel module loading with the power-on self test and the in-kernel test, then unloading clean.
AWS spot Graviton, aarch64: the cpuid change is in the part of cpuid.c shared by Intel, aarch64, 32-bit Arm and PowerPC, so the FIPS build with the Arm assembly lanes runs the test suite and the power-on only check there too.

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-pq

• FLASH: .text +384 B (+0.1%, 308,656 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +448 B (+0.1%, 309,616 B / 1,048,576 B, total: 30% used)

linuxkm-standard

• Data: __patchable_function_entries +48 B (+0.1%, 51,440 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11422

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs, wolfssl-src, wolfssl-bugs

Findings: 4
4 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[kaleb-himes]

retest this please

Dismiss to re-request

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11422

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs, wolfssl-src, wolfssl-bugs
Findings: 1

Required changes (1)

Small-memory signing configuration no longer compiles

File: wolfcrypt/src/wc_mldsa.c:10447
Function: mldsa_sign_with_seed_mu
Category: API contract violations

mldsa_sign_with_seed_mu() calls the new four-argument mldsa_vec_check_low() with two arguments under WOLFSSL_MLDSA_SIGN_SMALL_MEM and WOLFSSL_MLDSA_SIGN_CHECK_W0. Known #6771 instead concerns signing conformance.

Related known finding #6771 (similar but distinct): Both concern optional early-rejection checks in mldsa_sign_with_seed_mu, but this is a mismatched helper-call signature causing a compile failure, while #6771 is a FIPS-conformance deviation caused by the checks themselves; their fixes differ.

Suggested fix: Pass the vector length and &valid, assigning the helper's return value to ret as at the other updated call sites.
Basis: ISO/IEC 9899:2018 §6.5.2.2 requires function-call arguments to agree with the visible prototype.

Referenced code: wolfcrypt/src/wc_mldsa.c:10447-10448 (2 lines)

---

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Outdated review has been addressed and Fenrir is now disabled for reviews.

[Copilot]

Copilot review overview

🟡 Changes recommended

Failed ML-KEM public-key decoding can leave a stale private-key flag usable with partially replaced public state.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Clear stale private-key flag after failed public-key decode · New
• Update PR description for no-op FIPS APIs and revised testing · New

Resolved since last review (2)

• Failed ML-KEM decode leaves stale key state usable
• Global mask changes do not refresh cached CAST lanes