Skip to content

Fix virtualenv seed cache reuse with a namespace bump - #19721

Closed
SeanDougherty wants to merge 7 commits into
fasttrack/3.0from
users/sdougherty/python-virtualenv-seed-cache-19710
Closed

SeanDougherty wants to merge 7 commits into
fasttrack/3.0from
users/sdougherty/python-virtualenv-seed-cache-19710

Conversation

@SeanDougherty

@SeanDougherty SeanDougherty commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Either/or alternative

Draft alternative to #19732. Choose one approach, not both.

Both PRs independently target fasttrack/3.0 and update
python-virtualenv from 20.36.1-8 to 20.36.1-9. Neither depends on the
other. The original trust-store update, #19710, has already merged.

This PR uses a one-time cache namespace bump. #19732 uses the selected
wheel's content hash to handle future same-filename replacements automatically.

Problem

The trust-store update changes embedded pip contents without changing the
wheel filenames. The default app-data seeder can therefore reuse old images
containing the affected roots when creating new environments.

With downloads and periodic updates disabled, retained caches accepted
7/14 incident chains for pip 25.3 and 14/14 for pip 25.0.1. Selecting a newer,
corrected upstream pip can avoid this, but periodic updates do not guarantee
immediate remediation.

Change

  • Advance the seed-image namespace from image/1 to
    image/1-azl3-trust-202610.
  • Bypass legacy seed images for new environments without deleting files
    still referenced by existing environments.
  • Add four offline upgrade-path regression cases covering both pip wheels
    and copy/symlink seeders, including corrected-image reuse.
  • Preserve the parent's sources, signatures, security backports, trust-store
    helper, and %check runner.

This namespace is fixed, not incremented on each install or package release.
Another required invalidation would need a deliberate namespace change.

Deployment and cache cleanup

New environments seeded from the corrected bundled wheels automatically
bypass legacy seed images. No manual cache reset or additional package
update is needed for this cache invalidation.

Legacy files remain on disk to avoid breaking existing symlinked
environments. This does not repair existing environments that still use
affected pip CA bundles; those require a pip update or recreation.
Unused legacy images may be removed after confirming no environments
reference them. Deleting them is not required to protect new environments.

Validation

  • Complete RPM source preparation and spec guideline checks passed.
  • All four new regression cases failed on the parent and passed with the
    namespace change.
  • 8/8 targeted tests passed, including existing parallel and populated
    read-only cache tests.
  • The source-runtime reproduction rejected 28/28 incident-chain/store
    cases
    after the namespace change while retaining the app-data directory.
  • Fully prepared wheel RECORD entries and affected-root removal were verified.
  • Merging the current target into this branch preserved the validated package
    files byte-for-byte and removed already-merged parent changes from the PR diff.

This namespace alternative still needs binary RPM builds, configured full
package checks, and installed-RPM validation.

Separately, parent PR #19710 passed AMD64/ARM64 builds and configured package
checks in ADO build 1235565, with 396 passed, 40 skipped, and 2 deselected per
architecture. An installed AMD64 release-7-to-release-8 upgrade reproduced
the retained-cache issue. Those results do not validate this namespace patch.

Jon Slobodzian and others added 5 commits October 23, 2026 23:41
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e62137f2-2602-418d-ba0d-3b1f2ff0259e
Use private test environments and constrained dependencies from configured pip sources instead of replacing RPM-owned packages or requiring an offline source bundle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e62137f2-2602-418d-ba0d-3b1f2ff0259e
Use the built wheel and pinned test dependencies in a private environment. Run pytest directly without enabling subprocess coverage, retaining the existing test selection and all certificate/security fixes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e62137f2-2602-418d-ba0d-3b1f2ff0259e
Keep the same 21 dependency pins in a quoted heredoc inside %check. Remove the separate constraints source without changing test selection or certificate/security fixes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e62137f2-2602-418d-ba0d-3b1f2ff0259e
Use a one-time Azure Linux seed-image namespace to bypass pre-update caches while leaving existing environments untouched. Cover both embedded pip wheels and copy/symlink seeders with offline upgrade-path regression tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Base automatically changed from users/joslobo/emudhra-python-virtualenv-20261023 to fasttrack/3.0 October 25, 2026 03:45
@microsoft-github-policy-service microsoft-github-policy-service Bot added the fasttrack/3.0 PRs Destined for Azure Linux 3.0 label Oct 25, 2026
Keep this draft as an independent 8-to-9 alternative after PR 19710 was squash-merged. Preserve the previously validated package files unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SeanDougherty SeanDougherty changed the title Fix python-virtualenv seed cache reuse after trust updates Fix virtualenv seed cache reuse with a namespace bump Oct 25, 2026
Distinguish automatically bypassed seed images from existing environments that still need a pip update or recreation. Explain that legacy files are retained for environments referencing them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@SeanDougherty

Copy link
Copy Markdown
Contributor Author

Closing this draft because commit f59907689a01f381c20e92b2461e45e46b1f3968 in #19708 blocks the known compromised eMudhra intermediate keys in CPython's TLS verification, even with stale cached pip CA bundles.

Once the patched runtime is deployed and loaded by restarted processes, cache invalidation is not required for that protection. Certificate verification must remain enabled. The branch is retained for any future work on general same-filename cache consistency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fasttrack/3.0 PRs Destined for Azure Linux 3.0 Packaging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant