Skip to content

gh-160040: Distrust compromised eMudhra intermediate CA keys - #160042

Open
tiran wants to merge 7 commits into
python:mainfrom
tiran:ssl-distrust-emudhra-keys
Open

tiran wants to merge 7 commits into
python:mainfrom
tiran:ssl-distrust-emudhra-keys

Conversation

@tiran

@tiran tiran commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Several eMudhra intermediate CA keys were compromised and revoked. As defense in depth, reject any chain containing one of the affected public keys, so connections are blocked even when the root store (OS, container) has not yet dropped the issuing roots or certifi/pip are not updated.

The denylist in Modules/_ssl/distrusted_spki.h matches the DER-encoded SPKI, so cross-signed certificates that reuse a key are blocked too. Mirrors Golang's crypto/x509 change. See https://emudhra.we-have-a-problem.com/.

Several eMudhra intermediate CA keys were compromised and revoked. As defense in depth, reject any chain containing one of the affected public keys, so connections are blocked even when the root store (OS, container) has not yet dropped the issuing roots or certifi/pip are not updated.

The denylist in `Modules/_ssl/distrusted_spki.h` matches the DER-encoded SPKI, so cross-signed certificates that reuse a key are blocked too. Mirrors Golang's [crypto/x509](https://go.dev/cl/851047) change. See https://emudhra.we-have-a-problem.com/.

Signed-off-by: Christian Heimes <christian@python.org>
@tiran
tiran force-pushed the ssl-distrust-emudhra-keys branch from a2f652f to 6275d65 Compare October 21, 2026 15:03
@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Oct 21, 2026
Name the SPKI table struct type and list the const table in the c-analyzer
"should be const" ignore list, so check-c-globals passes. Document the new
verification behavior in What's New for 3.16.

Signed-off-by: Christian Heimes <christian@python.org>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tiran tiran added type-security A security issue and removed type-feature A feature request or enhancement labels Oct 21, 2026
@nchay1030-star nchay1030-star mentioned this pull request Oct 21, 2026
@read-the-docs-community

read-the-docs-community Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Documentation build overview

📚 cpython-previews | 🛠️ Build #35322568 | 📁 Comparing 42a0b3e against main (0d35e3b)

  🔍 Preview build  

48 files changed · + 1 added · ± 47 modified

+ Added

± Modified

Comment thread Tools/ssl/emudhra_denylist_check.py
Signed-off-by: Christian Heimes <christian@python.org>
Comment thread Modules/_ssl.c Outdated
Signed-off-by: Christian Heimes <christian@python.org>
@@ -0,0 +1,6 @@
-----BEGIN PRIVATE KEY-----

@sethmlarson sethmlarson Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We receive a steady stream of beg bountiesreports to PSRT about private keys being present in bugs.python.org, etc. A compromised private key would likely invite even more attention, can we obfuscate this in a way that prevents that from happening?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I was thinking the same thing based on past reports. also, distrusted_keycert.pem looks like a premade cert signed with the distrusted private key... rather than checking that hazmat-ish thing into our tree, could the test wanting that just generate one at runtime when an openssl command is available (too slow? we can tag it as @support.requires_resource('cpu') ).

I like that the filenames clearly have distrusted_ in their name. I wish things scanning for private key strings would grow up and learn to not report things like this but they tend to be a trivial grep.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's two ECC genkey operations and one ECC signing operation. That is incredible fast, maybe 10ms in total on a recent computer.

Is this really worth the trouble and extra complications? "Security scanners" are already complaining about the private RSA keys. And they will start to complain about the eMudhra CA certs, too. I could obfuscate the files with ROT13 or put them in a password-protected zip file (Brave Sir Robin?).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is really the dumb scanners that do stuff like blocking git push - I have to jump through all sorts of hoops to admin-override checks just to maintain a mirror at work, and then I also have to go and add comments to all these files to satisfy another scanner that this isn't a real secret (or else they're going to contact my manager 😟 ).

But yeah, even something that separates the header from the base 64 and writes the key file out just for the test would be a huge pain relieved from my perspective. Even renaming to .txt may well work, though I haven't tried that. I assume others face similar issues.

Comment thread Modules/_ssl/distrusted_spki.h Outdated
@@ -0,0 +1,6 @@
-----BEGIN PRIVATE KEY-----

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I was thinking the same thing based on past reports. also, distrusted_keycert.pem looks like a premade cert signed with the distrusted private key... rather than checking that hazmat-ish thing into our tree, could the test wanting that just generate one at runtime when an openssl command is available (too slow? we can tag it as @support.requires_resource('cpu') ).

I like that the filenames clearly have distrusted_ in their name. I wish things scanning for private key strings would grow up and learn to not report things like this but they tend to be a trivial grep.

@gpshead gpshead added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes needs backport to 3.13 only security fixes needs backport to 3.14 bugs and security fixes labels Oct 21, 2026
@gpshead gpshead added the needs backport to 3.15 bugs and security fixes label Oct 21, 2026

@sethmlarson sethmlarson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few comments, otherwise I am happy with the implementation.

@@ -0,0 +1,6 @@
:mod:`ssl` now rejects TLS certificate chains that contain any of a set of

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed in the parent issue, we should document clearly that this is a last-line-of-defense, and not meant to be relied on by users or distributions as fix for a CA certificate compromise.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated the blurb, what's new, and header file comment. Please review and suggest changes.

Comment thread Modules/_ssl/distrusted_spki.h Outdated
* Verification rejects any chain with a cert whose SPKI SHA-256 digest is in
* the _distrusted_spki table. We match the key, not a cert fingerprint, so
* cross-signs that reuse the key are blocked too. To block another key, add
* its digest.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a note here for our future selves about the high threshold for a compromise to be considered being added here.

Comment thread Modules/_ssl.c
@sethmlarson
sethmlarson requested a review from zooba October 21, 2026 19:21
…neration

* use SHA256_DIGEST_LENGTH
* verify cb: keep OpenSSL error, fail closed on internal errors
* generate intermediate key on the fly
* update docs to explain defense-in-depth / special circumstances

Signed-off-by: Christian Heimes <christian@python.org>
@tiran
tiran requested review from gpshead and sethmlarson October 21, 2026 21:25
Comment thread Doc/whatsnew/3.16.rst
verification while that update has not yet reached the OS, a container image,
or a bundled trust store in various Python packages. Keys are added to the
denylist only under a very high threshold.
(Contributed by Christian Heimes in :gh:`160040`.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documenting the change in What's New document makes it harder to backport to other branches. Would it be enough to document it as a Changelog entry instead?

If we keep it in What's New, it should be moved to special sections in old What's New documents (when the change will be backported), such as https://docs.python.org/dev/whatsnew/3.14.html#notable-changes-in-3-14-5.

tiran added 2 commits October 22, 2026 12:47
The compromised eMudhra CA certs and the leaf signed by them are kept in a
password-protected zip so naive security scanners don't flag them. test_ssl and
make_ssl_certs read the PEMs from the archive.

Signed-off-by: Christian Heimes <christian@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting core review needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes needs backport to 3.13 only security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes type-security A security issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants