r/Python • Python Software Foundation Staff • 6d ago

News Info for Python users regarding eMudhra / emSign CA certificate security incident

A security incident affecting eMudhra intermediate certificates has required removing the emSign root CA certificate from CA bundles such as β€œcertifi”. It’s important to immediately upgrade the software packages below to at least the versions specified and rebuild container images to fetch the latest system certificates. Failure to do so could result in monster-in-the-middle (MITM) attacks against TLS clients, including python.org and pypi.org .

(1): Advisory mentions 2026.10.18, but 2026.10.19 is recommended for compatibility with unaffected emSign root certificates.

For all existing virtual environments on your system, please upgrade pip and uv to the latest versions.

Please read the full post on discuss.python.org for up-to-date information: https://discuss.python.org/t/information-for-python-users-regarding-emudhra-emsign-ca-certificate-security-incident/109576

164 Upvotes

27 comments sorted by

43

u/AlSweigart Author of "Automate the Boring Stuff" 6d ago

What this means for regular folks: update your browser and update pip. You can update pip to a >= 26.2.2 version from a known good location with this command:

python -m pip install "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"

Posted by Seth Larson himself (PSF security engineer) on Discuss.

12

u/d3matt 6d ago

Unless you're running an EOL python... I feel like this deserves an exception to the "we don't support EOL python releases" rule.

8

u/zurtex 6d ago

pip maintainer here, we continue to support Python 3.10 even though the PSF has it as EoL.

LTS redistributors like Red Hat can, and do, devendor pip to provide their own version of certifi. Though I don't know their exact policy here, you'll need to check with each distro.

2

u/Dowlphin 2d ago edited 2d ago

I am slightly worried about mention of 3.15, 3.14 and 3.13 having been updated, but I got system-installed 3.12 on my Kubuntu 24. (Which rejects manual updates because that could break stuff.) For now I found on the web and added config entries instructing pip (2.4) to use the system certs ("legacy-resolver" I hope is right, "legacy-certs" was rejected), but I have no overview or full grasp of the virtual environments situation. Some info says pip 2.6.6 has been patched, not 2.4.

6

u/zurtex 2d ago

This will tell you where certifi is installed from (replace "pip" with how you normall call pip):

pip debug 2>/dev/null | grep -i "pip._vendor.certifi.where()"

A system store will look something like this:

pip._vendor.certifi.where(): /etc/ssl/certs/ca-certificates.crt

If you are not using a system store, but the Python package, you can check the version like this:

pip debug 2>/dev/null | grep -i "certifi=="

Finally, if you have regular internet access you can run:

for ca in dv ov ev; do for chain in g3b g3c g3b-aaa g3c-aaa; do pip --disable-pip-version-check --no-cache-dir index versions pip --index-url "https://$ca-$chain-emudhra.we-have-a-problem.com/" --retries 0 --timeout 10 -vv; done; done

And you should see errors that say something like CERTIFICATE_VERIFY_FAILED with unable to get local issuer certificate

2

u/Dowlphin 2d ago

Amazing response! πŸ˜ƒ

It is using system certs and the final test does yield verify failures. πŸ‘

Question is, if there is that thing with virtual environments, doesn't that mean that every app I previously installed with pip has outdated certs? Or is the virtual environment only generated (anew) when the app is launched and that gets the updated stuff? Because I might have apps installed that way that connect to the internet. Tartube is definitely one. (For which I had to manually install Deno and I managed to update that one already. Tartube uses yt-dlp. Dunno how it deals with certs from inside a virtual environment. As I said, this complexity makes my head smoke.)

1

u/zurtex 2d ago

If you're using the Python provided by the distro, then venvs should also be seeded with a patched version of pip. You can active any given venv and run that pip command.

2

u/d3matt 5d ago

Dang, it hadn't registered to me that 3.10 was EOL too already. 3.9 is still the default for the EL9 series (trivial to get 3.14, I know...)

I'll definitely be pushing people at the office to bump up.

2

u/zurtex 5d ago

I've not checked, but I am pretty sure EL9 patches pip to use their own certifi, not the one we distribute.

3

u/d3matt 5d ago

Oh neat... the EL python3.9-pip's vendored certifi is patched to look at /etc/pki/tls/certs/ca-bundle.crt for its certificates. My usual mechanism for creating a venv though is to immediately upgrade to the latest pip before installing anything else. This of course switches to the upstream certifi 2026.01.04 which I'm pretty sure includes the bad certs.

2

u/zurtex 5d ago

Correct, if you're using a Python version pip maintainers no longer support (Python 3.9 and below) but your Linux distro does, then you should use the pip distributed by them as they are guaranteeing security, not us.

11

u/zsol 6d ago

Thanks Seth for all you do for the community

7

u/HeartfireFlamewings 6d ago edited 6d ago

Correct me if i'm wrong, but creating virtual environments first installs 26.2.1 but can be upgraded with a --upgrade-deps flag, but this uses the vulnerable version to upgrade to one that is safe.

That seems a bit backwards, no?

8

u/ResidentPomegranate6 6d ago

system python upgrade won't touch venvs, each carries its own certifi. rebuild images with --no-cache or the old wheel sticks in a cached layer.

3

u/__peg_parser__ Pythonista 5d ago

I have many virtual envs, and it would be a pain to find each one, activate it, and update all those packages. There was a script provided in the discussion, but it's in bash, and only upgrades pip. Is there some official script that automates all this?

1

u/Boring-Meeting-17 5d ago

Anyone here succeeded in fixing the certs for the built-in /usr/bin/pip3 on macOS Tahoe? I'm following the instructions in https://emudhra.we-have-a-problem.com/#macos-curl but getting "Operation not permitted" when trying to create the "sitecustomize.py" file.

1

u/Distinct-League-474 21h ago

Did you figure it out? For me /usr/bin/python3 and pip3, the ones installed by Xcode command-line tools are still vulnerable and I cannot find a solution, Apple only discusses curl and git on their support page.

1

u/Boring-Meeting-17 2h ago

No, I think the pip that's part of Xcode can't be user-modified -- unless you turn off System Integrity Protection, I think.

OTOH, /usr/bin/python3 is fixed by the /etc/ssl/cert.pem edits. I checked by writing a little program to create an SSL connection to the test sites in the emudhra page. See https://docs.python.org/3/library/ssl.html#socket-creation

Note that the just-released Tahoe 26.7.2 overwrites /etc/ssl/cert.pem with a new version that's identical to the (insecure) 26.7.1 version, so you'll have to re-apply the fix again after the upgrade.

1

u/AcceptableAmount6028 6d ago

How to.avoid such issue in production setup, can u guide !? I want to learn

0

u/[deleted] 6d ago

[deleted]

11

u/sethmlarson_ Python Software Foundation Staff 5d ago

This is a good question, I am writing down the pain points in how Python responds to this incident here:

https://sethmlarson.dev/python-cert-dist-pain-points

certifi exists because Python uses OpenSSL as a TLS implementation and OpenSSL doesn't interface with certificate store APIs for Windows and macOS, it needs either files or directories with CA certificates. Very soon pip will default to using "truststore" (a library I maintain) without using certifi. If pip starts doing this then there won't need to be any updates on Windows and macOS in case of incidents like this.

4

u/d3matt 5d ago

Does it make more sense to get truststore (or something similar) into the standard library?

1

u/shinitakunai 4d ago

Probably yes. The windows ecosystem of python is huge, as it is so accesible for people learning

13

u/zurtex 5d ago

Pip maintainer here, to add onto what Seth has said, two main reasons:

  • certifi has historically been the easiest way to validate certificates in Python code, there was never good platform certificate store support

  • pip needs to bundle all it's dependencies because it's part of the bootstrapping chain, you can't pip install pip without a full copy of pip

I'm doing a full analysis now on what are the problems we might face switching to truststore only by default. I think we're all agreed that's the goal, but also I don't want to break users unexpectedly.

0

u/wRAR_ 5d ago

Why emphasis on Windows? Other OSes also have their own cert stores.