I always recommend to use weird numbers for different tasks. That way, you can see "oh my system is under severe load at *:13 (or 17, 23, 42, ...). That must be this cron job"
I do a similar thing with ping payload sizes to be able to fingerprint the traffic: I’ll use prime numbered packet sizes so that they can be detected even when the packet sizes are summed.
Btw, one of those numbers is not like the others (has more prime factors that evenly divide 60).
I don't think a UAF on its own allows control hijacking in an exploit. You will need further logic bugs or the ability to legitimately or illegitimately control memory, either via further requests / function calls, OOB writes and such. And it will always be extremely specific to the application vulnerable.
For example, I remember an exploit workshop around reuse of passwd entries. The vulnerable would validate a passwd entry for a non-root user, use it a bit, and later on free and zero it out for security purposes... except it then looked at the passwd entry again in some conditions. Suddenly, the uid was non-zero at first, and zero later on, so later decisions assumed the user was root.
In a more concurrent situation, a possible vulnerability was a shared cache of these passwd entries. It wouldn't zero them out anymore, but you could create a race condition of starting requests as user A, and then hammering requests as user B, so you'd end up mixing up the permission contexts of these users. Because the passwd entry that had been released back to the pool was overwritten, and then re-read with the value from another context.
The necessary line of thinking tends to be very different from thinking about program code regularly.
And even with a mutual agreement - default LLM output tends to be very hard and inefficient to read. We've spent quite a bit of time at work to setup a few writing skills for Claude.
For example, mine is a technical writing skill based on the old book "The Art of Scientific Writing": Front load information as much as possible - the first paragraph of a document should be a summary, the first sentence of a paragraph should summarize. Short sentences building on each other. Always show evidence.
Someone else spent quite a while optimizing a standardized PR description template, with a section for intent, expected changes, risk, rollout coordination and such.
And strictly speaking you cannot trust the pip upgrade of pip with the vulnerable cert in the trust store due to certifi, which you also cannot upgrade safely with pip for the same reason. All of these are at one out of two compromises from being MITM'd during the upgrade.
So strictly we have to burn everything python to the ground and reinstall entirely.
Heh. This is part of why I always preferred getting my python runtime and all these core libs from my linux distro packager. But python die-hards look at me like I'm insane when I suggest this.
Even in today's LLM-addled world, I still think it's a better tradeoff for smaller system/site operators. Get the distro-level testing and regular auto-patching, rather than telling yourself it is a good idea to "freeze" your python world at the pace of your slowest app/system integrator.
It's the same reason I think docker is an anti-pattern for these small groups. You really cannot safely use these strategies to bake an image and divorce yourself from the maintained distro layer unless you are going to adopt CI/CD that does the same thing. Keep tracking all these patches to the entire runtime with all the dependencies, rebuild, re-test, and redeploy.
It's total cargo culting to adopt the freeze/image techniques and just abandon apps in place as if that is robust!
But, it's also more than just automatically updating everything. There is an editorial aspect to distro management to help secure the supply chain to your app. Most smaller orgs cannot feasibly do this as well as the regular linux distro maintainers. Some large orgs can probably do it better, but that doesn't matter if you aren't doing it large org style with dedicated security folks tracking all this against well-developed policies and procedures...
> But python die-hards look at me like I'm insane when I suggest this.
They have a point. The distro is only carrying a small fraction of what's available on PyPI, and will have out-of-date versions that may lack security patches (and for projects like yt-dlp, are highly unlikely to work at all). Realistically they aren't applying security backports; there would rarely be a reason not to just use the new version, and the resources just aren't there for that scale of patching anyway.
Further, pip will interfere with your system package manager's tracking of the system environment (PEP 668), and any attempt you make to install from .deb/.rpm/etc. to a virtual environment is a hack at best. Maybe you could symlink individual package folders to the system environment while installing a superset, but ugh. Oh, and if you're a developer (like most "Python die-hards" are) then this isn't very helpful, as you aren't testing the installation of your package the way that your users will do it.
The mitigation is to ignore nearly all that stuff in PyPI, realizing it is barely a step up from copy-pasting out of stack overflow. Same goes for, e.g. python-whatever RPMs that look like they were drive-by contributions.
So your trusted base is the Python runtime and maybe a few choice libraries that actually have engaged package management. Sadly, it is a trap to think you can put things on the internet as a small player by leveraging lots of third-party stuff, when you don't have the resources to actually maintain that large dependency base.
This also may mean you want a base OS like Fedora and not some crusty "enterprise" variant that tries to bundle a frozen Python from 15 years ago. This is getting a little better in recent years, with some enterprise platforms having optional package streams for parallel install of newer runtime versions, etc.
Edit to add: on the Fedora-derived platforms, pip doesn't really interfere much anymore. The platform python-* RPMs go into /usr/lib/... while pip installs into /usr/local/lib/... or your ~/.local/lib/... hierarchy. The only mysterious behavior you get is if you manage to install in both places and then you have to understand the path search order. Just like if you do strange things with user and virtual environments.
Eh, it's a personal choice and a not unreasonable position to take, but there are a lot of things on PyPi that aren't packaged by e.g. Debian. Database clients, immensely popular frameworks, scientific computing libraries ...
Basically once you leave the realm of the 5-10 most popular building blocks in an area (which OS packagers do distribute), you're still left with plenty of packages that have 10ks+ of users, sane maintainer/security/ownership models, and code that would be very difficult to just copy-paste a la stackoverflow (e.g. native/compiled deps with complex wheel build tooling).
Yes, that's why I left wiggle room for a few choice packages you track yourself.
What isn't really tenable for a small team is to drag in scores to hundreds of different third-party packages and rely on them all when you have no staff allocated to really evaluate and monitor so much upstream stuff.
I agree about getting libs (binaries especially) from my distro, and it generally seems to be regarded as the "wrong thing to do". Freezing anything that might mean you get security updates slower seems to be a mistake to me.
Or - you can just upgrade pip with a SHA hash pin, or set the magic env vars to use the system bundle.
You do need to be careful about creating new venvs with the old version of pip bundled in ensurepip in older Python releases. `uv` kindly patches that to newest pip, but not every Python binary does.
I had a pretty fun time with it myself. Rather than a system package for pip, or the `ensurepip`-provided one, I've been using the copy managed by pipx. I had to:
* Update system certificates
* Locate that pip and directly invoke it to self-update with hash pinning, rather than going through pipx
* Fix a few stray venvs that contained their own pip
* Use the fixed pipx to update uv (although I guess the uv self-update should work even for a pipx-installed copy?)
The legacy-certs hint is neat. I did it the first way but would rather not have since I'm on Mint.
Yep! I tested it with my own vibe-coded MITM proxy that hijacks pypi installations, which I created for ~$0.30 spend on DeepSeek API.
$ python -m pip --no-cache install "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"
Collecting pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70
Downloading pip-26.2.2-py3-none-any.whl (1.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 570.0 MB/s 0:00:00
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70 from https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70:
Expected sha256 fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70
Got 8c448177668e735cdde7d6178694b90c084ba18559348cc394c4b1546eeec89c
The second one works too:
$ PIP_USE_DEPRECATED=legacy-certs PIP_CERT=/etc/ssl/certs/ca-certificates.crt python -m pip install --no-cache --upgrade "pip>=26.2.2"
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082)'))': /simple/pip/
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1082) - skipping
ERROR: Could not find a version that satisfies the requirement pip>=26.2.2 (from versions: none)
ERROR: No matching distribution found for pip>=26.2.2
As compared with a naive installation through the proxy:
Oh nice. Then I have to thank you for a possible non-horrible upgrade path. If you were close by, I'd invite you for some icecream or a round of pool :)
Yep. Per the main thread on the Python forums (https://discuss.python.org/t/information-for-python-users-re...), this now happens even with the outdated pip because the built-in `ssl` module has been patched to reject it. The relevant commit is https://github.com/python/cpython/commit/7dd8b4736a1419750b9... , featuring a rather amusing workaround for security scanners that might mistake the blacklisted certificates (included for testing purposes) for "leaked" certificates accidentally included in a source package.
From code comments, because I found it interesting:
* This is an extraordinary measure and last line of defense. Removing trust is
* primarily the job of the platform root store. This list only blocks keys while
* that update is still propagating to the OS, container images, or root CAs
* bundled in Python packages. The threshold for adding an entry is very high.
*
* See https://github.com/python/cpython/issues/160040 for the eMudhra discussion.
As a Mint 22.x user, I wonder if and when Debian or Ubuntu will pick up the change and release another patch for the system Python (basically, to my understanding, a base of 3.12.3 with security patches from other patch releases backported). Since they've already updated ca-certificates it might not seem necessary, at least as long as the pip package gets updated (although I don't use that anyway).
BTW, I think I saw your comments in other places about the 30-cent vibe-coded MITM demo proxy, unless someone else had the same idea. Thanks for helping educate.
Edit: apparently (https://discuss.python.org/t/avoiding-compromised-certificat...), the relevant Debian and Ubuntu packages already patch the distro-provided pip to enforce using the system certificates. (Indeed, `ca-certificates` is listed as "Depends" for the pip package, rather than "Recommends".) But maybe they'll still be interested in the key blocking, on defense-in-depth principles.
> All of these are at one out of two compromises from being MITM'd during the upgrade.
This is not true; pip supports hash pinning, so an attacker would have to be able to defeat SHA256 as well (or compromise both the official Python discussion forum and PyPI itself, so as to forged the published hash). Other hashes can also be used. Instructions are provided on the Python discussion forum to self-upgrade pip with hash pinning: https://discuss.python.org/t/information-for-python-users-re...
Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
Mmmh so you will be trying to supplant the destination server with your own, with this emudhra leaked cert, to see what's being sent? Or how are you planning to do that? Just curious :)
You're probably right. Based on its MAC and some other quirks about it, I'm certain it's just an ESP32 inside. So if I end up not being able to bend the firmware to my will, I'll be able to bend the hardware to my will instead.
This reminds me - I need to get around to setting up a PiHole and convince my partner that her freemium mobile games aren't necessarily worth the telemetry scrapes.
a high quality set of FOSS, community inspected "freemium clones" may do more for national security and individual privacy compared to yet another crypto library with zero novel mathematics just another reimplementation of specs...
I've had a PiHole running on a NUC for about 2.5 years. I adore it. It should be built into every consumer router.
My wife still plays her freemium games, the ads just don't load. She took out one of those games when we were sitting at the airport once. She was frustrated that her once-pleasant strategy game was suddenly crammed full of ads for boomer slot machine apps, and I had to explain that the app had probably always been that way.
That makes sense and seems like it might be a net positive. While she's waiting for her ad countdown, she can gaze into my eyes and realize how much she appreciates me lol
My girlfriend deliberately watches the extra ads to get in-game rewards, so I can't even pihole the whole home network unless I exclude her phone. (I find that most of these games work fine if you block ads, but they also don't give you the rewards for watching an ad that never loaded.)
Wondering if there's some way to 'load' it by triggering whatever it counts as engagement but still not serving it to the client. Claude, you have $3 left and daddy doesn't want to make mommy mad again today
> Yes, but in the end that's not their fault, and in a sense it's good that this happened.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.
The leadership page is comedy gold ("Leadership That Inspires Trust"):
> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...
> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...
> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.
You say comedy gold. However in Vishwaguru Indian universe they are much vaunted South Indian Brahmins who anyone in the world accept that they are the best in anything they do. Hence they are pure gold.
Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
There's a bunch of changes recently / coming up in Canada, so that might be good enough. I've started occasionally reading the tzdb mailing list archives for fun because of that.
Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.
We've had to rework part of our application and certificate use because of this.
It really is. It's basically chaos monkey for pki.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
The problem isn't revocation of end-user identity certificates. There's hundreds of millions of those floating around at any given point, so revocation happens very regularly. The issue is revocation of intermediates and roots, which involves invalidating tens of millions of certificates and often find themselves in embedded/non-web applications (financial, legal signatures, medical data exchange).
An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.
None of the test certs would be used for real, whether they're intermediate or leaves.
They'd be created with big red warning signs, installed into browsers and other places by anyone who wants to test, and then publicly revoked every month. Rinse, repeat.
You can do that, and I'm sure people already do. But you can't truly test things at scale. Are OCSP servers going to get overloaded? What about IoT devices and things out of support? You have deployments which configure their systems in non-standard ways, maybe some corporate manager decided to mandate cert pinning.
The original comment was to do a fire drill and revoke live certificates to force everybody to participate in the test.
You can have a fire drill and evacuate an office building, but you can't hold a full-scale forest fire/volcano/hurricane drill and evacuate 3.5 million people to test mass evacuation routes, and also you got the edge cases that don't participate in drills, like homebound seniors or hospitals, where disaster strikes.
> I'm sure people already do. But you can't truly test things at scale.
Do they, actually? I got the impression from the article that little revocation testing of any scale happens. If so, then even a limited test would be useful.
It baffles me that software on Linux/Windows/MacOS doesn't just use the system certificate store, then it would just be a single place to update. Instead we need to be update pip, UV, node, etc separately.
I can see the case on phones: the OS may no longer be supported by the vendor, so browsers shipping their own set makes sense. And on embedded there isn't (and on IOT there might not be) an OS, but then you should probably just ship the the single CA that you use, not all of them.
It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!
I wasn't implying inspiration, the point is that Python is not the only major language that does this. Node.js is apparently another. Rust crates like webpki-roots also bundle Mozilla's store.
There are reasons to do this, like support for minimal environments, consistent cross-platform behavior, simpler deployment, and application control over root certs.
I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.
Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.
It looks to me like the world was decently prepared to deal with it. The bad certificates are being blocked and there don't appear to be widespread compromises.
That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.
Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.
I want to go back to talks by bored defcon guys or gals calmly stating how they obtained private keys of a globally trusted CA with just a bit of math. Which, mind you, ate the other part of the teams time.
Now I know how much I hustled today likely due to that atrocity of a video. I think I will take a vacation day and sit in a forest near a river on friday.
reply