I am shocked that one of the private keys was literally “4”. Anyone could have broken into this. I really hope there is an industry-wide effort to stop things like this in the future.... but I doubt it.
There is a variety of projects and researchers scanning for weak keys; possibly that's how this was found (haven't seen anything on that). These certs existed for a year and a half; that's bad but could be worse still.
One of the issues with sub-CAs (well, X509 certs in general) is that they can be ghosts in that there's no way you'll ever get it as a response on a TLS connection, so automated scanners on that won't find it.
CT should cover this (i.e. expose the cert so you can check for weak keys), but I have no idea if/how CT works for sub-CAs. [ed.: see below, doesn't seem to be in CT]
[ed.: these are technically sub-CAs, but rather cross-signed for bootstrapping a new root CA. Thanks @agwa for pointing that out.]
For their own stuff, Mozilla has talked in the past about specifically just only trusting intermediates they've seen. Their modern root agreement says the CA will disclose all relevant intermediates, so arguably they have a complete list and further if that CA "forgot" about a few of them whether that's because your firm is too disorganised ("Oh, right, the one on Barry's laptop, forgot that one") or an actual attempt to sidestep this rule - then we certainly don't want to trust these undisclosed intermediates, do we ? They're either purposefully shady, or they're forgotten about and likely never have been audited.
Nothing for "CN=EM DV TLS CA - G3B,O=eMudhra BV" (not sure which sha256 this uses, 2nd is the spki sha256)
I should be clear here my understanding is/was these are sub-CAs. Apparently they're (trying to be?) roots themselves? The point of logging a root cert in CT is somewhat moot… though tbh I would argue anything issued by a CA should be in CT logs, even cross-signing another CA that is becoming a root itself.
> I would argue anything issued by a CA should be in CT logs, even cross-signing another CA that is becoming a root itself.
And it is, or at least, was. The CT logs when I last spent too long staring at them were complete indefinite logs. These days most (all? almost all?) are time limited so e.g. there'll be Clown 2026 and then Clown 2027 and a certificate is logged to the CT log corresponding to a not-after date IIRC. This way you can throw away a log of certificates expiring in 2025 a few weeks into 2026 because who cares, they've expired, and so they don't just grow forever.
I no longer have a role where I care a great deal about this for my day job, and I do not know how to ask ctlogs.io for the thing I want. So probably this is still logged but I couldn't promise you without getting back into this. Maybe at the weekend.
> One of the issues with sub-CAs (well, X509 certs in general) is that they
> can be ghosts in that there's no way you'll ever get it as a response
> on a TLS connection, so automated scanners on that won't find it.
Could you expand on this? I don't have a full grasp of what you're saying (I'm not super knowledgeable in CA stuff).
My understanding is that if something, like a browser, connects to a SSL site, they will either have or can obtain, one way or another, a complete chain all the way up to the root, so there can be no "ghost" certificate. And I would imagine a scanner could do all those same steps, and thus access all the same certs.
Is this a deficiency in scanner software? You seem to indicate it is a fundamental weakness in X509; I'm just trying to understand what.
Could you maybe describe a specific example of what you're describing?
If you're doing a MITM attack, you can selectively intercept certain clients, and only serve the compromised certificate in those cases only. Automated scanners won't pick that up. None of this actually works though, because browsers started mandating certificate transparency.
Please be careful with your phrasing, as there are MANY other critical things on this planet that use TLS certs which are not browsers, and some are extremely difficult/impossible to update.
VPNs, embedded software (IoT, automotive, aerospace, medical, industrial etc.), business systems, you name it.
In the most extreme case, you can just issue a certificate and not use it anywhere. If it leaks, it's still a security issue, because it's valid.
More realistically, especially with sub-CAs, they might only be used on e.g. corporate networks. TLS scanners may not be able to reach any of the hosts that uses such a cert. As the sibling post points out, clients checking for the cert in Certificate Transparency logs counters that (adding a requirement that the cert be publicly known, which TLS/X509 by itself very much does not have) — but of course this only helps if your client does CT…
Ah okay, thanks. So to be concrete, something like automation running 'curl' on an internal network might see the full chain, including the weak cert, and be vulnerable, but scanners on the wider internet won't see it. I was confused by "...there's no way you'll ever get it as a response on a TLS connection..." — I guess you just meant a _public_ TLS connection. Or it could be otherwise installed on corporate machines' local cert bundles.
Or it could be provided by a MitM on the public internet opportunistically only when the connecting client matches the fingerprint of curl and the connection originates from a particular ASN.
Is this real? This seems so ridiculous it makes me wonder for how long this CA has been in circulation in systems and if it has been exploited by someone previously discovering it in that time (state level actor or not). My old phone running Android 6 does not have any certificate from EMudhra, at the very least.
There are plenty of ways you would never be able to tell if a fake cert had been generated and used via this attack vector (let's ignore validation attacks):
1. CT does not do anything if a state actor demands a CA in its borders generate a cert without publishing to the CT logs (or a three-letter-agency doing this illicitly). They can put SCTs in the cert, and still not publish to the logs, nobody is auditing the logs for every cert every browser sees.
2. Browsers won't check for SCTs in certs if they come from a private CA. you can MITM any domain through ZScaler all day long and nobody will notice
3. Wildcard certs make it easy to make/renew a cert that looks normal but hides requests on new/different hosts
I feel somewhat confident in speculating that a bunch of people at large companies will, in fact, be auditing the logs following this. That’s their main value; they’re not really an online/active prophylactic.
“They can put SCTs in the cert, and still not publish to the logs, nobody is auditing the logs for every cert every browser sees”
I don’t think you can get the SCT from a log without the log publishing it? That would defeat the purpose.
“Certificates that support certificate transparency must include one or more signed certificate timestamps (SCTs), which is a promise from a log operator to include the certificate in their log within a maximum merge delay (MMD).[4][3] At some point within the maximum merge delay, the log operator adds the certificate to their log.”
> I don’t think you can get the SCT from a log without the log publishing it?
Well you can, but only if the log cooperates.
That is, the SCT is merely a promise to add the certificate to the log by some particular deadline. Nothing technically stops the log from disregarding that promise. But if it does fail to log the certificate, the SCT itself is proof of its misbehavior.
And it's not enough to subvert a single log, because browsers require SCTs from multiple distinct operators in order to consider a certificate valid.
The SCT may be proof that a log was supposed to be issued, but browsers don't submit every SCT they see (did you know "anonymizing" proxies send CT log companies a sampling of all your TLS traffic? fun stuff!), so in practice these logs aren't audited. As far as multiple operators, almost all of them are based in the US, so just pray the USG doesn't want to target you.
I hold eMudhra's private key. I generate a cert signed by their intermediate on my box. eMudhra didn't issue the cert, so it is not sent to any CT logs, and I have no SCT to put in the cert.
I think I might just be talking past you. If your point is "anyone with these broken eMudhra keys can attack users without caring about CT", that's totally true --- it's just that browser/no-browser doesn't factor into it. I'm really only on this subthread because of the notion that CT enforcement requires browsers.
Right, I think I'm with you that browser/no browser shouldn't be a factor. I think I'd like to hear from Rust and Go webpki people on whether we could help the ecosystem enforce CT.
My point is more that as a user of a high-level library like reqwest, I'd like to talk safer defaults when root CAs are in use
Sure, but a client that doesn't do CT wouldn't care if they don't include a SCT promise. [ed.: this said "break the SCT promise", that doesn't quite make sense] Which you'd only notice if you see the certificate somewhere else afterwards…
It's all rather hypothetical and unlikely, but you could certainly try attacking something like downloads.openwrt.org¹ like that — especially if you can tell apart the TLS client in opkg from a normal browser.
(¹ luckily, OpenWRT and most other package repos don't actually rely on HTTPS for package integrity and authenticity. And OpenWRT would be the most susceptible due to the "exotic" TLS client stack, if any.)
To spell it out a bit further, an attack strictly in a non-browser context that is reasonably well targeted will most likely go undetected since what the attacker is up against is the chance of inadvertently running into a paranoid user who goes out of his way to perform verification.
The baddies also take a look at the CT logs and look for fresh wordpress instances that haven't yet been configured and will helpfully configure them for you within a few minutes.
Yes but a lot of non-browser software doesn't check CT logs, so if an attacker were to target such software they could use an unlogged cert and no one would notice.
The CT log is a property of the individual leaf certificate not a property of the issuing chain. You could have the same CA in the trust store issue an unlogged certificate and non browsers would trust it.
A CT log isn't super relevant (it's not the CRL, it's just a log), but it's maintained/delivered by the CA not the leaf. Never the leaf.
If you mean CRL/Revocation list, everything but the root [you can't revoke a root] is maintained by the tier above. Leaf -> intermediate (these), [Intermediate -> Intermediate*], Intermediate -> Root.
Not sure why you're mentioning transparency logs, they have nothing to do with this (nor help).
Because the thread: "a lot of non-browser software doesn't check CT logs", you replied saying it's a reason to use the OS trust store, not sure how that fixes things? Maybe it would relieve them of the responsibility of pushing out a distrust update?
That's not what woodruff is talking about (I hope). The CT logs allow us to (later) find who/why it happened. CT Logs in no way help you stop something. (Depends a bit on how software reacts to apparently invalid issuance - but since these have the real keys the issuance will look normal).
Revocation lists allow you to stop/block bad issues. Both are maintained/decided (you could post to another service) by the issuer, not the leaf. CRL records would stop this (but some software has decided that's too slow). In general software uses the OS trust-store first (fragmented on Linux). Firefox and Java have their own independent paths. Chrome ignores CRLs
CT logs only tell you something happened if the certificate was registered in the log. But if you are attacking software that doesn't check if the cert was registered, then you don't need to register the cert, and there won't be anything in the log.
My point is you can forge a cert, without putting it in the log, and a lot of software will still be vulnerable to it.
Curl does not check that certificates have a valid SCT attached. Thus it will accept a certificate that isn't in the CT logs. That's mostly independent of the OS trust stores.
Nope. You can find this from the Chrome Root Program.
> Because no TLS certificates chaining to these six intermediate CAs were logged to CT prior to our CRLSet block on October 17, no fraudulent certificates issued by these CAs could have been trusted by default in Chrome or other CT-enforcing browsers.
They are not exclusive to each other. The most successful criminal organizations work with, and from within, nation states, their intelligence agencies and law enforcement. Oh hi, we see you.
I admittedly don't know much about how CAs work but why would a state actor care about such an exploit? Wouldn't they just ask any local cert issuer and receive a "legitimate" cert?
If FBI went to Godaddy and said, “We need a cert for www.google.com”, Godaddy would most likely refuse. Issuing that cert would leave a trail that would be extremely undesirable for both FBI and Godaddy.
To be clear, emSign is another name for EMudhra, and Comodo is now known as Sectigo. (You can't just look at the organization name in a certificate subject; you have to consult the CCADB: https://www.agwa.name/blog/post/the_certificate_issuer_field...)
Third party audits from a qualified auditor are required, although the CA chooses and pays the auditor, and the auditors are basically just accountants. So it's... not a perfect system.
Inclusion in the Chrome Root Store requires that a CA Owner continuously demonstrate the technical competence and operational rigor necessary to protect Chrome users. Due to the seriousness of this incident, the Chrome Root Program is conducting a thorough evaluation regarding the status of eMudhra’s remaining CA certificate to ensure these standards continue to be met. We will share further updates as our review progresses.
I bet Entrust never made a cert with a private key of 4.
The problem is the death penalty case is aimed at the wrong target: The entire Web PKI system remains dumb and bad, and the correct answer is to dissolve the CA/B and find people with practical security experience.
Entrust (now Sectigo) is involved with this [0] (cross signed an intermediate) [1] Even last year buying a Sectigo cert got you an Entrust cert with no mention of Sectigo/Comodo[2] (maybe more of a Character.AI situation than a sale)
This is probably a rough definition for "Entrust was banned from being a CA and sold the business off to a competitor". In part, because I also do not think the root programs would just allow Entrust to continue operating the same business under another brand name, unless they are even dumber than I already thought.
I'm not exactly sure which of the roots from eMudra were compromised, but the "emSign ECC Root CA - G3" which is mentioned in the OP site as one you should blacklist, is present on my phone which is running Android 12 with the last security update from three years ago. Having a better timeframe for when the impacted certificates appeared really would have been nice.
None of the roots were compromised. (Some of) Their own sub-CAs for TLS certs, which are also cross-signed by Sectigo (since their own roots are rather recent), are affected.
How you have that root cert on your phone with a 3 year old CA set I have no idea, presumably something updated your CAs in the meantime? (Also, aren't you incredibly exposed to the general onslaught of vulnerabilities from the past 3 years?)
Well that makes sense. The "emSign ECC Root CA - G3" root is listed as being originally issued on 18 February 2018, so would have been really concerning if that itself was the compromised certificate.
I believe Android updates things like CA certs independently of their monthly security patch. I'm not sure Android 12 uses the same system, but on newer Android I think those are shipped via Google Play system updates, which don't require waiting for the phone vendor.
Yeah, in Android 14 they separated it out so that the CA store can be updated from Google Play. Versions below that you're reliant on the manufacturer to ship updates.
It's not within the realm of believability. The chance of "human screwed up" astronomically exceeds this being fairly randomly chosen, and the Firefox bug states, (I think this is supposition, not known fact, but IDK)
> The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order.
When you have multiple selections you can show that the probability of selecting a set with a range much smaller than the population range is exponentially unlikely, strong evidence to reject the hypothesis that the subset was chosen at random. But when N=1 you can't conclude anything.
Regardless of the probability that this cosmically unlikely occurrence naturally happened 8 times in a row, we all know what the privkeys are now so the certs can't be trusted anymore.
This is actually interesting. I'm no cryptographer. Do modern algorithms take into account that certain values should be avoided because they would be obvious things to try first in a brute force attack?
The algorithms don't care because it doesn't make sense to care, the same reason a "smallest non-interesting number" wouldn't make sense - now that number is interesting precisely because it was cited as the smallest non-interesting number.
However practical applications do care, and so e.g. the "Debian Weak Keys" are a relatively small set of keys which would have been generated if you used a particular patched OpenSSL to generate "random" key pairs after a mistake patched out most of the randomness. Many of them were used by Debian sysadmins particularly in the window between the patch and the discovery - and so all CAs are expected to check that you're not trying to get a certificate for the public half of one of those pairs.
There's nothing "bad" about those keys except that well, we know some people used them so bad guys will try the private halves of those keys just in case. Likewise there are a few apps which shipped with a baked in "default" private key, and all the paperwork to apply for a certificate but obviously a user may not realise that if they just fill it out with their name and details, they're now vulnerable to attack by literally any other user of that software. They were supposed to generate a random key pair and re-generate the paperwork for the resulting random values. If they don't realise we need the CA to say "Nope, that's a bad idea, not issuing" even though these keys aren't special in any other way than they're given to every single user of that software.
IIRC when I worked for a famous US corporation their middleboxes relied on such a well-known key and at some point I asked "Hey, this seems wrong. Why did you do this?" and as often has happened in my career I got no actual reply to my question but magically the issue was fixed in short order. The first clumsy attempt to fix it broke some things, but they gradually figured it out. Glad I could help, I guess.
If your generation is any good the odds of ending up on such a number should be astronomically small.
Biasing something that should be random tends to badly backfire. See the Enigma who avoided enciphering letters to themselves, which was then used to break it.
That's assuming perfect random generation. If you assume real world where mistakes can happen during generation, 4 is way more likely than 25105785070300501195559896594243230376961191116602934339784363418717203371363
Definitely not. Their CPS requires that these keys be generated by an HSM and no HSM would ever generate these, so there's little reason to believe they're following any of their other controls.
My vague and inexperienced read on their description boils down to "we were doing something weird and non-standard as part of our key ceremony because we intended these keys to be exported, and because of that process weirdness we blundered and gave the HSM faulty instructions."
The most likely explanation is that they generated the weak key externally, imported it into the HSM, generated the CSR from it, and finally had Root CA sign it.
A FIPS HSM does not forbid one to import keys into it.
But shouldn't such a HSM support key attestation (and the parent require such attestation)? Attestation would also track whether a key was imported or generated on the HSM, and of sufficient length.
Attestation is not required by the compliance standard for browser CAs (WebTrust) and it is also quite vendor-specific and not very interoperable, so it is not commonly done.
The Root CA must sign requests after authentication, but a "manual" approval is also accepted if done with the appropriate oversight with witnesses (clearly not the case here).
Of course it's not required, as we see here. The question is why not? Getting a simple code-signing cert seems to have more stringent requirements in that it either requires attestation or CA-issued hardware.
My big fear about HSM is that they are inscrutable by design. There is no way to verify they are working as advertised. Plus side, all sorts of fancy hardware to protect your keys. Down side, all sorts of fancy hardware that obscures flaws in the implementation.
I think I'd prefer open source hardware built from off the shelf components glued into a secure enclosure. That way you can destructively audit a few at random if the application matters enough to justify the cost.
While you are right in that it is impossible to verify they work as advertised, it is not impossible to detect certain categories of failures. Like only generating values 0-255.
You could choose to detect small keys, but my understanding is that implementations entirely eliminate the issue by only searching factors with leading bits set (i.e. large numbers).
Not only should they never be trusted again, the individuals responsible can't either. They need to be blacklisted somewhere. This wasn't garden variety stupidity, this was (IMHO) do-not-collect-$200 stuff.
> The other two certificates are for codesigning and trusted by Microsoft
That seems like a WAY bigger problem than TLS, but AFAIK Windows still validates code signing certs against CRLs before trusting them (can't remember if it soft-fails).
Why is that a way bigger problem? Windows will happily run unsigned binaries. Code signing just decreases the friction for users slightly, but is in no way required. TLS is definitely a way bigger problem.
> You already need compromised systems to install malware, signed or not. Not a real problem.
Is this true? Ads pointing to a malicious Notepad++ is already an existing thing. Now their malicious version will pass signing checks. Users will happily install these without the system itself being compromised.
Organizations that are serious about security should have a whitelist of binaries that can run on their computers. Enforcing code signing isn’t good enough.
> Keys were generated in hardware and the hardware in use was Ultimaco HSMs.
The plot thickens.
Broken HSM? Misconfiguration? Confusion? Straight up lie?
> Excel attached. The Ultimaco devices were used to generate the ECC, RSA key pairs for the issuers. They were not used to generate any other key material. But it doesnt look like the device is at fault.
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
Reading further, it seems like they say they first generated some keys on-HSM, but then used 'test keys' (that were used to test wrap/unwrap of key material) instead by accident. Fishy.
They used the "correct" key for signing the CRLs (which of course was then an invalid signature):
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
And THAT is where I'm drawing the line between "honest mistake" and "gross negligence". For that to not raise giant red flags is an unacceptable failure in process and work culture.
I've seen speculation that there was a language barrier, given that this is an Indian company.
The theory was that someone misinterpreted 384-bit keys to mean the key must be a number between 1-384.
I think that's entirely made up, but it wouldn't be completely implausible to have happened while someone tried to troubleshoot why the first attempt failed.
Ultimaco HSMs! Well Ultimaco today published a marketing blog post entitled Global Encryption Day: Encryption Is Only as Strong as the Keys Behind It. It seems like a subtle dig at their customer’s incompetence.
> "The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys."
Ah yes.
"The system says something is wrong, but it's also 4.30PM. Let's just re-sign the thing and go home."
Anyone have a mirror for the first link? It served me a page assumedly intended for AI scrapers and then promptly blocked me when I refreshed a few times.
I BET this is an error code. they wanted to generate some key via something and it errors and put error code 4 as key and since encryption is dumb and people are people, no one looked at the key material.
edit:
The reported private scalars were 4, 56, 66, 115, 155, 156, 230, 237 across the eight certificates
so its unlikely i will win my bet :'D
Someone answering “questions for emudhra” also say the keys were intended to be exported to US and EU data centers. I don’t know if that would have precluded inspection (hopefully it’s encrypted some way, but why not have the destination HSM generate the CSR?)
Edit: I see the below paragraph is quoted extensively below. I hadn’t seen those comments yet. Oh well
We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
The RSA issuers went through a different path as per our assessment of the code and are therefore not impacted.
Some additional context - these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one).
Wrapping/unwrapping means the HSM would generate the private key and then encrypt it with a key only known to the other HSMs. Those other HSMs would then decrypt the key on import. The private key should still have been impossible to examine outside the HSMs.
Hey that's a byte of entropy! And one of the rare cases where choosing a larger value might offer you better "protection" against serialized brute force attacks
The first 6 keys are literally someone typing them in lazily on a keyboard. You can almost see their fingers barely moving across the top numeric row as they generate them.
Looking at the pattern I would actually suspect a numberpad over top row of the keyboard. That explains how the responsible party got 1-5-6 without exhausting their delicate digits.
This should be higher. They mentioned a "coding error", and getting valid keys from the HSM and then truncating the string to a single byte through buggy array handling (see above) is exactly how you'd end up with a bunch of integers in the range 0-255.
This is the C explanation perspective, but a similar thing could happen in Python, if you write `s[0]` instead of `s`, perhaps due to a refactoring that changes an array of keys to a single key. This is because Python does not distinguish characters from strings. (Not an indictment of Python; just another proposal for how this could happen, and I would say it is more likely to use Python than C in some hacked-up signing script.)
Well, my example is C++, it relies on function overloading existing for, for example, to_string(uint8_t[]) and to_string(uint8_t) separately, which means this error would come down to overloading instead of catching the type error.
Just saying, because C++ is very popular everywhere in this "kind" of company, if you catch my drift.
For fun I gave the public key to Opus 5.5, telling it that this is a puzzle, and it should solve the puzzle without web search. Opus 5.5 told me the private key of 4 after two minutes, including the time to write and run a 30-line Python script implementing a baby step giant step search for discrete logarithm. As far as I can see, it did not use web search or use any command that would have internet access.
Do we call this human's slop? Should we be so hard on AI when humans do something like this? I'm pretty sure LLMs did not and would not allow such things to happen.
On the bright side, hopefully with AI we won't see silly slops like this.
The impact of this incident is very unevenly spread.
Over 10 years ago, Mozilla shipped OneCRL [1]. We remediated this incident in a couple of hours for Firefox users who were protected automatically without the need for updates. Chrome, Apple and Microsoft have similar tooling. Since then, we've built more advanced tooling to handle revocation at scale for leaf certs (CRLite [2]).
However, for the rest of the ecosystem, there has been little to no investment in revocation and for those clients, this is a massive problem. Hopefully, we'll see folks invest now. Efforts like upki [3] look promising.
I'm hoping some organizations will step forward to fund work on this.
(Note that curl on Ubuntu 26.10 now has CRLite support baked via upki -- unfortunately that doesn't help with this because AIUI CRLite is only for end entities, not for intermediate certificates.)
You mean the response was uneven. The impact was a complete and total potential compromise of literally everyone. Every domain. Every client using one of the public root programs. Every user of those clients.
I mean, it's an emergency, but the acute severity is debatable; many times a month the major browsers quietly deal with vulnerabilities that yield attackers direct control of victim computers; here, attackers get the ability to spoof websites if they're in a position to hijack connections.
The middlebox situation, with non-updatable roots, is very annoying. But middleboxes also have unfixable RCEs, so...
It's totally fair to evaluate different projects by how well positioned they are to update their WebPKI trust anchors. I'm just saying, keep perspective.
So it other words, it allowed attackers to do the one thing that Web PKI is supposed to prevent.
Being able to quickly update trust roots is at best a hack to paper over the issue when there is no good reason why every podunk CA should have authority to generate certificates for every domain in the first place. It's an inexcusable design flaw. Or a feature that gives power to root stores at the expense of everyones security, from the perspective of the root store operators.
Keep in mind that a _lot_ of software out there uses TLS as its only trust anchor for updates, commands or similar.
This can and will be weaponized for take-overs of systems. They will be of the consumer kind, but they will also be of the industry kind. Let's pray that they will not be of the state or military kind.
Firefox Mobile 158.0 on Android 17 (Pixel 7) connects to all the demo sites with zero warnings. Any idea why the "automatic protection without updates" doesn't seem to be working with that combination?
Pixel 10 with 158.0.1 still connects to the emSign ones, even though the patch notes for 158.0.1 mention that it updates the CRL for those without 'Remote Improvements' enabled?
edit: Disabling all the eMudhra CAs in Android settings does nothing, cool?
edit: I guess there's no way to manually ask for CRL update, cool??
edit: Gave up and reinstalled Firefox. Seems to be fine now.
You mean you and Chrome stopped supporting normal CRLs and OCSP, replacing (for normal users) these distributed-trust technologies with central lists controlled by browsers
OCSP stapling works fine technically, which is one reason it's still available in both browsers via enterprise policy. LE dropping support for it and browsers having insufficient courage to hard-fail doesn't make the model broken. It instead makes the ecosystem broken.
Just another step towards a tiny number of browser vendors and infrastructure companies controlling the internet.
The screening process has been more involved in that since around the late 2000's. I wouldn't call 20 years "very recently," especially in the context of the history of the web.
circa 2007 I was looking over a college textbook on computer networking that I saw at a family member's house. It still said "While networks are inherently unreliable, it's not a stretch to imagine a future where one can stream video on-demand over a WAN like the Internet"
Which is to say, sub-CAs in itself are a giant loophole that needs plugging. Especially since AFAIK nobody is bothering with name constraints (RFC5280 section 4.2.1.10).
They hold intermediate certificates issued by TWO different root CAs, neither of which they operate.
So two separate entities didn't bother to read their certificate policy or inspect their operations. Have we not learned anything from past mistakes?
Roots should be responsible for all their sub-CAs, full stop. If they don't trust them enough to accept that responsibility, then the system is broken.
The reason nobody bothers with NCs is because they don't work very well in practice. They also have their own baroque matching rules that are not intuitive if you're from the land of UNIX globs.
(In fairness this is a self-fulfilling prophecy: a lot of validators handle NCs poorly or not at all, so there's no impetus to rely on them. But that's not something we can really save in the Web PKI's current state.)
> But that's not something we can really save in the Web PKI's current state.
I agree with your assessment but it's not an excuse to give up without trying. I hope this is just because I'm not looking in the right places, but I don't see people trying.
What really makes me salty is that there was enough energy to mess with the rules around EKU, the primary effect of which is that non-browser software is now starting to just ignore the EKU bits. Even if it would've amounted to nothing, I'd really rather that effort had gone into NCs.
I think a lot of people have tried! The reason EKUs stuck where NCs haven’t is because EKU rules are simple (in relative terms) and OpenSSL didn’t make as much of a mess out of them.
I can't help but read your post as "OpenSSL didn't make a mess of EKUs so the CA/B had to fill that role instead" :/
(I really am salty about the EKUs. In a few years the EKU field will just be useless, due to operational annoyance driving devs to ignore it. Except in browsers of course.)
eMudhra is also a root CA operator. One root CA operator cross-signing another root CA operator is a good thing actually, and it's what allowed Let's Encrypt to work before their root certificate was widely trusted.
Issuing sub CAs to non-root operators has historically been a big loophole though it's not as bad these days since multiple root programs now require approval before CAs can do it.
Didn't Let's Encrypt have to jump through all sorts of hoops to prove they were competent and trustworthy before that happened?
Now it's coming out these guys used "4" as a private key. That's a crazy amount of incompetence and malfeasance. Did they vibe-code a CA? HOW DOES THIS HAPPEN? I am dumbfounded.
You couldn't even do this if you installed Microsoft Small Business Server 2008 today, unpatched, with the CA on default settings.
It's very inconsistent for me even on just Firefox. On multiple systems running different instances of Firefox, checking with https://emudhra.we-have-a-problem.com/:
One blocks everything; one blocks 6 out of 12; one blocks the other 6 out of 12; one blocks nothing.
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
They don't blame the HSM.
Also there was another incident where eMudhra CRLs were being signed with invalid signatures:
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
> these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one)
This bit sets my hair on fire. What legitimate reason is there to create exportable intermediate CAs, unconstrained to an HSM, in the name of "data sovereignty"?
This is a really, really dumb way of doing online spying (presumably, you mean by running a program of deliberate misissuance) since every cert they issue is recorded in a public cryptographic ledger.
I'm not saying this will never happen, just that rolling out a dramatic infrastructure change for "data sovereignty" just to pull off a SIGINT heist that is almost certain to get your CA distrusted seems like a poor risk/reward calculation. I fully expect IC agencies to get some CAs killed and be fine with that.
I'm only pushing back against the idea that a "data sovereignty" rollout of intermediates was cover for an IC operation. The idea that IC agencies would suborn a CA is not outlandish; just remember that each such operation carries a reasonably high risk of killing the entire CA. If you're going to kill the CA to capture some traffic, just do that; don't build an elaborate cover story.
> rolling out a dramatic infrastructure change for "data sovereignty" just to pull off a SIGINT heist that is almost certain to get your CA distrusted seems like a poor risk/reward calculation
Well, meet the Indian government that does these calculations all the time.
Well, I honestly believe it is rank incompetence. Speaking as someone who's monthly electronic GST filings, commercial authentications, and digital payments, i.e. continued livelihood, is beholden to cert and PKI authorities like eMudhra having their shit together...
As for the odds of this being state-sponsored... It depends on the state's incentive, weighed against its continued behaviour, over the last 10-odd years. Everything below is easy to substantiate with citations. There are too many to cite... I put myself at real risk just by stating any of the following under my real name.
---My take: Never attribute to malice, what one can to (rank) incompetence.---
I expect eMudhra-esque collossal gaffes, as a normal course of life, stemming from the apathetic, insular, feudal-minded, shoddily run bureaucracies that the majority of our government and quasi-government organisations are.
Basic day-to-day operations being unavailable, because "Sorry sir, site is down." is a normal everyday life experience of their own staff, not to mention the common person. eMudhra is one of that tribe, and I'd say they are not the worst offenders of the lot, by a wide margin. I'd probably give the "best of the worst" title to our national Employee Provident Fund.
Of course, one can't totally rule out nation-state-level stupidity, such as BGP poisoning by the Pakistanis.
If our current federal administration has incentive to do it, they will... And, given their track record, if it does turn out to be nation-state-engineered for short term tactical gains (to hell with the consequences for anyone else), all I will do is sigh and move on with my day. Cornered politicians can go to any length to save their own skin (usually, just to save face)... Collateral damage be damned. Popular examples include world superpowers bombing distant nations, for whatever gain or goal (usually short-term, because well you most probably won't be President a second time (and never a third)), using whatever pretext they can sell to get others to also get behind (the individual's psychological cover in a firing squad, scaled up to many allied powers).
In India, absolute majority governments have amply demonstrated the same failure mode as US heads of state empowered with de-facto fatwa powers, as of the US PATRIOT act. Given motive and incentive, they easily turn the full might of their uncheckable wrath and ire at our constitution, as well as directly at the common public.
You know, like sending out military forces to "calm down" the pesky student protesters acting against "the national interest".
As we speak, massive nationwide student protests have been bringing shame and angst to the government of our ruling party at the centre (BJP and allies), as well as their various governments across states. Many protests have happened at this scale in the last decade+ of the same ruling party's unbroken streak as a majority power in the centre (federal government). Those other protests were like water off a duck's back. The ongoing student protests, for reasons I've not dug into, appear to be a clear and present existential threat to the incumbents.
My gut feeling is... the incumbents aren't passing the electoral vibe check any more.
The sentiment on the streets of the national capital, and large urban centres nationwide---almost entirely comprised of a new crop of young voters (18 y.o. up to early 30s)----is that they feel robbed of an economically viable future. They've organised at scale (millions on the street) with persistence (they are literally owning the "Cockroach" epithet thrown at them by the powers that be). And in short order, material consequence---unprecedented in its decade-long winning streak so far---befell the incumbent party's administration. The Education Minister had to resign, and they've generally lost face with that whole voting-age cohort. Brand-name loss of key leadership is further disrupting their internal party cohesion. Plenty of hopefuls in there, salivating after the top dog position, but thus far too weak to sway the cult following of the public faces of the party. Even an hint of weakness gives the wannabes a real opening to try. Textbook demagoguery.
If that is indeed true, then in their place, I too would seriously worry about losing even more ground in the next national election... It's not far now. Because? "Winning by overwhelming majority" has been the one and only priority of the incumbents, driving all their work post-election (where work includes constitutional shenanigans)... Being a populist party borne of, and cultivated by far-right nationalist ideologues (the RSS and "Sangh" community), with a longstanding desire for religious supremacy (Hindutva and Sanatan dharma), they publicly and loudly dream of a hundred-year golden age. Uh, no, now I think it's become a thousand-year era (really!). That's not totally insane if you're suddenly goated at winning electoral politics, because our head of state can enjoy unlimited tenure. Yes, you heard that right, India has no constitutional limits to number of re-elections to the Big Chair, whether successive or alternating.
Without looking at absolute numbers, I feel confident that the influx of new voters (lots of people can turn 18 in a single government tenure) far outstrips the older ("whatsapp uncle / aunty") voter base that the incumbents pander to. This has been a lucky break for the fumbling Congress party because their legacy name is finally regaining some cachet... The much-mocked Rahul Gandhi presents like someone the young'uns can actually talk to, like a human being.
But the Gandhi-brand Congress were once an absolute majority power too. At which time they suspended our constitution, and weaponised federal agencies, and created the playbooks that all future governments have been able to deploy at various scales. I expect nothing less from any other party that grabs absolute majority. Currently it's the BJP. Tomorrow, it could be anybody else. Thus far, it's been the secular-presenting Congress, followed by the Hindutva-majoritarian BJP. Woe betide us if next the Communist Party of India gets a bite at that juicy apple. 'cause, like, I don't have to explain what that sort of government can do with absolute power. <shudders>
---Nation-state action: Whither evidence of having the means and using them with impunity?---
Ref: our liberally interpretable Information Technology Act. Among other things, it enables and empowers arbitrary blanket Internet Shutdowns and Blackouts, by local states as well as national authorities, to "protect the peace". In some deeply conflicted areas, like Jammu and Kashmir, censorship measures under martial law would give both, the Great Firewall of China and the Stasi a run for their money.
Ref: at-scale weaponisation of Pegasus, as of 2021. Federal and even state authorities like police departments have weaponised nation-state electronic warfare technology with impunity. Usual suspects / targets are politically inconvenient individuals and groups... opposition politicians, bureaucrats with a spine or conscience, students, media, artists, NGOs etc...
All this mixes badly with pre-existing supercharged Sedition laws. These effectively obviate habeus corpus, and are routinely used to prosecute speech interpreted as anti-national / seditious under the generous ambit of our IT laws (we don't have a real privacy law). Mere suspicion and/or fear and/or dislike + Pegasus + garbage IT laws = authority can brazenly plant incriminating evidence on your device... it's reportedly been done multiple times.
People have been held in prison for less (just inconvenient speech), without possibility of bail, for months and years on end, without trial. The main goal is to cause irreparable reputational damage, to create future deterrence... It's pretty effective. A secondary goal is to take the stubborn ones off the board, with no legal recourse, until they break. And of course, to permanently defang the truly outspoken and indomitable few... Some of those people have been in high-security federal prison for years on end now.
In conclusion:
I'm confident all federal administrations of India, current and future, will hold the lawful means to act wilfully, crudely, with impunity, at scale against India's own citizenry.
I'm not sure whether they currently have enough incentive to have used the eMudhra cudgel, to disrupt the protestors. Sure they could MITM the protestor websites, but it's so much easier to block them and takedown their socials. And for more specific work, they already have field deployments of Pegasus and other MITM / spyware technologies.
India is more impacted from this incident than the West. Wondering did Mythos found this and informed Mozilla about this? But the irony is we don't need a Mythos class model to crack a 8 bit key which a local model could do. eMudhra is having bad days.
Additionally, from the Mozilla researcher verifying the compromise:
> I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates."
This incident somehow exposes another previously overlooked issue. I just notice that so many softwares vendor their own version of CA store for no particular reason [1][2]. It is relatively easy to patch system CA bundles. It would be quite challenging to patch every binary that embedded their own CA. As such, many systems will remain vulnerable for quite some times.
There is a particular reason: Users are bad about keeping their servers up-to-date.
Try to distribute a program without an embedded CA bundle to any number of users, and after a week you will surrender due to the bug reports you receive because users are running 5+ year old system CA bundles, which are not sufficient for connecting to some of the most popular API endpoints out there.
> users are running 5+ year old system CA bundles, which are not sufficient for connecting to some of the most popular API endpoints out there.
Really?
I was under the impression that when a new CA starts operating, they get their certificates cross-signed by older CAs, to ensure their certificates work on older devices? Lets Encrypt did something like that [1] - although they started operating in ~2015, their certificate chain started with a root valid from October 2000 to October 2021 [2]
After all, many website operators want users to be able to connect even if they're using prehistoric Android versions.
You can access Wikipedia with Android 4.4 which was released 13 years ago [3] (although apparently not on the slightly older Android 4.3)
From what the internet is telling me, the keys were allegedly found on https://repository.emsign.com/certs/<redacted>.crt
Which, if true, is definitely a lot of fun.
Hope you people are ready for lots and lots of suddenly hackable IoT.
__
Edit: I might also be misinterpreting what I am seeing and it's actually weak logic that created the keys, in the sense that one could guess the private key?
Someone actually qualified please give credible answers.
I'm just pretending.
I think you’re right. And since the intermediate certs expire around 2039 and the private keys are known, we have plenty of smooth MitM fun against IoT in front of us.
Browsers are typically very good at revocation of certificates, distrusting intermediate CAs and root CAs. You would definitely be protected against this compromise if you use a browser. Command line tools less so. Try this curl one liner
> After Wayne's report of key recovery, I was able to recover the private keys for the impacted keys trivially. The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order. The 8 impacted keys all used integers less than 256. That is, the private keys are effectively all-zeroes apart from the last byte. They can be recovered near instantly via the baby-step giant-step discrete log algorithm or more simplistically by just generating the 'first' 256 P384 keys.
Yeah that’s just oof. Not hard to imagine the kind of code error that could cause this, but hard to imagine that any serious vendor of cryptography hardware or software could make a mistake like this.
Linux - the platform which offers no ability to revoke an intermediate certificate, thus necessitating every certificate library to implement its own blocking?
Windows is already blocking the compromised CAs as of today with no need for users to upgrade anything.
Every certificate library has to implement its own blocking no matter which platform you run on. Windows provides an API to do this at the system level, which means nothing if a given application vendors their own TLS library.
It's uncommon on Windows for applications to use a third party certificate verification library because Windows provides an API that does it. On Linux, you have no choice because all the platform provides is a bundle of PEM certificates.
> uncommon on Windows for applications to use a third party certificate verification library
I bet you have at least one copy of OpenSSL on your Windows system, probably multiple. Using the Windows native APIs is work. OpenSSL just works everywhere.
(Also, "the platform" doesn't technically even provide the cert bundle, depending on exact distro & packages.)
The Go and Rustls upgrades specifically block the intermediate certs because neither of these use the system revocation lists, I believe. Once you upgrade to a trust store without the root cert for the CA this will also solve the issue. Not sure about other ecosystems.
In that case it was ssh sessions being intercepted.
But my question is if a tor exit node or a vpn provider or your internet provider has a private key to any CA then can they mitm any normal web traffic even though it will show up as https without warnings in your browser?
When you're using a modern browser that browser will require any certificate to have a certificate transparency log stamp. Certificate Transparency logs are public append only logs of issued certificates.
So you access example.org. Your ISP uses the compromised CA certificate to create and sign a rogue certificate for example.org. It now has to submit this certificate to a public certificate transparency log and gets back a proof for that. Now it can use that rogue certificate and the proof that it's in a CT log to impersonate example.org and MITM your traffic.
CT does not prevent compromise, but it makes it far easier to detect. If the people behind example.org have a look at the CT logs they will see the rogue certificate there that they did not request. This makes rogue certificates and by extension CA compromise detectable. So your ISP cannot just do this forever in secret.
That depends on if the browser checks the certificate transparency log records or not. If it doesn't, this is a CA every browser trusts.
Browsers will demand for modern certificates that they're accompanied by SCTs which indicate the certificates have been logged publicly. Certificates signed locally by an attacker aren't and the revoked CA certs should be immediately rejected by the CT providers.
If the attacker has the key before disclosure, Google.com appearing in the CT logs would immediately set off alarm bells. If it doesn't get a CT log entry, browsers reject the certificate.
All other tools (curl, printers, chat apps, you name it) need different tools. Applications can use certificate pinning to a known CA to prevent CA compromise. Generic tools will not have an obvious recourse.
I find difficult to understand why intermediate certificates are present in trust stores. The whole adavantage of having intermediate CA certificate was that they can be quickly revocated in cases like this, without impacting the root CA and so without having to touch trust stores.
> I find difficult to understand why intermediate certificates are present in trust stores.
They're not generally (there is a specific case of preloaded intermediates, but that's not the main problem here). The certs here aren't in any trust stores I'm aware of directly. The key change to mitigate this is indeed revocation, but that's hard, because many non-browser clients don't actually support revocation in any functional way.
Firefox has gone further though, and distrusted the corresponding root completely in their store. That's not strictly required if you support revocation for the intermediate, but their trust store is widely reused elsewhere and this guarantees the certs won't be trusted even without revocation support. And honestly everybody is now suspicious as hell of eMudhra, so they'd rather be over cautious.
The Mozilla rep who created that thread (not the reporter, to be clear), created https://emudhra.we-have-a-problem.com/, so I suspect that this page is better for general public disclosure.
With that being said, that Bugzilla is the best thread to understand "how the fuck did this happen?"
> I think it's worth noting that this compromise is much worse than the private keys simply being stolen by a researcher. They are inherently weak single-byte keys that anyone can factor, and the issue cannot be contained.
> This would have been possible since the certificates were created in April 2025. ... CT-enforcing clients like Firefox, Chrome, Edge and Safari could not have been attacked.
Not trying to throw shade at curl, although I don’t think this feature can realistically be done by a drive-by contributor.
No, the article listed some apps that were not vulnerable, and the first question to me is “so what is vulnerable?” And the answer is “every curl | sh installer on the internet since April 2025” (among assuredly many other less prolific apps).
While this is of course very bad, I am happy to see for once something which everyone works together to fix and understand in the bugzilla thread.
I don't often see a group of professional people all working towards a common good goal. (Which here is simply to revoke the certs from everywhere and make sure everyone has the information, but it still makes me happy)
Why are the intermediates, and not the root cert, being revoked?
The intermediates seem to be difficult in that the root stores don't seem to have a mechanism for it (… kinda seems like an oversight, at this 20:20 vision point?). If eight of the CA's intermediates are hosed … how many are still "good", and are they good good, or are we just fooling ourselves?
I guess the other question I have is just how many intermediates does this one CA have? They've had 8 compromised thus far; that's not the entire lot of them?
You know the old saying: if you find 8 cockroaches in your house, then a realistic estimate for the total number of cockroaches in your house is exactly 8.
I assume every state level actor is now quietly trying the first few hundred million private keys possible or so just to see what root ca or issuing ca key pairs match.
- Could this be used to run a benign mitm attack on the traffic of some apps, consoles etc? I could see some interesting uses for reverse engineering. For example you can't simply add a CA to the system store on Android anymore. But this one is already there. It won't work for long but it could be used on any unpatched client.
- Can't we just revoke the entire CA? Just like what was done with Diginotar. Edit: apparently they just revoked the intermediates.
- Why isn't there a mechanism where a domain owner can specify which CA it allows? Similar to SPF which allows you to specify which mail server can represent you. It's kinda insane that foreign CAs can just fake anyone's certificate.
> - Could this be used to run a benign mitm attack on the traffic of some apps, consoles etc? I could see some interesting uses for reverse engineering. For example you can't simply add a CA to the system store on Android anymore. But this one is already there. It won't work for long but it could be used on any unpatched client.
One issue might be that from what I understand Android itself also enforces CT, but it has fail-open design if it cannot obtain the updated list of qualified CT operators for 70 days. So you would need to block the access to that endpoint.
This is assuming the CT logs will stop/stopped actually giving SCTs for those CAs.
> Why isn't there a mechanism where a domain owner can specify which CA it allows?
CAA records do this, but only for issuance, not for use, so it only protects against eg. unexpected domain validation, because the CA has to respect the CAA record. Extending that to use seems like a logical step to me.
A domain owner can publish DANE DNS records with details on the certificate and/or CA used for the domain. With DANE, if a misbehaving CA emits a certificate for your domain, it won't be valid since that certificate doesn't match what you're declared as permitted. This is further solidified with DNSSEC.
Regrettably, none of the major browsers implement DANE, so if any CA emits a certificate for your domain, they'll trust it implicitly.
So is the way to go forward that browser-vendors implement bruteforcing of private keys for a couple of seconds whenever the browser encounters a previously unseen certificate?
Seems like the root cause here was human error during EMudhra's key generation ceremony process.
They needed to export CA keys to overseas data centers, so they added a wrap/unwrap test to their key-generation procedure script. Somehow that generated a second set of weak (test?) keys from the CryptoServer Se-Series Gen2 HSM.
Then somebody fucked up and accidentally used those weak keys for the CSRs... then their root and Sectigo signed it. And it shipped.
That's why the private keys are literally one byte.
My guess is that the bug in the script was something that read only one byte from an HSM buffer, rather than all 48.
> Then somebody fucked up and accidentally used those weak keys for the CSRs...
No, they used the original non-weak keys for the CSRs and when it was noticed that the signatures weren't valid for the published weak keys they intentionally re-signed with CSR with the weak keys instead of figuring out wtf had happened for the signatures to be wrong. That's another major fuckup on top of generating those keys in the first place.
Tangential, probably useless, anecdote: DeepSeek 1017 refused to open the page and read it for me, suggesting that we-have-a-problem.com is very likely a scam domain.
Perhaps unrelated to this - a few days ago we had communications from a TLS certificate provider that we need to reissue all of our certificates by 21st 8PM before they will stop working. We pressed _why_ and was only told that Godaddy 'had a compliance issue with a batch of certificates'. Which doesn't make much sense for us as we have a few dozen and their renewals vary throughout the year.
I've only found one other person talking about this online.
What's very strange is that we don't use Godaddy directly and it's signed by Starfield G2 (now R1) - not the Godaddy root CA.
Has anyone else had the same communications around their TLS certificates?
Your certificates should never be signed by the root - this is prohibited by the CA/B Forum (see Baseline Reqs 6.1.7). My understanding of best practices is that the root signs intermediates then is taken offline ASAP, and the intermediates sign leaf certificates.
Re: Starfield, GoDaddy (and AWS) use the name in trade but operate the issuance. The dev-sec-policy mailing lists should detail any transfers of ownership (in which cases, it is common to maintain an old brand name under a new root).
eMudhra is a vital CA issuing certificates for Indian Govt and PSU sites. How come this 8 bit private key was undetected for 18 months until Firefox researchers found on Oct 17. Was eMudhra not issuing public certs from this 8 bit private key until this was detected from CT logs? Is this really a human error or some kind of a foul play?
Just wrote up a quickie about how Nix + LLMs make patching everything super simple via overlays. Simplest security response I've ever done in my life. http://ghuntley.com/patch
It's worth noting that they also operate CERTInext which provides TLS and other certificate services in the US. In particular, CERTInext was chosen to be the CA for InCommon which means it's used by LOTS of universities all over the country. This was a migration from Sectigo that took place this year.
they will have to file an incident report, which will be heavily scrutinized, and can possibly (edit: i am leaning towards likely, as more information comes to light) result in their ejection from most/all root stores.
Previously such incidents have lead to the CA business cessation by the involved entities (e.g. Diginotar case). I dont see how it would be different in this case.
This is so blatently terrible, its hard to imagine they won't be permenantly removed from the CA ecosystem. I'm pretty sure other CA's have been removed for less.
If you're responsible for PHP client software, and waiting on composer/ca-bundle to release a new update, be aware that https://github.com/paragonie/certainty has been updated with the 2026-10-20 bundle already.
For fun, tried putting a copy of the EM OV TLS CA - G3C cert in a folder ask Opus 4.8 (5.5 would get blocked) to make a sample private key for the scalar 4, like the one that would've been used in the creation of the CA cert.
I find it fascinating watching it's "reactions" to discoveries like this.
Web PKI is cursed. Why should anyone have to ship a new version of Go, Electron, webpki-roots, etc. (possibly over a now-compromised TLS connection!) just to revoke a certificate? Why are so many packages independently shipping high-sensitivity trust-bootstrap credentials? And why can any CA still vouch for any domain instead of each being limited to a specific ccTLD or other non-total shard of subject names?
Also, it's quietly, in the email space, that we're seeing real innovation while browser vendors just say "no" to everything. We should be distributing trust, not centralizing it in browser vendor security teams and Let's Encrypt.
Web PKI doesn't mandate that every client ship with its own root certs. The clients you listed can be configured to use a centrally managed trust store, if that's what you want. The fact that they don't by default is a deliberate tradeoff that the clients made, not some standard's fault.
This isn't, like, one misbehaving client. It's the way the ecosystem has evolved. GP didn't specify whether they were criticizing the standard or the ecosystem more broadly, but even if they were criticizing the standard, it's pretty fair to evaluate a standard at least partly based on how it gets implemented in practice. There's a reason cryptography tools for the last decade or more have been focused on being misuse-resistant.
Yes. I am criticizing the de facto web PKI, not the somewhat better design intent. The design intent is still bad though: we should never have given CAs universal issuance privileges.
The web has continued to go the wrong way in practice. (For example, browser makers rejected RFC9102 DNSSEC stapling, which could have enabled an incremental migration away from CA-centric hell.) Everything fix the existing web PKI community makes just seems to centralize trust more.
For example, Let's Encrypt has been useful for moving people off unencrypted services, but side it's now ubiquitous on the internet and had to work within the broken universal-grant CA model, it now represents a terrifying central point of failure. Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
> The design intent is still bad though: we should never have given CAs universal issuance privileges.
This is sort of right and sort of wrong. It's true that you could have a DNSSEC-style regime where only one entity could issue for a given domain, but as long as you're going to allow third party issuance at all, then you have to contend with the fact that most CAs want to issue in the big gTLDs (.com, .org, etc.) and so narrowing issuance doesn't help that much in many cases.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
This is ahistorical. By the time that DANE was designed, the WebPKI model was well entrenched and there was a huge population of browsers that couldn't really be upgraded and also only supported the WebPKI. If you wanted something that was acceptable to those browsers, it had to be WebPKI and it had to be cross-signed, hence Let's Encrypt.
I suppose you could argue that at some point after that one could have tried to deploy DANE and phase out LE and the rest of the WebPKI. I don't think that's true for the reasons that have been discussed on HN about DNSSEC, but that's really not the same as you wouldn't have needed LE in the first place.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
Had we been using DANE instead of WebPKI, the response to this sort of compromise would be "aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem that there's no way to route around this DNS provider being untrustworthy."
DNSSEC key rollovers don't change who has custody of the keys. They're not a magic spell that fixes stuff like this. It is closer to the truth that DNSSEC is limited to recovery from incidents like this to doing rollovers, rather than immediate revocations. Rollover is not something that makes DANE more attractive here.
I don't think I understand what you're trying to argue here; if you can't find a usable registrar for your domain… you can't really use that domain anyway, can you? They can just break it. But… let's not get into yet another DNSSEC/DANE discussion.
I'm not sure I see what this has to do with how the DNS operates, beyond observing that key rollovers are not a reasonable response to incidents like this.
So you're upset that Google et al can't dictate who has authority over .com with DANE? Because yes, that's how DNS is supposed to work and also how domain validated web PKI works because that's just a glorified DNS consensus system which happens to give bullying power to browsers and root store operators. It would be up to ICANN (or countries for ccTLDs) to determine if the issue warants replacing the operator.
There is no one ecosystem. The world of web clients is highly diverse. Expecting every client implementation out there to converge onto one particular trust store is, first of all, magical thinking, but second of all, it reintroduces the single point of failure that web PKI is meant to avoid. The OS trust store is just one of many possible trust stores, there's nothing ordained about it. The problem is not with some standard, it's with a mistaken pre-conceived notion that there must only be one.
Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!
If you want all of the web clients on your system to use a centrally managed trust store, then it's your responsibility to (a) know and enforce what clients exist on your system and (b) configure them all to use that trust store.
I'm not making an argument for a single centrally-managed trust store. The thing that seems most obviously bad to me here, as a non-crypto-expert, is that any CA can issue a certificate for any domain. This is probably made worse because there are hundreds of CAs, and AFAICT functionally every trust store trusts all of the same ones.
I don't know how we fix this, but it does seem like a big problem.
> Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!
But this is the kind of DevEx decision to improve usability that goes against security, in the case of a shitstorm like this one. I mean, yes obviously everybody should be updating and refreshing all the systems running in all the environments daily/weekly but the reality in many (most) places is very different.
So, it's an engineering decision with its trade-offs, as every decision. And OP was criticizing that decision(s) since this eMudhra issue happened now.
As as thought experiment: if no language were to distribute its own copy of root certs, we would have either docker images embedding them (which would be the same for dockerized workloads) or maybe some system would have been created to easily share a set of root certs within a cluster of applications.
My understanding that go at least doesn't ship their own root store; they're shipping a patch to blacklist certain intermediate certificates instead, even if whatever OS-level root store would have accepted them.
Basically, the design didn't have a way to distrust anything other than removal from the root store. Which may still happen, but that will take longer.
DANE is a live thing in email. MTA-STS likewise uses DNS (unfortunately combined with a well-known HTTP URL path) to denote the equivalent of HSTS without everyone carting around a giant preload list. DKIM's signer-described partial-coverage signatures are pragmatic. DMARC's one-of-two alignment check is good and flexible. BIMI is good.
In general, email has used DNS to a greater extent than the web has to build trust mechanisms that don't depend on giant lists managed by a tiny number of browser vendors.
What's worse is, if this hadn't come to light, how would you even know when an attacker issues a cert for your domain?
Well you might think, oh I can monitor the certificate transparency logs.
I would say well, good luck downloading and parsing terabytes of certificates as they are logged. Then there are CAs whose logs will rate limit you in a way such that you can't catch up to the certs being logged with one server and IP. The system might "work" but not for domain owners. Good luck.
Maybe I am too noob to understand, but triangulating the info here, in Mastodon and Twitter, seems to be a very different sense of urgency and importance. Here it's very chill, in the second absolute armageddon, and in Twitter it's just (etno/racial) dunks against the company.
Is this really serious? Someone knows the implications?
trying to understand where this service is mainly used. judging by a quick google search ...
is this related to Government of India related websites and their services/websites like tax filing etc? is the impact expected to be concentrated around indian citizens availing these services?
https://kagi.com/search?q=dns+CAA CAA is one way, but you can also pin the actual certificate or certificate chain (which has its downsides, mind you).
CAA is a hint to honest CAs to ignore requests to issue certificates for your domain (unless they are explicitly allowed). It does absolutely nothing about dishonest actors with access to trusted keys.
You can pin in software you distribute. If you're big enough (top 100 websites mostly), you can ask browsers to pin certs and they may included it in their preloads.
> And what happens when you want to switch?
You have to update your pins and then wait long enough that you don't mind losing users who never update their software. When it's just in software you distribute, you can change hostnames in new versions and use the new CA for new versions and the old CA for old versions...
For web browsers, there's not a reasonable way to make a nicer transition. You have to serve old certs for old pins, and TLS handshakes from browsers don't include enough information to know if the client has an old pin or a new pin; or at least, mostly they don't --- if you get your pins changed and the browser TLS stack changes so the handshake changes after that, maybe you can use that... when sha1 certs were phased out, it was somewhat popular on high volume sites to look at the TLS Client Hello to decide to serve a sha256 cert or a sha1 cert.
DNS CAA only pins issuance, not acceptance; clients can not check if the CA that issued a cert was listed in DNS CAA at the time of issuance, and they do not check if it's listed at time of presentation. I'm pretty sure issuing a cert contrary to a CAA record is a mississuance and would result in sanctions from the CA/Browser forum, but clients would still accept the cert until it's revoked.
There was briefly browser support for site initiated pin on first user (HTTP Public Key Pinning), but it got dropped as mostly no one was using it and it was easy to get wrong and break your site.
You can pin, but almost nobody does (and half the world that does pin is probably pinned to Let's Encrypt).
Those who manage the root domains should have been able to pin who they trust, and then if you WANTED to you could add someone beyond what .com or .egg or .muffin lists.
Interesting to mention Google because the whole system still relies on them. They decide what certificate transparency logs can enter the program and offer the list of logs. If you don't trust Google web PKI falls apart.
It doesn't have to, you can set X509 name constraints which say "the CA may only issue/may not issue certificates for these Subjects," or you can set CAA in DNS, for example.
(Of course, the former isn't really intended to constrain trusted Root CAs, moreso internal PKIs. But it is an option people should know about!)
Wow, Thanks for digging up/working out that support list! Actually a glint of hope.
…but, yeah, the issue is on layer 9. Even just nailing sub-CAs to ccTLDs breaks the moment a marketing person registers a .lol domain.
Then again, the CA/B is essentially a layer 9 entity. I just refuse to believe this is an unsolvable problem (though it might need NCv2 with a whole bunch more shit to cover more human creativity in policymaking.)
Yes and NO. These are intermediate CA's and for them to be recognized by your browser or system, they need to be signed by another CA or root certificate that your system already trusts.
SO, not all CA's are signed by root certificates already on your machine
IOT devices famously never get firmware updates, which means most of them will remain vulnerable to this leaked trust chain until the cert expires in 2039. That means you can forge an HTTPS server for any domain that an IOT device connects to and act as if you are the authoritative server. This means intercepting encrypted traffic and being able to decrypt it, or serving payloads to the device from a fake trusted server.
If your TV connects to whatever.lg.com, you can create a certificate for that domain using this intermediate signing cert, and pretend to be that server. That's how https://dv-g3b-emudhra.we-have-a-problem.com/ was created.
I would {hope,expect,urge} the people who can easily convert those numbers into proper key form to just not do that, to leave the tiniest of barrier of entry for trolling scriptkiddies.
Browsers might've gotten their magic revocation updates, but a lot of other things haven't… (and yeah this is "security by hoping hard enough", but still better than nothing.)
Nah, the more trolling script kiddies causing mayhem we get the quicker things get fixed before they can be exploited by actors interested in doing actual damage.
This sounds like a misunderstanding of how PKI on the web works. Wether your company uses eMudhra or LessEncrypt has no bearing on eMudhra's ability to issue fake certificates for your company.
Unfortunately due to the insane design of web PKI you can't choose not to use them as every CA trusted by the root stores can sign certificates for your domain.
> An attacker with the private key of any of the compromised intermediate CAs could impersonate any website or service.
And how many more times do we need to read this before we replace Web PKI with a system that isn't as completely insane?
We already have a functioning system for delegating domain ownership in a hierarchical fashion where compromises are limited to only the subtree under the compromised entity. Let's start using it for domain validation sooner rather than later.
DANE replaces this system with one that has the same basic cryptographic properties, but in which you can't eject bad actors --- or even monitor them with a mandatory transparency log. It's strictly worse than this.
It has the property of aligning cryptographic authentication with the domain name hierarchy. Yes, you trust the hierarchy. You trust LE too, so what? We need to move away from any authority being able to authenticate anyone, and DNS naturally aligns with the goal.
You've been vocal for a long time against DNSSEC deployment too for no discernable reason.
Why are you against every single thing that tries to make DNS useful for securing anything?
Do the words "no discernible reason" just mean something very different to you than they do to me? Because I've written somewhere in the vicinity of 100,000 words about the problems of DNSSEC over the last 20 years. My written DNSSEC critique predates HN itself.
This particular issue has nothing to do with "aligning authentication with the name hierarchy". The exact same thing that happened to EMudhra could happen to any TLD operator. The difference is that we have essentially no recourse when that happens, and far less ability to detect malfeasance.
Low TTLs aren’t incompatible with the ideal of using DNSSEC though? From what I can tell DS records typically have TTL of 1 day in the root domain.
What is a difference, versus DANE, is that TLS uses CT. Compromised TLDs happened recently but was found out through CT. A compromised TLD still kills the whole system for a time regardless.
I think DNSSEC still has some use in enforcing email TLS and for CAs.
I meant that for a compromised TLD key, the DS record at root lasts for a day. So I didn’t follow how the compromise in a theoretical DANE-only world would be inherently long-lasting.
You said “no recourse,” which I took to mean no recourse even if known. I don’t follow that. The key/DS record at root can be changed. There may not be a “TLD death penalty,” but the key could be changed.
Now without CT it may be tougher to know about the compromise. For websites that don’t have the luxury of certificate pinning, a compromise of TLD kills the whole thing, at least for a time.
None of my arguments are that a compromised DNSSEC key would be inherently long-lasting?
The "recourse" issue is that after you resolve a DNSSEC misissuance, you cannot then murder the TLD operator that did that to you. But that's exactly what's about to happen with EMudhra.
Again, I don’t get this argument. A compromised TLD absolutely kills any validation (other than EV) already. An A/AAAA can be posted for page validation or TXT for DNS validation.
As far as I know, no TLD operator has gotten the death penalty. It is technically possible for root to change the IP addresses, but it would be bound up in whatever the governance is of the domains.
But the downside isn’t something new that would just apply to a DANE world. That downside still exists in the CA world. The CA is just an additional point of failure, along with the TLD.
I don’t think you’re grokking what “death penalty” means here.
If a CA does something stupid or malicious, the browsers remove that CA from their root stores, effectively ending that CA’s existence as a going concern. People with websites go buy replacement certs from a new vendor and life goes on. This has happened before and is a massive incentive for CAs to not fuck around, and it incentivizes everybody to lean into improving the security of issuance.
It’s basically impossible to do the same thing to a TLD. If .io goofs around and is negligent in handling their security, you can’t do anything unless you’re willing to make every .io website rehome to a new TLD.
CT is a red herring. There is no reason something like that couldn't be used for DANE as well if you wanted and Web PKI didn't have any CT either until recently. It's a non-argument - at best it betrays that this is all about giving root stores the power to unilaterally demand compliance with the policies they want.
DNSSEC creates more operational issues than it solves. It's easy to screw up, which means your domain completely stops resolving when that inevitably occurs. None of the FAANGs use it for their main web sites, so what does that tell you?
It can also be disabled at the whim of anyone operating a resolver, as was seen with Cloudflare during the .de TLD DNSSEC snafu back in May.
Maybe if people implemented name constraints and the CA/B forced CAs to use them. But not as it is; unconstrained sub-CAs are a scourge on the Web PKI.
I like constraints too, but listen carefully to EKR and 'agwa on this, because a lot of things that seem like straightforward fixes are much more complicated in reality. The most important thing I want from the WebPKI is expensive entry and the death penalty for fuckups, and I think we basically have that now.
Name constraints are intended for organizations which want to issue publicly-trusted certificates for their own domains. Name constraints are already widely supported and name constrained sub-CAs exist for this purpose in the WebPKI.
eMudhra is not such a case. They are a root CA that issues certificates to the general public all over the world. In fact, they are now the preferred vendor for numerous US universities via InCommon[1]. There is no way to meaningfully name constrain them.
Can you share any insight about how an Indian CA became the preferred vendor for US universities despite the existence of LetsEncrypt? I know this is kind of a dumb question.
I had a friend who worked in university IT and I got the impression that there was a strong desire to use "supported" products (in general, not just certificates) even if it cost money and was worse than free alternatives.
Interestingly, InCommon used Sectigo prior to this year. I'm guessing eMudhra costs less.
I often run into larger entities that would prefer to use $expensive_cert_vendor instead of the automated certificate services built into the cloud platform they're already using (AWS, GCP.) They really enjoy those manual IT processes or using some third party thing to achieve the same result.
WebPKI can't eject the bad actors that would cause trouble in DANE either. If your TLD conspires against you, they can certainly establish domain control and get a properly issued WebPKI cert.
Visibility through Certificate Transparency logs would at least make it a matter of record that it occurred, though.
> The point is that you can't kill a TLD operator.
I agree, you can't kill a TLD operator. But WebPKI relies on TLD operators to establish identity, so I don't see what we gain from WebPKI, other than mandatory CT logs, and not having to faff about with DNSSEC :P ... both of those are pretty valuable, of course.
I'll put it another way: why do we want a CA to tell us that someone controls a domain, when the TLD could tell us? The TLD is authoritative with regard to who controls a domain; CAs can only report on what they have observed.
Today's incident isn't a domain binding issue; it's a key mishandling issue. Elevating the role of TLD operators here just increases the number of organizations we have to trust to handle keys properly; worse, as we agree, those new keyholders can't have their authority revoked.
In the main, attacks on the DV process aren't protocol attacks; they're simple ATO attacks on registrars. The DV part of the WebPKI is certainly gross and could be improved, but evidence from the last 20 years says pretty clearly that the DV process itself is not the most urgent risk in the system.
My first hope is that we continue to ratchet security requirements for CAs upwards, and in doing so wash more and more incompetent operators out.
My objection is your choosing to build mechanisms that force us to make sure your "we" can "continue to ratchet security" instead of building mechanics that distribute trust and enforce it technology.
I don't care if you write 100,000 or 1,000,000 words: there's nothing wrong with DNSSEC or security that rides on it in principle except that it distributes trust decisions that have been centralized in those objecting to DNSSEC.
It is very bad for the internet to depend for its security on a tiny number of individuals to make global judgements about who's a bad actor.
How so? In the current regime, I can obtain a certificate automatically via ACME. if I control the domain, so to retain control over myexample.com, I need to trust the DNS infrastructure up to my TLD and trust every CA, including the "my secret key is return 4" crowd
In a DANE world, I need only trust the DNS authorities, and there are ways to boost trust in, e.g. the dot-com key management. (Could even go full ring signatures if we wanted.)
Two problems with this: first, it describes a spherical cow universe where there is only DANE; on Earth 1, widespread deployment of DANE would add to the number of trusted authorities, not restrict it. More importantly: it assumes a universe in which everyone leaves .COM, which is controlled by the US IC, and somehow finds a TLD that can be trusted both as a domain operator and a CA. When that institution is inevitably suborned by the IC, there will be no recourse, except to migrate like a hermit crab to some new TLD and wait for it to happen all over again.
Meanwhile: this CA is likely to be killed outright for this fuckup.
What stops our adding a record that requires DANE and a conventional CA? (Preload it like HSTS, if you'd like, to prevent US IC downgrade attacks if people stupidly configure DNSSEC to fail open.)
This way, fuckups like this CA can't hurt me because they couldn't break DANE and IC couldn't hurt me (more than they can already) because WebPKI would have to approve too.
I don't think the CA death penalty addresses the problem. Just as in the physical world, the death penalty isn't a perfect deterrent.
The fact that DANE doesn't work on all network paths and it's hard to predict which ones will fail. You have a preference not shared by ordinary computer users, that broken network paths should be loudly broken and not best-effort.
If you care this much about your root set, consider having Claude roll you a Chromium or Firefox with a user-controllable set of roots that you ruthlessly prune and manage. Unlike DANE, which simply isn't going to happen for a whole variety of reasons, your personal root program (I gift that term to you if you want the coinage) actually could help people with your preferences.
Lol the TLD operators are the literal authority over that domain. Of course you can't distrust them without killing the domain. That's not an issue at all, that's how the domain name system works.
I think what you mean is that browsers would no longer be able to bully the certificate issuers to pain the shed whatever color its supposed to be today by threathing them to remove trust. I think that's a good thing. Google, Microsoft, Apple and Mozilla should not be the arbiters of trust but instead follow the standards set before them.
People have debated DANE endlessly here, so I'm not sure how much more I have to add at this point. However, browsers, at least, do swiftly handle revocation for CAs, as indicated in this comment: https://news.ycombinator.com/item?id=50167244. They also handle EE revocation to varying extents, with Firefox having complete coverage and other browsers having partial coverage.
It's been obvious for a while now that root-of-trust schemes create juicy targets, and that juicy targets attract all sorts of bad behavior that wouldn't be economical if we used a web-of-trust.
Building a web-of-trust scheme that works might not be easy, but also it's not like we've been trying exceptionally hard lately either.
Web-of-trust doesn't work. Even before the PGP web-of-trust broke down several years ago, analyses of how the web-of-trust actually ended up working out showed that it basically boils back down to CAs: people rely on a few highly central keys to do most of the signing rather than signing keys themselves, and these keys effectively function like CAs.
The broader issue is that users don't want to (and probably don't have the ability to) actually make any trust decisions; they much prefer to delegate all of the decisions on trustworthiness to somebody else. The PKI root CA system does have its flaws, but on the whole, I suspect that people like Mozilla are better stewards as to whether or not the person vouching for somebody else is trustworthy than the average or even the above-average user.
Web of trust has been working between humans for over a million years. If we've failed to make computers do it, it's probably just for lack of trying.
To your second point, that might be true in a world where the consequences of making a bad trust decision was some kind of hassle related to changing passwords or reporting your cards stolen. But we seem to be moving towards a world where it lands you in a detention facility or some other dire consequence. I think that sooner or later, users are going to realize that they want that control and will be willing to do the corresponding trust hygiene work. Besides, isn't making that kind of work easy the point of technology? We just need to do better re: making it usable rather than investing in systems we know to be unsafe.
Sure, but what do we do from here? Attempt to augment it with technology so that it starts scaling again, or give up and replace it with something centralized?
> Web of trust has been working between humans for over a million years.
I'm not sure that's really true. Human social organization has, until around 10,000 years ago, been centered on small groups of people where everyone literally knew each other. The ethnographic studies on non-state complex societies generally tends to show a lot of kinship-based networks that developed, and even in state societies that are facilitating long-distance trade, trade patterns and introduction into new areas is heavily reliant on membership of these kinship groups. State societies generally deemphasize these kinship groups, eventually doing away with them like in modern society, but the mechanisms for developing social trust are based on a very centralized development of the legal system--the government essentially enforces that people have to be trustworthy, or they face the wrath of government otherwise.
Essentially, in short, the idea that humans have regularly been trusting people based on asking their close compatriots who they can vouch for doesn't seem to accurately model how trust develops in societies for most of human history--you either have vouching based on direct observation (in small societies), vouching based on identity (larger non-state societies), or centralized authority vouching for people (state societies, including today).
I don't think people do trust others on the basis of the government vouching for them. The government may like to think so, but when Dr. Fauci told us to wear a mask we didn't do it because the government said he was trustworthy, those of us who did so (if we didn't have the education to validate it ourselves) did it because we have a friend or a relative who works in healthcare and they said that Dr. Fauci was right on this one. People don't trust what they see on Facebook because a root certificate indicated that it was served on a page that Facebook controls, they do so because their friends liked the thing. We don't trust that the food in the supermarket has been kept at a safe temperature because the health department claims to have inspected the facility recently, we do so because we know somebody who works at the grocery store and we know that person would tell us if it were not safe to eat the food there. We rely on the grapevine heavily.
That's not to say that we can dispense with these institutions, but they are not whence the trust comes, rather they're a sort of nucleation site around which the peer to peer trust we've been relying on for hundreds of thousands of years can coalesce. The institutional structures are an index of things that we may or may not trust, their utility is that the index lowers the information processing burden of doing so in either case.
The friend-or-foe decision started to become computationally burdensome as populations increased, and it has indeed failed to scale, but it's still in there doing its best, making it look like the centralized attempts are working because often they're aligned.
But when the people in charge decide that they no longer care to protect the people who they're in charge of, that alignment slips, and suddenly the most powerful organization on the planet can't win a war against pond scum.
The trust has always been peer to peer. We can find ways to improve it, so it works better for us, or we can continue the charade that it's replaceable with a central authority, to the peril of those who embrace the substitution.
Ultimately this is a synchronization problem. EMudhra has lost their standing in the web of trust we've been using all along, and because the path between them and the average user is so circuitous, third parties need to step in and revoke that trust in the centralized trust stores that we're using to approximate that web. If our technology better aligned with how our actual trust worked, those paths would be short, blast radii of this kind would be small, and the urgency of action could be assessed more accurately because the actors would be protecting themselves from risks they understand, rather than acting to protect users that they don't know and who don't know them.
Since I haven't seen this mentioned enough yet:
The private key for all 8 "compromised" CAs is a single byte: https://mastodon.catgirl.cloud/@49016/117474217042191790
One of the private keys is literally "4".
I'm putting "compromised" in quotes here, because I can't call it that without cringing and cursing.
[cf. also: https://news.ycombinator.com/item?id=50169109]
I am shocked that one of the private keys was literally “4”. Anyone could have broken into this. I really hope there is an industry-wide effort to stop things like this in the future.... but I doubt it.
There is a variety of projects and researchers scanning for weak keys; possibly that's how this was found (haven't seen anything on that). These certs existed for a year and a half; that's bad but could be worse still.
One of the issues with sub-CAs (well, X509 certs in general) is that they can be ghosts in that there's no way you'll ever get it as a response on a TLS connection, so automated scanners on that won't find it.
CT should cover this (i.e. expose the cert so you can check for weak keys), but I have no idea if/how CT works for sub-CAs. [ed.: see below, doesn't seem to be in CT]
[ed.: these are technically sub-CAs, but rather cross-signed for bootstrapping a new root CA. Thanks @agwa for pointing that out.]
For their own stuff, Mozilla has talked in the past about specifically just only trusting intermediates they've seen. Their modern root agreement says the CA will disclose all relevant intermediates, so arguably they have a complete list and further if that CA "forgot" about a few of them whether that's because your firm is too disorganised ("Oh, right, the one on Barry's laptop, forgot that one") or an actual attempt to sidestep this rule - then we certainly don't want to trust these undisclosed intermediates, do we ? They're either purposefully shady, or they're forgotten about and likely never have been audited.
Y'know… let's just check…
https://ctlogs.io/search?mode=fingerprint&q=42c4accd12c4f595...
https://ctlogs.dev/search?q=42C4ACCD12C4F59597925E1A08A7A901...
https://ctlogs.io/search?q=EBED6475C2C2D80FCB199F3FE8DD6E6B3...
https://ctlogs.dev/search?q=EBED6475C2C2D80FCB199F3FE8DD6E6B...
Nothing for "CN=EM DV TLS CA - G3B,O=eMudhra BV" (not sure which sha256 this uses, 2nd is the spki sha256)
I should be clear here my understanding is/was these are sub-CAs. Apparently they're (trying to be?) roots themselves? The point of logging a root cert in CT is somewhat moot… though tbh I would argue anything issued by a CA should be in CT logs, even cross-signing another CA that is becoming a root itself.
> I would argue anything issued by a CA should be in CT logs, even cross-signing another CA that is becoming a root itself.
And it is, or at least, was. The CT logs when I last spent too long staring at them were complete indefinite logs. These days most (all? almost all?) are time limited so e.g. there'll be Clown 2026 and then Clown 2027 and a certificate is logged to the CT log corresponding to a not-after date IIRC. This way you can throw away a log of certificates expiring in 2025 a few weeks into 2026 because who cares, they've expired, and so they don't just grow forever.
I no longer have a role where I care a great deal about this for my day job, and I do not know how to ask ctlogs.io for the thing I want. So probably this is still logged but I couldn't promise you without getting back into this. Maybe at the weekend.
Could you expand on this? I don't have a full grasp of what you're saying (I'm not super knowledgeable in CA stuff).
My understanding is that if something, like a browser, connects to a SSL site, they will either have or can obtain, one way or another, a complete chain all the way up to the root, so there can be no "ghost" certificate. And I would imagine a scanner could do all those same steps, and thus access all the same certs.
Is this a deficiency in scanner software? You seem to indicate it is a fundamental weakness in X509; I'm just trying to understand what.
Could you maybe describe a specific example of what you're describing?
If you're doing a MITM attack, you can selectively intercept certain clients, and only serve the compromised certificate in those cases only. Automated scanners won't pick that up. None of this actually works though, because browsers started mandating certificate transparency.
Please be careful with your phrasing, as there are MANY other critical things on this planet that use TLS certs which are not browsers, and some are extremely difficult/impossible to update.
VPNs, embedded software (IoT, automotive, aerospace, medical, industrial etc.), business systems, you name it.
In the most extreme case, you can just issue a certificate and not use it anywhere. If it leaks, it's still a security issue, because it's valid.
More realistically, especially with sub-CAs, they might only be used on e.g. corporate networks. TLS scanners may not be able to reach any of the hosts that uses such a cert. As the sibling post points out, clients checking for the cert in Certificate Transparency logs counters that (adding a requirement that the cert be publicly known, which TLS/X509 by itself very much does not have) — but of course this only helps if your client does CT…
Ah okay, thanks. So to be concrete, something like automation running 'curl' on an internal network might see the full chain, including the weak cert, and be vulnerable, but scanners on the wider internet won't see it. I was confused by "...there's no way you'll ever get it as a response on a TLS connection..." — I guess you just meant a _public_ TLS connection. Or it could be otherwise installed on corporate machines' local cert bundles.
Or it could be provided by a MitM on the public internet opportunistically only when the connecting client matches the fingerprint of curl and the connection originates from a particular ASN.
Of all the things... i never expected there to be an XKCD for this: https://xkcd.com/221/
That was inspired by the PS3 signing key PRNG. Sony Computer Entertainment, you have been redeemed.
True randomness respected.
Is this real? This seems so ridiculous it makes me wonder for how long this CA has been in circulation in systems and if it has been exploited by someone previously discovering it in that time (state level actor or not). My old phone running Android 6 does not have any certificate from EMudhra, at the very least.
It's real.
Almost certainly noticed and quietly exploited by a state actor
The Web PKI uses certificate transparency, so we’ll have a definitive answer about whether this is/was exploited soon.
(But in any case, CT ensures that it can’t be “quiet.”)
There were no certificates from the compromised intermediates logged to CT prior to remediation by the browsers.
Which is good because certificates not in CT logs are treated as invalid/untrusted.
But only by browsers. curl, autoupdaters etc. would still all accept them.
There are plenty of ways you would never be able to tell if a fake cert had been generated and used via this attack vector (let's ignore validation attacks):
1. CT does not do anything if a state actor demands a CA in its borders generate a cert without publishing to the CT logs (or a three-letter-agency doing this illicitly). They can put SCTs in the cert, and still not publish to the logs, nobody is auditing the logs for every cert every browser sees.
2. Browsers won't check for SCTs in certs if they come from a private CA. you can MITM any domain through ZScaler all day long and nobody will notice
3. Wildcard certs make it easy to make/renew a cert that looks normal but hides requests on new/different hosts
4. Non-Browsers do not validate SCTs
I feel somewhat confident in speculating that a bunch of people at large companies will, in fact, be auditing the logs following this. That’s their main value; they’re not really an online/active prophylactic.
“They can put SCTs in the cert, and still not publish to the logs, nobody is auditing the logs for every cert every browser sees”
I don’t think you can get the SCT from a log without the log publishing it? That would defeat the purpose.
“Certificates that support certificate transparency must include one or more signed certificate timestamps (SCTs), which is a promise from a log operator to include the certificate in their log within a maximum merge delay (MMD).[4][3] At some point within the maximum merge delay, the log operator adds the certificate to their log.”
> I don’t think you can get the SCT from a log without the log publishing it?
Well you can, but only if the log cooperates.
That is, the SCT is merely a promise to add the certificate to the log by some particular deadline. Nothing technically stops the log from disregarding that promise. But if it does fail to log the certificate, the SCT itself is proof of its misbehavior.
And it's not enough to subvert a single log, because browsers require SCTs from multiple distinct operators in order to consider a certificate valid.
The SCT may be proof that a log was supposed to be issued, but browsers don't submit every SCT they see (did you know "anonymizing" proxies send CT log companies a sampling of all your TLS traffic? fun stuff!), so in practice these logs aren't audited. As far as multiple operators, almost all of them are based in the US, so just pray the USG doesn't want to target you.
CT is great for browsers. Largely irrelevant for anything being fetched outside of a browser.
That’s why I’m talking about it purely in the context of incident response/forensics.
You claimed CT would “definitive”ly prove whether it has or has not been exploited, which is false.
The lack of CT entries only proves that (most likely) browsers were not MITM’d.
I'm not following. Browser or no browser, every cert eMudhra issues has to generate an SCT.
SCT enforcement falls to the client though, yeah? What's stopping someone from signing a rogue cert silently which is accepted by CT-lenient clients?
Right, but you don't have to wait for some hapless random browser to check; you can write your own monitor, and people do that.
I hold eMudhra's private key. I generate a cert signed by their intermediate on my box. eMudhra didn't issue the cert, so it is not sent to any CT logs, and I have no SCT to put in the cert.
Am I missing what you mean by monitor?
I think I might just be talking past you. If your point is "anyone with these broken eMudhra keys can attack users without caring about CT", that's totally true --- it's just that browser/no-browser doesn't factor into it. I'm really only on this subthread because of the notion that CT enforcement requires browsers.
Right, I think I'm with you that browser/no browser shouldn't be a factor. I think I'd like to hear from Rust and Go webpki people on whether we could help the ecosystem enforce CT.
My point is more that as a user of a high-level library like reqwest, I'd like to talk safer defaults when root CAs are in use
Right, OK, I was misunderstanding the point you were making. Thanks!
Sure, but a client that doesn't do CT wouldn't care if they don't include a SCT promise. [ed.: this said "break the SCT promise", that doesn't quite make sense] Which you'd only notice if you see the certificate somewhere else afterwards…
It's all rather hypothetical and unlikely, but you could certainly try attacking something like downloads.openwrt.org¹ like that — especially if you can tell apart the TLS client in opkg from a normal browser.
(¹ luckily, OpenWRT and most other package repos don't actually rely on HTTPS for package integrity and authenticity. And OpenWRT would be the most susceptible due to the "exotic" TLS client stack, if any.)
No, it doesn't. A certificate may or may not include CT info, or include fake CT info. Most TLS clients accept certificates without checking for CT.
I’m surprised to see this username above this comment.
No, the cert can be issued with no (or invalid/junk) SCT and will work fine with clients who don’t check SCT, which is most non-browser interaction.
To spell it out a bit further, an attack strictly in a non-browser context that is reasonably well targeted will most likely go undetected since what the attacker is up against is the chance of inadvertently running into a paranoid user who goes out of his way to perform verification.
Every cert an attacker issues with eMudhra's private key has to appear in the CT logs if they want it to validate in a modern browser.
But if you only want to MITM the connection used by 'pip' to install python wheels - that doesn't check for CT records at all.
Seems like we need to upgrade all software to start using CT records then?
We would still know if certs were issued, though, which is the thing we want to check
How would you know it? If the CA was compromised, why would the attacker publish the fake certificates in CT?
Basically no one outside the major browsers verifies CT logs, so you can still attack a lot of people even without publishing.
The baddies also take a look at the CT logs and look for fresh wordpress instances that haven't yet been configured and will helpfully configure them for you within a few minutes.
No we wouldn’t.
We wouldn’t.
CAs publish to CT logs. If an attacker has the private key, nothing stops them from signing certs without the involvement of the CA or CT logs.
Nothing stops them from doing so, but major browsers and possibly other programs won't trust such certificates. https://en.wikipedia.org/wiki/Certificate_Transparency#Manda...
Yes but a lot of non-browser software doesn't check CT logs, so if an attacker were to target such software they could use an unlogged cert and no one would notice.
This is why there are OS trust stores. Electron - if you use the system CA [0] and cURL[1] apps use it, which are you talking about?
[0]: https://github.com/electron/electron/issues/45674
[1]: https://curl.se/docs/sslcerts.html
The CT log is a property of the individual leaf certificate not a property of the issuing chain. You could have the same CA in the trust store issue an unlogged certificate and non browsers would trust it.
A CT log isn't super relevant (it's not the CRL, it's just a log), but it's maintained/delivered by the CA not the leaf. Never the leaf.
If you mean CRL/Revocation list, everything but the root [you can't revoke a root] is maintained by the tier above. Leaf -> intermediate (these), [Intermediate -> Intermediate*], Intermediate -> Root.
Not sure why you're mentioning transparency logs, they have nothing to do with this (nor help).
Because the thread: "a lot of non-browser software doesn't check CT logs", you replied saying it's a reason to use the OS trust store, not sure how that fixes things? Maybe it would relieve them of the responsibility of pushing out a distrust update?
That's not what woodruff is talking about (I hope). The CT logs allow us to (later) find who/why it happened. CT Logs in no way help you stop something. (Depends a bit on how software reacts to apparently invalid issuance - but since these have the real keys the issuance will look normal).
Revocation lists allow you to stop/block bad issues. Both are maintained/decided (you could post to another service) by the issuer, not the leaf. CRL records would stop this (but some software has decided that's too slow). In general software uses the OS trust-store first (fragmented on Linux). Firefox and Java have their own independent paths. Chrome ignores CRLs
CT logs only tell you something happened if the certificate was registered in the log. But if you are attacking software that doesn't check if the cert was registered, then you don't need to register the cert, and there won't be anything in the log.
My point is you can forge a cert, without putting it in the log, and a lot of software will still be vulnerable to it.
Curl does not check that certificates have a valid SCT attached. Thus it will accept a certificate that isn't in the CT logs. That's mostly independent of the OS trust stores.
CT is enforced in Android 17 by default.
A grumpy state actor today.
Nope. You can find this from the Chrome Root Program.
> Because no TLS certificates chaining to these six intermediate CAs were logged to CT prior to our CRLSet block on October 17, no fraudulent certificates issued by these CAs could have been trusted by default in Chrome or other CT-enforcing browsers.
Not all TLS traffic involves a browser.
organized crime is generally as or more capable these days. they have bigger budgets than most nation states.
They are not exclusive to each other. The most successful criminal organizations work with, and from within, nation states, their intelligence agencies and law enforcement. Oh hi, we see you.
I admittedly don't know much about how CAs work but why would a state actor care about such an exploit? Wouldn't they just ask any local cert issuer and receive a "legitimate" cert?
You could cloak where it's coming from more easily this way.
If FBI went to Godaddy and said, “We need a cert for www.google.com”, Godaddy would most likely refuse. Issuing that cert would leave a trail that would be extremely undesirable for both FBI and Godaddy.
I believe these intermediates chained up to a third party root.
They chain up to emSign & Comodo:
https://groups.google.com/a/mozilla.org/group/dev-security-p...
To be clear, emSign is another name for EMudhra, and Comodo is now known as Sectigo. (You can't just look at the organization name in a certificate subject; you have to consult the CCADB: https://www.agwa.name/blog/post/the_certificate_issuer_field...)
Argh. Thanks for clearing that up.
Since they are also a root CA, are they allowed to audit themselves, or do the browser root programs require third-party audits?
I can't imagine one part of a business being on the up and up, and the guys down the hall running completely rogue.
Third party audits from a qualified auditor are required, although the CA chooses and pays the auditor, and the auditors are basically just accountants. So it's... not a perfect system.
Love this quote:
Inclusion in the Chrome Root Store requires that a CA Owner continuously demonstrate the technical competence and operational rigor necessary to protect Chrome users. Due to the seriousness of this incident, the Chrome Root Program is conducting a thorough evaluation regarding the status of eMudhra’s remaining CA certificate to ensure these standards continue to be met. We will share further updates as our review progresses.
from a post on https://groups.google.com/a/mozilla.org/g/dev-security-polic...
It's been a while since there was a good CA death penalty case.
Given Wayne's hints in the timeline that this issue was discovered by an agent, it might be one of the first big corporate death sentences due to AI?
I bet Entrust never made a cert with a private key of 4.
The problem is the death penalty case is aimed at the wrong target: The entire Web PKI system remains dumb and bad, and the correct answer is to dissolve the CA/B and find people with practical security experience.
Entrust (now Sectigo) is involved with this [0] (cross signed an intermediate) [1] Even last year buying a Sectigo cert got you an Entrust cert with no mention of Sectigo/Comodo[2] (maybe more of a Character.AI situation than a sale)
[0] https://emudhra.we-have-a-problem.com/roots.json
[1] https://www.entrust.com/company/newsroom/entrust-sells-publi...
[2] https://www.sectigo.com/comodo
> Entrust (now Sectigo) is involved with this
This is probably a rough definition for "Entrust was banned from being a CA and sold the business off to a competitor". In part, because I also do not think the root programs would just allow Entrust to continue operating the same business under another brand name, unless they are even dumber than I already thought.
The other solution is web of trust. Which works waaaaay better but is really complicated to implement.
I'm not sure there is a 3rd solution to this.
Slightly funny about this: even Wayne (the person who reported it to Chrome & Mozilla) got confused [https://bugzilla.mozilla.org/show_bug.cgi?id=2083449]:
(…the "cross-signs" use the same keypair of course.)
Apparently they've only been included in early October https://www.mail-archive.com/dev-security-policy@mozilla.org...
I'm not exactly sure which of the roots from eMudra were compromised, but the "emSign ECC Root CA - G3" which is mentioned in the OP site as one you should blacklist, is present on my phone which is running Android 12 with the last security update from three years ago. Having a better timeframe for when the impacted certificates appeared really would have been nice.
None of the roots were compromised. (Some of) Their own sub-CAs for TLS certs, which are also cross-signed by Sectigo (since their own roots are rather recent), are affected.
How you have that root cert on your phone with a 3 year old CA set I have no idea, presumably something updated your CAs in the meantime? (Also, aren't you incredibly exposed to the general onslaught of vulnerabilities from the past 3 years?)
Well that makes sense. The "emSign ECC Root CA - G3" root is listed as being originally issued on 18 February 2018, so would have been really concerning if that itself was the compromised certificate.
I believe Android updates things like CA certs independently of their monthly security patch. I'm not sure Android 12 uses the same system, but on newer Android I think those are shipped via Google Play system updates, which don't require waiting for the phone vendor.
Yeah, in Android 14 they separated it out so that the CA store can be updated from Google Play. Versions below that you're reliant on the manufacturer to ship updates.
its well-known , we took certificates from them for our company. its due for renewal, iam pausing this for now.
chosen by fair dice roll, no doubt
https://xkcd.com/221/
Given that 4 is the value shown in this well known XKCD panel, it almost seems like a (im)practical joke or trolling.
4 is as likely as any other randomly chosen number over a given range.
Well, all eight of the compromised certificates had keys which were < 256. So a really poor choice of range.
How could you prove it was not a fair random generation? It's totally possible.
It's not within the realm of believability. The chance of "human screwed up" astronomically exceeds this being fairly randomly chosen, and the Firefox bug states, (I think this is supposition, not known fact, but IDK)
> The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order.
That is within the realm of believability.
When you have multiple selections you can show that the probability of selecting a set with a range much smaller than the population range is exponentially unlikely, strong evidence to reject the hypothesis that the subset was chosen at random. But when N=1 you can't conclude anything.
Regardless of the probability that this cosmically unlikely occurrence naturally happened 8 times in a row, we all know what the privkeys are now so the certs can't be trusted anymore.
This is actually interesting. I'm no cryptographer. Do modern algorithms take into account that certain values should be avoided because they would be obvious things to try first in a brute force attack?
I have no idea but I can imagine why you'd want to bias things away from principled randomness to avoid picking 0 or 4 or whatever.
The algorithms don't care because it doesn't make sense to care, the same reason a "smallest non-interesting number" wouldn't make sense - now that number is interesting precisely because it was cited as the smallest non-interesting number.
However practical applications do care, and so e.g. the "Debian Weak Keys" are a relatively small set of keys which would have been generated if you used a particular patched OpenSSL to generate "random" key pairs after a mistake patched out most of the randomness. Many of them were used by Debian sysadmins particularly in the window between the patch and the discovery - and so all CAs are expected to check that you're not trying to get a certificate for the public half of one of those pairs.
There's nothing "bad" about those keys except that well, we know some people used them so bad guys will try the private halves of those keys just in case. Likewise there are a few apps which shipped with a baked in "default" private key, and all the paperwork to apply for a certificate but obviously a user may not realise that if they just fill it out with their name and details, they're now vulnerable to attack by literally any other user of that software. They were supposed to generate a random key pair and re-generate the paperwork for the resulting random values. If they don't realise we need the CA to say "Nope, that's a bad idea, not issuing" even though these keys aren't special in any other way than they're given to every single user of that software.
IIRC when I worked for a famous US corporation their middleboxes relied on such a well-known key and at some point I asked "Hey, this seems wrong. Why did you do this?" and as often has happened in my career I got no actual reply to my question but magically the issue was fixed in short order. The first clumsy attempt to fix it broke some things, but they gradually figured it out. Glad I could help, I guess.
If your generation is any good the odds of ending up on such a number should be astronomically small.
Biasing something that should be random tends to badly backfire. See the Enigma who avoided enciphering letters to themselves, which was then used to break it.
That's assuming perfect random generation. If you assume real world where mistakes can happen during generation, 4 is way more likely than 25105785070300501195559896594243230376961191116602934339784363418717203371363
Yikes. I’d have to wonder if this CA should even be trusted ever again?
Definitely not. Their CPS requires that these keys be generated by an HSM and no HSM would ever generate these, so there's little reason to believe they're following any of their other controls.
According to https://bugzilla.mozilla.org/show_bug.cgi?id=2083449 they actually did use an HSM? Though some of the jargon is a bit beyond me, maybe this will finally get me to learn more PKI stuff.
> The hardware used for each of the issuers is in the attached file. These HSMs are FIPS 140-2 Level 3 compliant.
My vague and inexperienced read on their description boils down to "we were doing something weird and non-standard as part of our key ceremony because we intended these keys to be exported, and because of that process weirdness we blundered and gave the HSM faulty instructions."
The most likely explanation is that they generated the weak key externally, imported it into the HSM, generated the CSR from it, and finally had Root CA sign it.
A FIPS HSM does not forbid one to import keys into it.
But shouldn't such a HSM support key attestation (and the parent require such attestation)? Attestation would also track whether a key was imported or generated on the HSM, and of sufficient length.
Attestation is not required by the compliance standard for browser CAs (WebTrust) and it is also quite vendor-specific and not very interoperable, so it is not commonly done.
The Root CA must sign requests after authentication, but a "manual" approval is also accepted if done with the appropriate oversight with witnesses (clearly not the case here).
Of course it's not required, as we see here. The question is why not? Getting a simple code-signing cert seems to have more stringent requirements in that it either requires attestation or CA-issued hardware.
I’ve asked this question before and the answer was that not all HSMs support it and they're too expensive to replace.
> no HSM would ever generate these
You trust the magic black box far more than I do.
My big fear about HSM is that they are inscrutable by design. There is no way to verify they are working as advertised. Plus side, all sorts of fancy hardware to protect your keys. Down side, all sorts of fancy hardware that obscures flaws in the implementation.
I think I'd prefer open source hardware built from off the shelf components glued into a secure enclosure. That way you can destructively audit a few at random if the application matters enough to justify the cost.
While you are right in that it is impossible to verify they work as advertised, it is not impossible to detect certain categories of failures. Like only generating values 0-255.
You could choose to detect small keys, but my understanding is that implementations entirely eliminate the issue by only searching factors with leading bits set (i.e. large numbers).
it should not
Yeah IMO they should just close up shop. Just like Diginotar.
They had one job, and they blew it. The system is based on trust and they've lost it.
I'm going to remove their roots from all my stuff in any case.
Not only should they never be trusted again, the individuals responsible can't either. They need to be blacklisted somewhere. This wasn't garden variety stupidity, this was (IMHO) do-not-collect-$200 stuff.
They used the xkcd PRNG: https://xkcd.com/221/
More details in the now-public bugzilla issue:
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
> The other two certificates are for codesigning and trusted by Microsoft
That seems like a WAY bigger problem than TLS, but AFAIK Windows still validates code signing certs against CRLs before trusting them (can't remember if it soft-fails).
Why is that a way bigger problem? Windows will happily run unsigned binaries. Code signing just decreases the friction for users slightly, but is in no way required. TLS is definitely a way bigger problem.
> Windows will happily run unsigned binaries
This is absolutely false, it's a simple policy change to enforce code signing, any organization that is serious about security can do it.
I'll give you the benefit of the doubt and assume you're another Linux guy who doesn't understand Windows security.
The concern of course is with malware being signed, which is far more likely to be abused than some TLS MITM fantasy.
> The concern of course is with malware being signed, which is far more likely to be abused than some TLS MITM fantasy.
You already need compromised systems to install malware, signed or not. Not a real problem.
For TLS MitM you just need to use a shared network, wired or wireless.
> You already need compromised systems to install malware, signed or not. Not a real problem.
Is this true? Ads pointing to a malicious Notepad++ is already an existing thing. Now their malicious version will pass signing checks. Users will happily install these without the system itself being compromised.
Plenty of major programs are not signed, and thus plenty of organizations don't enforce code signing.
For the exceptions, you can sign your own using internal CA.
And keep signing it with every release. No thanks.
Organizations that are serious about security should have a whitelist of binaries that can run on their computers. Enforcing code signing isn’t good enough.
> Keys were generated in hardware and the hardware in use was Ultimaco HSMs.
The plot thickens.
Broken HSM? Misconfiguration? Confusion? Straight up lie?
> Excel attached. The Ultimaco devices were used to generate the ECC, RSA key pairs for the issuers. They were not used to generate any other key material. But it doesnt look like the device is at fault.
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
…
Reading further, it seems like they say they first generated some keys on-HSM, but then used 'test keys' (that were used to test wrap/unwrap of key material) instead by accident. Fishy.
They used the "correct" key for signing the CRLs (which of course was then an invalid signature):
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
And THAT is where I'm drawing the line between "honest mistake" and "gross negligence". For that to not raise giant red flags is an unacceptable failure in process and work culture.
Unreal. Absolutely unreal.
I've seen speculation that there was a language barrier, given that this is an Indian company.
The theory was that someone misinterpreted 384-bit keys to mean the key must be a number between 1-384.
I think that's entirely made up, but it wouldn't be completely implausible to have happened while someone tried to troubleshoot why the first attempt failed.
> an approved deviation to our usual KGC to test
The whole point of a key ceremony is that you're not going off-script and YOLOing random crap.
Ultimaco HSMs! Well Ultimaco today published a marketing blog post entitled Global Encryption Day: Encryption Is Only as Strong as the Keys Behind It. It seems like a subtle dig at their customer’s incompetence.
https://utimaco.com/news/blog-posts/global-encryption-day-en...
> "The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys."
Ah yes.
"The system says something is wrong, but it's also 4.30PM. Let's just re-sign the thing and go home."
Good luck recovering from this.
What a change a month makes. Public discussion on eMudhra getting their own root CA inclusion ended and Mozilla had them slated for approval.
https://bugzilla.mozilla.org/show_bug.cgi?id=1889859
> One of the private keys is literally "4"
https://i.sstatic.net/Y3zE3.gif
Anyone have a mirror for the first link? It served me a page assumedly intended for AI scrapers and then promptly blocked me when I refreshed a few times.
> One of the private keys is literally "4".
Oh no, they reverse engineered the random number generator!
[0]: https://xkcd.com/221/
I BET this is an error code. they wanted to generate some key via something and it errors and put error code 4 as key and since encryption is dumb and people are people, no one looked at the key material.
edit: The reported private scalars were 4, 56, 66, 115, 155, 156, 230, 237 across the eight certificates so its unlikely i will win my bet :'D
That was a blasting key generation ceremony.
Chosen by fair dice roll
More than that, they claim the key was in an HSM, so they likely could not look at it.
i wonder how many keys they thought it had. reading them byte by byte xD 'this HSM is excellent we can litterally put _all_ our keys in it!'
and thx for the context. i rly wonder now how it went down lol
Someone answering “questions for emudhra” also say the keys were intended to be exported to US and EU data centers. I don’t know if that would have precluded inspection (hopefully it’s encrypted some way, but why not have the destination HSM generate the CSR?)
Edit: I see the below paragraph is quoted extensively below. I hadn’t seen those comments yet. Oh well
Source: https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c11
Content:
We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
The RSA issuers went through a different path as per our assessment of the code and are therefore not impacted. Some additional context - these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one).
Wrapping/unwrapping means the HSM would generate the private key and then encrypt it with a key only known to the other HSMs. Those other HSMs would then decrypt the key on import. The private key should still have been impossible to examine outside the HSMs.
That could be wining numbers for a lottery
These read like Lotto numbers and I bet someone in the underworld has made bank with them.
Random number 0-255
Hey that's a byte of entropy! And one of the rare cases where choosing a larger value might offer you better "protection" against serialized brute force attacks
Doesn't even seem to be all that random within that range. For example all digits within each number go left to right on a keyboard.
The first 6 keys are literally someone typing them in lazily on a keyboard. You can almost see their fingers barely moving across the top numeric row as they generate them.
You can see the thought at 230 of "Ok, that was dumb, gotta move my fingers on the numpad slightly this time to throw them a curveball"
An elliptic curveball even
My guess is that someone was instructed to generate "random bytes" and just did 'char secret = rand()' or something similar.
Looking at the pattern I would actually suspect a numberpad over top row of the keyboard. That explains how the responsible party got 1-5-6 without exhausting their delicate digits.
Maybe somewhere in the code it says
instead of
This should be higher. They mentioned a "coding error", and getting valid keys from the HSM and then truncating the string to a single byte through buggy array handling (see above) is exactly how you'd end up with a bunch of integers in the range 0-255.
This is the C explanation perspective, but a similar thing could happen in Python, if you write `s[0]` instead of `s`, perhaps due to a refactoring that changes an array of keys to a single key. This is because Python does not distinguish characters from strings. (Not an indictment of Python; just another proposal for how this could happen, and I would say it is more likely to use Python than C in some hacked-up signing script.)
That should still produce uniformly random 8-bit numbers but these keys don't even look all that random in that range.
Well, my example is C++, it relies on function overloading existing for, for example, to_string(uint8_t[]) and to_string(uint8_t) separately, which means this error would come down to overloading instead of catching the type error.
Just saying, because C++ is very popular everywhere in this "kind" of company, if you catch my drift.
Vanity private keys
"The private key for all 8 "compromised" CAs is a single byte: https://mastodon.catgirl.cloud/@49016/117474217042191790"
Compromised _keys_ not compromised CAs:
"The private keys of eight eMudhra intermediate CA certificates have been compromised."
"The private keys for the intermediate certificates were incorrectly generated by the CA."
The certificates with incorrectly generated keys have been "revoked" but will the eMudhra CA remain in business
Mastodon link contains gibberish supposedly as an anti-bot measure
Looks fine to me. What are you seeing?
For fun I gave the public key to Opus 5.5, telling it that this is a puzzle, and it should solve the puzzle without web search. Opus 5.5 told me the private key of 4 after two minutes, including the time to write and run a 30-line Python script implementing a baby step giant step search for discrete logarithm. As far as I can see, it did not use web search or use any command that would have internet access.
try an offline model. they are unable to cheat.
To be fair, the fact that you gave it the task at all is a clue that you know the answer might be uncharacteristically simple.
Yup. And Opus 5.5 did exploit this fact. I framed the task as a puzzle and Opus picked up on that framing.
Do we call this human's slop? Should we be so hard on AI when humans do something like this? I'm pretty sure LLMs did not and would not allow such things to happen.
On the bright side, hopefully with AI we won't see silly slops like this.
The impact of this incident is very unevenly spread.
Over 10 years ago, Mozilla shipped OneCRL [1]. We remediated this incident in a couple of hours for Firefox users who were protected automatically without the need for updates. Chrome, Apple and Microsoft have similar tooling. Since then, we've built more advanced tooling to handle revocation at scale for leaf certs (CRLite [2]).
However, for the rest of the ecosystem, there has been little to no investment in revocation and for those clients, this is a massive problem. Hopefully, we'll see folks invest now. Efforts like upki [3] look promising.
[1] https://blog.mozilla.org/security/2015/03/03/revoking-interm...
[2] https://hacks.mozilla.org/2025/08/crlite-fast-private-and-co...
[3] https://github.com/rustls/upki
More on upki:
https://discourse.ubuntu.com/t/an-update-on-upki/77063
My interest in building OneCRL support into upki has definitely increased this week. Issue:
https://github.com/rustls/upki/issues/229
I'm hoping some organizations will step forward to fund work on this.
(Note that curl on Ubuntu 26.10 now has CRLite support baked via upki -- unfortunately that doesn't help with this because AIUI CRLite is only for end entities, not for intermediate certificates.)
You mean the response was uneven. The impact was a complete and total potential compromise of literally everyone. Every domain. Every client using one of the public root programs. Every user of those clients.
I mean, it's an emergency, but the acute severity is debatable; many times a month the major browsers quietly deal with vulnerabilities that yield attackers direct control of victim computers; here, attackers get the ability to spoof websites if they're in a position to hijack connections.
The middlebox situation, with non-updatable roots, is very annoying. But middleboxes also have unfixable RCEs, so...
It's totally fair to evaluate different projects by how well positioned they are to update their WebPKI trust anchors. I'm just saying, keep perspective.
So it other words, it allowed attackers to do the one thing that Web PKI is supposed to prevent.
Being able to quickly update trust roots is at best a hack to paper over the issue when there is no good reason why every podunk CA should have authority to generate certificates for every domain in the first place. It's an inexcusable design flaw. Or a feature that gives power to root stores at the expense of everyones security, from the perspective of the root store operators.
Ok, now do that same paragraph, but for drive-by RCE vulnerabilities.
Nobody thinks this is a good thing. It's bad. Hopefully, eMudhra gets distrusted. We've done that to bigger CAs.
Keep in mind that a _lot_ of software out there uses TLS as its only trust anchor for updates, commands or similar.
This can and will be weaponized for take-overs of systems. They will be of the consumer kind, but they will also be of the industry kind. Let's pray that they will not be of the state or military kind.
Firefox Mobile 158.0 on Android 17 (Pixel 7) connects to all the demo sites with zero warnings. Any idea why the "automatic protection without updates" doesn't seem to be working with that combination?
Pixel 10 with 158.0.1 still connects to the emSign ones, even though the patch notes for 158.0.1 mention that it updates the CRL for those without 'Remote Improvements' enabled?
edit: Disabling all the eMudhra CAs in Android settings does nothing, cool?
edit: I guess there's no way to manually ask for CRL update, cool??
edit: Gave up and reinstalled Firefox. Seems to be fine now.
I am getting the same on GrapheneOS on a P10.
I fully expected disabling them in the phone settings and rebooting would have fixed it, but no?
Don't really wanna reinstall FF and lose all my configruation.
I'm on the same Firefox version on android and they are blocked now
On my just updated Firefox 158.0.1 on Android they are not blocked.
Edit: however on Firefox Focus 157.0.1 they are blocked.
I am seeing the same thing in the beta app (159.0b4) on an identical device. The sites are blocked correctly in Chrome.
Update: properly blocked now. Seems like it takes a few hours (days?) for the CRL update to propagate.
I'm having the same issue. It seems that their CRL updates aren't working well on Android?
A fix is being released. Thanks for reporting! https://blog.mozilla.org/security/2026/10/20/protecting-fire...
Wiping local app storage fixes this.
You mean you and Chrome stopped supporting normal CRLs and OCSP, replacing (for normal users) these distributed-trust technologies with central lists controlled by browsers
There is a good reason they did. CRL and especially OCSP are both terrible and not particularly effective
OCSP stapling works fine technically, which is one reason it's still available in both browsers via enterprise policy. LE dropping support for it and browsers having insufficient courage to hard-fail doesn't make the model broken. It instead makes the ecosystem broken.
Just another step towards a tiny number of browser vendors and infrastructure companies controlling the internet.
Here's a question for Mozilla et. al.:
It's cute you have a magic DIY CRL. You accepted these assholes into your root program.
They lost the KEYS.
What is the screening process for CA's that can't be bothered to use a HSM?
Even Honest Achmed wasn't this careless.
IIRC up until very recently the screening process was "they report that they're good boys."
The screening process has been more involved in that since around the late 2000's. I wouldn't call 20 years "very recently," especially in the context of the history of the web.
circa 2007 I was looking over a college textbook on computer networking that I saw at a family member's house. It still said "While networks are inherently unreliable, it's not a stretch to imagine a future where one can stream video on-demand over a WAN like the Internet"
circa 2007, Youtube had been around for two years already.
Yes, that was my point.
They didn't lose the keys, a sub-CA did.
Which is to say, sub-CAs in itself are a giant loophole that needs plugging. Especially since AFAIK nobody is bothering with name constraints (RFC5280 section 4.2.1.10).
The situation is even worse than I thought.
They hold intermediate certificates issued by TWO different root CAs, neither of which they operate.
So two separate entities didn't bother to read their certificate policy or inspect their operations. Have we not learned anything from past mistakes?
Roots should be responsible for all their sub-CAs, full stop. If they don't trust them enough to accept that responsibility, then the system is broken.
If you mean Comodo and Setigo, they're the same company, pre- and post-rename.
Let's just say they've had issues with signing sub-CAs before where those sub-CAs went on to misuse them before… looks like it was around 2011?
This CA business is a scam. You cannot buy trust with money.
The reason nobody bothers with NCs is because they don't work very well in practice. They also have their own baroque matching rules that are not intuitive if you're from the land of UNIX globs.
(In fairness this is a self-fulfilling prophecy: a lot of validators handle NCs poorly or not at all, so there's no impetus to rely on them. But that's not something we can really save in the Web PKI's current state.)
> But that's not something we can really save in the Web PKI's current state.
I agree with your assessment but it's not an excuse to give up without trying. I hope this is just because I'm not looking in the right places, but I don't see people trying.
What really makes me salty is that there was enough energy to mess with the rules around EKU, the primary effect of which is that non-browser software is now starting to just ignore the EKU bits. Even if it would've amounted to nothing, I'd really rather that effort had gone into NCs.
I think a lot of people have tried! The reason EKUs stuck where NCs haven’t is because EKU rules are simple (in relative terms) and OpenSSL didn’t make as much of a mess out of them.
I can't help but read your post as "OpenSSL didn't make a mess of EKUs so the CA/B had to fill that role instead" :/
(I really am salty about the EKUs. In a few years the EKU field will just be useless, due to operational annoyance driving devs to ignore it. Except in browsers of course.)
eMudhra is also a root CA operator. One root CA operator cross-signing another root CA operator is a good thing actually, and it's what allowed Let's Encrypt to work before their root certificate was widely trusted.
Issuing sub CAs to non-root operators has historically been a big loophole though it's not as bad these days since multiple root programs now require approval before CAs can do it.
Didn't Let's Encrypt have to jump through all sorts of hoops to prove they were competent and trustworthy before that happened?
Now it's coming out these guys used "4" as a private key. That's a crazy amount of incompetence and malfeasance. Did they vibe-code a CA? HOW DOES THIS HAPPEN? I am dumbfounded.
You couldn't even do this if you installed Microsoft Small Business Server 2008 today, unpatched, with the CA on default settings.
eMudhra had to jump through even more hoops.
Obviously the system failed here, but it's a failure of a root certificate operator, not a sub-CA operator as claimed by eqvinox.
>What is the screening process for CA's
it's done in the open, and you're even allowed to voice your cute opinion during the multi-week public commentary periods.
here is an overview https://wiki.mozilla.org/CA/Root_Inclusion_Overview
It's very inconsistent for me even on just Firefox. On multiple systems running different instances of Firefox, checking with https://emudhra.we-have-a-problem.com/:
One blocks everything; one blocks 6 out of 12; one blocks the other 6 out of 12; one blocks nothing.
Explanation from eMudhra in the Mozilla bug report https://bugzilla.mozilla.org/show_bug.cgi?id=2083449:
> We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo)
They don't blame the HSM.
Also there was another incident where eMudhra CRLs were being signed with invalid signatures:
> The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys.
> these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one)
This bit sets my hair on fire. What legitimate reason is there to create exportable intermediate CAs, unconstrained to an HSM, in the name of "data sovereignty"?
India has state-mandated spying on network traffic (69B & Telecom Act 2023) so as dumb as this looks, it's possible this could've been purposeful
This is a really, really dumb way of doing online spying (presumably, you mean by running a program of deliberate misissuance) since every cert they issue is recorded in a public cryptographic ledger.
I'm not saying this will never happen, just that rolling out a dramatic infrastructure change for "data sovereignty" just to pull off a SIGINT heist that is almost certain to get your CA distrusted seems like a poor risk/reward calculation. I fully expect IC agencies to get some CAs killed and be fine with that.
Yep, but it was an easy zero-touch MITM vector for billions of devices that's only now being closed down
I'm only pushing back against the idea that a "data sovereignty" rollout of intermediates was cover for an IC operation. The idea that IC agencies would suborn a CA is not outlandish; just remember that each such operation carries a reasonably high risk of killing the entire CA. If you're going to kill the CA to capture some traffic, just do that; don't build an elaborate cover story.
In any case, not much sovereignty to gain from only 8 bits :^(
> rolling out a dramatic infrastructure change for "data sovereignty" just to pull off a SIGINT heist that is almost certain to get your CA distrusted seems like a poor risk/reward calculation
Well, meet the Indian government that does these calculations all the time.
Some examples:
https://www.thehindu.com/news/national/neet-ug-re-exam-teleg...
https://www.medianama.com/2026/07/223-india-bitchat-ban/
https://www.drishtiias.com/daily-updates/daily-news-analysis...
https://vajiramandravi.com/current-affairs/bill-to-curb-inde...
Well, I honestly believe it is rank incompetence. Speaking as someone who's monthly electronic GST filings, commercial authentications, and digital payments, i.e. continued livelihood, is beholden to cert and PKI authorities like eMudhra having their shit together...
As for the odds of this being state-sponsored... It depends on the state's incentive, weighed against its continued behaviour, over the last 10-odd years. Everything below is easy to substantiate with citations. There are too many to cite... I put myself at real risk just by stating any of the following under my real name.
I expect eMudhra-esque collossal gaffes, as a normal course of life, stemming from the apathetic, insular, feudal-minded, shoddily run bureaucracies that the majority of our government and quasi-government organisations are.
Basic day-to-day operations being unavailable, because "Sorry sir, site is down." is a normal everyday life experience of their own staff, not to mention the common person. eMudhra is one of that tribe, and I'd say they are not the worst offenders of the lot, by a wide margin. I'd probably give the "best of the worst" title to our national Employee Provident Fund.
Of course, one can't totally rule out nation-state-level stupidity, such as BGP poisoning by the Pakistanis.
If our current federal administration has incentive to do it, they will... And, given their track record, if it does turn out to be nation-state-engineered for short term tactical gains (to hell with the consequences for anyone else), all I will do is sigh and move on with my day. Cornered politicians can go to any length to save their own skin (usually, just to save face)... Collateral damage be damned. Popular examples include world superpowers bombing distant nations, for whatever gain or goal (usually short-term, because well you most probably won't be President a second time (and never a third)), using whatever pretext they can sell to get others to also get behind (the individual's psychological cover in a firing squad, scaled up to many allied powers).
In India, absolute majority governments have amply demonstrated the same failure mode as US heads of state empowered with de-facto fatwa powers, as of the US PATRIOT act. Given motive and incentive, they easily turn the full might of their uncheckable wrath and ire at our constitution, as well as directly at the common public.
You know, like sending out military forces to "calm down" the pesky student protesters acting against "the national interest".
As we speak, massive nationwide student protests have been bringing shame and angst to the government of our ruling party at the centre (BJP and allies), as well as their various governments across states. Many protests have happened at this scale in the last decade+ of the same ruling party's unbroken streak as a majority power in the centre (federal government). Those other protests were like water off a duck's back. The ongoing student protests, for reasons I've not dug into, appear to be a clear and present existential threat to the incumbents.
My gut feeling is... the incumbents aren't passing the electoral vibe check any more.
The sentiment on the streets of the national capital, and large urban centres nationwide---almost entirely comprised of a new crop of young voters (18 y.o. up to early 30s)----is that they feel robbed of an economically viable future. They've organised at scale (millions on the street) with persistence (they are literally owning the "Cockroach" epithet thrown at them by the powers that be). And in short order, material consequence---unprecedented in its decade-long winning streak so far---befell the incumbent party's administration. The Education Minister had to resign, and they've generally lost face with that whole voting-age cohort. Brand-name loss of key leadership is further disrupting their internal party cohesion. Plenty of hopefuls in there, salivating after the top dog position, but thus far too weak to sway the cult following of the public faces of the party. Even an hint of weakness gives the wannabes a real opening to try. Textbook demagoguery.
If that is indeed true, then in their place, I too would seriously worry about losing even more ground in the next national election... It's not far now. Because? "Winning by overwhelming majority" has been the one and only priority of the incumbents, driving all their work post-election (where work includes constitutional shenanigans)... Being a populist party borne of, and cultivated by far-right nationalist ideologues (the RSS and "Sangh" community), with a longstanding desire for religious supremacy (Hindutva and Sanatan dharma), they publicly and loudly dream of a hundred-year golden age. Uh, no, now I think it's become a thousand-year era (really!). That's not totally insane if you're suddenly goated at winning electoral politics, because our head of state can enjoy unlimited tenure. Yes, you heard that right, India has no constitutional limits to number of re-elections to the Big Chair, whether successive or alternating.
Without looking at absolute numbers, I feel confident that the influx of new voters (lots of people can turn 18 in a single government tenure) far outstrips the older ("whatsapp uncle / aunty") voter base that the incumbents pander to. This has been a lucky break for the fumbling Congress party because their legacy name is finally regaining some cachet... The much-mocked Rahul Gandhi presents like someone the young'uns can actually talk to, like a human being.
But the Gandhi-brand Congress were once an absolute majority power too. At which time they suspended our constitution, and weaponised federal agencies, and created the playbooks that all future governments have been able to deploy at various scales. I expect nothing less from any other party that grabs absolute majority. Currently it's the BJP. Tomorrow, it could be anybody else. Thus far, it's been the secular-presenting Congress, followed by the Hindutva-majoritarian BJP. Woe betide us if next the Communist Party of India gets a bite at that juicy apple. 'cause, like, I don't have to explain what that sort of government can do with absolute power. <shudders>
Ref: our liberally interpretable Information Technology Act. Among other things, it enables and empowers arbitrary blanket Internet Shutdowns and Blackouts, by local states as well as national authorities, to "protect the peace". In some deeply conflicted areas, like Jammu and Kashmir, censorship measures under martial law would give both, the Great Firewall of China and the Stasi a run for their money.
Ref: at-scale weaponisation of Pegasus, as of 2021. Federal and even state authorities like police departments have weaponised nation-state electronic warfare technology with impunity. Usual suspects / targets are politically inconvenient individuals and groups... opposition politicians, bureaucrats with a spine or conscience, students, media, artists, NGOs etc...
All this mixes badly with pre-existing supercharged Sedition laws. These effectively obviate habeus corpus, and are routinely used to prosecute speech interpreted as anti-national / seditious under the generous ambit of our IT laws (we don't have a real privacy law). Mere suspicion and/or fear and/or dislike + Pegasus + garbage IT laws = authority can brazenly plant incriminating evidence on your device... it's reportedly been done multiple times.
People have been held in prison for less (just inconvenient speech), without possibility of bail, for months and years on end, without trial. The main goal is to cause irreparable reputational damage, to create future deterrence... It's pretty effective. A secondary goal is to take the stubborn ones off the board, with no legal recourse, until they break. And of course, to permanently defang the truly outspoken and indomitable few... Some of those people have been in high-security federal prison for years on end now.
In conclusion:
I'm confident all federal administrations of India, current and future, will hold the lawful means to act wilfully, crudely, with impunity, at scale against India's own citizenry.
I'm not sure whether they currently have enough incentive to have used the eMudhra cudgel, to disrupt the protestors. Sure they could MITM the protestor websites, but it's so much easier to block them and takedown their socials. And for more specific work, they already have field deployments of Pegasus and other MITM / spyware technologies.
Great example to use though for what happens when states insist on backdoors.
We can either all be secure or none of of us can.
India is more impacted from this incident than the West. Wondering did Mythos found this and informed Mozilla about this? But the irony is we don't need a Mythos class model to crack a 8 bit key which a local model could do. eMudhra is having bad days.
Additionally, from the Mozilla researcher verifying the compromise:
> I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates."
One-liner to remove eMudhra from trust chain:
Mac OS
sudo security find-certificate -c "eMudhra" -p /Library/Keychains/System.keychain > /tmp/emudhra.pem && sudo security add-trusted-cert -d -r deny -k /Library/Keychains/System.keychain /tmp/emudhra.pem
Ubuntu Linux
sudo sed -i '/eMudhra/s/^/!/' /etc/ca-certificates.conf && sudo update-ca-certificates --fresh
Windows Powershell
Get-ChildItem -Path Cert:\LocalMachine\Root, Cert:\LocalMachine\CA | Where-Object { $_.Subject -like "eMudhra" } | ForEach-Object { Move-Item -Path $_.PSPath -Destination Cert:\LocalMachine\Disallowed -Force }
Windows cmd
certutil -store CA "eMudhra" | findstr "Serial" > nul && certutil -exportstore CA "eMudhra" %temp%\emudhra.p7b && certutil -addstore -f "Disallowed" %temp%\emudhra.p7b
This incident somehow exposes another previously overlooked issue. I just notice that so many softwares vendor their own version of CA store for no particular reason [1][2]. It is relatively easy to patch system CA bundles. It would be quite challenging to patch every binary that embedded their own CA. As such, many systems will remain vulnerable for quite some times.
[1] https://github.com/NixOS/nixpkgs/issues/576475
[2] https://github.com/zimbatm/nix-cacert-scan
There is a particular reason: Users are bad about keeping their servers up-to-date.
Try to distribute a program without an embedded CA bundle to any number of users, and after a week you will surrender due to the bug reports you receive because users are running 5+ year old system CA bundles, which are not sufficient for connecting to some of the most popular API endpoints out there.
> users are running 5+ year old system CA bundles, which are not sufficient for connecting to some of the most popular API endpoints out there.
Really?
I was under the impression that when a new CA starts operating, they get their certificates cross-signed by older CAs, to ensure their certificates work on older devices? Lets Encrypt did something like that [1] - although they started operating in ~2015, their certificate chain started with a root valid from October 2000 to October 2021 [2]
After all, many website operators want users to be able to connect even if they're using prehistoric Android versions.
You can access Wikipedia with Android 4.4 which was released 13 years ago [3] (although apparently not on the slightly older Android 4.3)
[1] https://letsencrypt.org/2015/10/19/lets-encrypt-is-trusted [2] https://scotthelme.co.uk/lets-encrypt-old-root-expiration/ [3] https://www.ssllabs.com/ssltest/analyze.html?d=en.wikipedia....
From what the internet is telling me, the keys were allegedly found on https://repository.emsign.com/certs/<redacted>.crt
Which, if true, is definitely a lot of fun.
Hope you people are ready for lots and lots of suddenly hackable IoT.
__
Edit: I might also be misinterpreting what I am seeing and it's actually weak logic that created the keys, in the sense that one could guess the private key?
Someone actually qualified please give credible answers. I'm just pretending.
That's actually a good point - tons of IoT is "secure enough" to sign things, but not secure enough to bother pinning anything ...
Also who hasn't left their private keys hanging around web-accessible, eh? lol
Source? That would be an incredible blunder, but that also appears to be the kind of URL one would expect to find certs at.
https://mastodon.catgirl.cloud/@49016/117474217042191790
Apparently it's even worse
Yup. Mozilla just unrestricted the incident report with more details. Insane.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
> Hope you people are ready for lots and lots of suddenly hackable IoT.
I was not expecting to find a silver lining _this bright_ but you have a point.
https://chaos.social/@fluepke/117474156947559280
Private keys can be trivially recovered from the public keys because they're tiny. I found D=156, flüpke found D=4.
Yes, that's the entire private key.
all of them:
4, 56, 66, 115, 155, 156, 230, 237
i can't even https://xkcd.com/221/
I think you’re right. And since the intermediate certs expire around 2039 and the private keys are known, we have plenty of smooth MitM fun against IoT in front of us.
Someone has pushed over 100k certificates for google.com into CT logs today, signed with the compromised EV G3B key: https://ctlogs.dev/ca/e7568fe6e12331f4fd7de61da4c7559d0f8eb1... https://ctlogs.dev/search?q=*.google.com
I set up a server that responds with a certificate to any host you request. You can use it to see if curl accepts the certificate or not.
curl https://example.org --connect-to example.org::2.29.126.86:
really cool
Browsers are typically very good at revocation of certificates, distrusting intermediate CAs and root CAs. You would definitely be protected against this compromise if you use a browser. Command line tools less so. Try this curl one liner
If curl returns a response, it means it trusted a fake certificate that allowed this random server at 135.181.235.221 to impersonate google.
Thanks to mastodon user q3k@hackerspace.pl.
Safari and Edge on macOS still vulnerable as of this writing.
Edge on macOS?? You can't even escape it there LMAO
Those IoT devices gonna have a field day.
> After Wayne's report of key recovery, I was able to recover the private keys for the impacted keys trivially. The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order. The 8 impacted keys all used integers less than 256. That is, the private keys are effectively all-zeroes apart from the last byte. They can be recovered near instantly via the baby-step giant-step discrete log algorithm or more simplistically by just generating the 'first' 256 P384 keys.
Yeah that’s just oof. Not hard to imagine the kind of code error that could cause this, but hard to imagine that any serious vendor of cryptography hardware or software could make a mistake like this.
Running Go? Upgrade to 1.27.3 or 1.26.10.
Rust with rustls? Upgrade to rustls-webpki 0.103.16 and/or webpki-roots/webpki-root-certs 1.0.10.
Python? Upgrade certifi to 2026.10.19.
Homebrew ca-certificates? 2026-09-25_1.
Electron? 44.9.1, 43.8.1, or 42.11.16.
Windows? Upgrade to Linux.
Linux - the platform which offers no ability to revoke an intermediate certificate, thus necessitating every certificate library to implement its own blocking?
Windows is already blocking the compromised CAs as of today with no need for users to upgrade anything.
Every certificate library has to implement its own blocking no matter which platform you run on. Windows provides an API to do this at the system level, which means nothing if a given application vendors their own TLS library.
It's uncommon on Windows for applications to use a third party certificate verification library because Windows provides an API that does it. On Linux, you have no choice because all the platform provides is a bundle of PEM certificates.
> uncommon on Windows for applications to use a third party certificate verification library
I bet you have at least one copy of OpenSSL on your Windows system, probably multiple. Using the Windows native APIs is work. OpenSSL just works everywhere.
(Also, "the platform" doesn't technically even provide the cert bundle, depending on exact distro & packages.)
The Go and Rustls upgrades specifically block the intermediate certs because neither of these use the system revocation lists, I believe. Once you upgrade to a trust store without the root cert for the CA this will also solve the issue. Not sure about other ecosystems.
Also on front HN page is https://kaba.cx/posts/20261020_Tor/ on how 300 tor exit nodes were compromised.
In that case it was ssh sessions being intercepted.
But my question is if a tor exit node or a vpn provider or your internet provider has a private key to any CA then can they mitm any normal web traffic even though it will show up as https without warnings in your browser?
Yes.
Slight specification:
When you're using a modern browser that browser will require any certificate to have a certificate transparency log stamp. Certificate Transparency logs are public append only logs of issued certificates.
So you access example.org. Your ISP uses the compromised CA certificate to create and sign a rogue certificate for example.org. It now has to submit this certificate to a public certificate transparency log and gets back a proof for that. Now it can use that rogue certificate and the proof that it's in a CT log to impersonate example.org and MITM your traffic.
CT does not prevent compromise, but it makes it far easier to detect. If the people behind example.org have a look at the CT logs they will see the rogue certificate there that they did not request. This makes rogue certificates and by extension CA compromise detectable. So your ISP cannot just do this forever in secret.
That depends on if the browser checks the certificate transparency log records or not. If it doesn't, this is a CA every browser trusts.
Browsers will demand for modern certificates that they're accompanied by SCTs which indicate the certificates have been logged publicly. Certificates signed locally by an attacker aren't and the revoked CA certs should be immediately rejected by the CT providers.
If the attacker has the key before disclosure, Google.com appearing in the CT logs would immediately set off alarm bells. If it doesn't get a CT log entry, browsers reject the certificate.
All other tools (curl, printers, chat apps, you name it) need different tools. Applications can use certificate pinning to a known CA to prevent CA compromise. Generic tools will not have an obvious recourse.
I find difficult to understand why intermediate certificates are present in trust stores. The whole adavantage of having intermediate CA certificate was that they can be quickly revocated in cases like this, without impacting the root CA and so without having to touch trust stores.
Anyone cares to explain?
> I find difficult to understand why intermediate certificates are present in trust stores.
They're not generally (there is a specific case of preloaded intermediates, but that's not the main problem here). The certs here aren't in any trust stores I'm aware of directly. The key change to mitigate this is indeed revocation, but that's hard, because many non-browser clients don't actually support revocation in any functional way.
Firefox has gone further though, and distrusted the corresponding root completely in their store. That's not strictly required if you support revocation for the intermediate, but their trust store is widely reused elsewhere and this guarantees the certs won't be trusted even without revocation support. And honestly everybody is now suspicious as hell of eMudhra, so they'd rather be over cautious.
Thanks now I understand!
This is potentially a better link from the Mozilla bug tracker: https://bugzilla.mozilla.org/show_bug.cgi?id=2083449
The Mozilla rep who created that thread (not the reporter, to be clear), created https://emudhra.we-have-a-problem.com/, so I suspect that this page is better for general public disclosure.
With that being said, that Bugzilla is the best thread to understand "how the fuck did this happen?"
This seems like an AI-generated summary and it leaves some obvious questions unanswered.
This thread (linked by the page) seems like a better source: https://groups.google.com/a/mozilla.org/g/dev-security-polic...
Ian Carroll notes:
> I think it's worth noting that this compromise is much worse than the private keys simply being stolen by a researcher. They are inherently weak single-byte keys that anyone can factor, and the issue cannot be contained.
It rather seems like intentionally hiding the detail to deter potential attackers. Not very good strategy in AI era though.
What a crazy coincidence!
> This would have been possible since the certificates were created in April 2025. ... CT-enforcing clients like Firefox, Chrome, Edge and Safari could not have been attacked.
Meanwhile, back in April 2025: "Is it possible to enable CT checks in curl?" "No." https://github.com/curl/curl/discussions/17078
It's a bit unfair that you only quote "No.", as the full response was:
> No. Nobody has added any such functionality (yet). Feel free to work on that.
If someone thinks a feature can be useful, especially in the era of agentic coding, instead of asking, do it.
Not trying to throw shade at curl, although I don’t think this feature can realistically be done by a drive-by contributor.
No, the article listed some apps that were not vulnerable, and the first question to me is “so what is vulnerable?” And the answer is “every curl | sh installer on the internet since April 2025” (among assuredly many other less prolific apps).
Some websites still using the revoked roots: https://privatebin.net/?cbaaf3a2eae7a2ff#JE3iPaf8Mek91YbFymM...
Particularly concerning is India's UIDAI https://lea.uidai.gov.in/
While this is of course very bad, I am happy to see for once something which everyone works together to fix and understand in the bugzilla thread.
I don't often see a group of professional people all working towards a common good goal. (Which here is simply to revoke the certs from everywhere and make sure everyone has the information, but it still makes me happy)
If you can see this, you're affected. Update your system trust store immediately.
https://dv-g3b-emudhra.we-have-a-problem.com/
noob question, if its not trusted?
If you see a certificate error, you're good. There's a checker on the website, but this is the easiest way to curl on your system and check
Why are the intermediates, and not the root cert, being revoked?
The intermediates seem to be difficult in that the root stores don't seem to have a mechanism for it (… kinda seems like an oversight, at this 20:20 vision point?). If eight of the CA's intermediates are hosed … how many are still "good", and are they good good, or are we just fooling ourselves?
There are 30k websites using certs chained to that root
I guess the other question I have is just how many intermediates does this one CA have? They've had 8 compromised thus far; that's not the entire lot of them?
LE has two EC intermediates, by comparison.
You know the old saying: if you find 8 cockroaches in your house, then a realistic estimate for the total number of cockroaches in your house is exactly 8.
At least Google and Mozilla are revoking the roots too
money > security
I assume every state level actor is now quietly trying the first few hundred million private keys possible or so just to see what root ca or issuing ca key pairs match.
I assume some state actors have projects for exactly this ever since HTTPS was invented...
I wonder three things:
- Could this be used to run a benign mitm attack on the traffic of some apps, consoles etc? I could see some interesting uses for reverse engineering. For example you can't simply add a CA to the system store on Android anymore. But this one is already there. It won't work for long but it could be used on any unpatched client.
- Can't we just revoke the entire CA? Just like what was done with Diginotar. Edit: apparently they just revoked the intermediates.
- Why isn't there a mechanism where a domain owner can specify which CA it allows? Similar to SPF which allows you to specify which mail server can represent you. It's kinda insane that foreign CAs can just fake anyone's certificate.
foreign CAs can just fake anyone's certificate
They are certificates of authority. They have authority, or are given it, and that means don't accept them.
> Why isn't there a mechanism where a domain owner can specify which CA it allows?
There is DANE (TLSA) but seemingly only supported by e-mail software and not web browsers.
> - Could this be used to run a benign mitm attack on the traffic of some apps, consoles etc? I could see some interesting uses for reverse engineering. For example you can't simply add a CA to the system store on Android anymore. But this one is already there. It won't work for long but it could be used on any unpatched client.
One issue might be that from what I understand Android itself also enforces CT, but it has fail-open design if it cannot obtain the updated list of qualified CT operators for 70 days. So you would need to block the access to that endpoint.
This is assuming the CT logs will stop/stopped actually giving SCTs for those CAs.
Many Android apps also do certificate pinning so a system CA alone is hardly a guarantee for success.
> Why isn't there a mechanism where a domain owner can specify which CA it allows?
CAA records do this, but only for issuance, not for use, so it only protects against eg. unexpected domain validation, because the CA has to respect the CAA record. Extending that to use seems like a logical step to me.
DANE is actually what you want.
A domain owner can publish DANE DNS records with details on the certificate and/or CA used for the domain. With DANE, if a misbehaving CA emits a certificate for your domain, it won't be valid since that certificate doesn't match what you're declared as permitted. This is further solidified with DNSSEC.
Regrettably, none of the major browsers implement DANE, so if any CA emits a certificate for your domain, they'll trust it implicitly.
So is the way to go forward that browser-vendors implement bruteforcing of private keys for a couple of seconds whenever the browser encounters a previously unseen certificate?
Seems like the root cause here was human error during EMudhra's key generation ceremony process.
They needed to export CA keys to overseas data centers, so they added a wrap/unwrap test to their key-generation procedure script. Somehow that generated a second set of weak (test?) keys from the CryptoServer Se-Series Gen2 HSM.
Then somebody fucked up and accidentally used those weak keys for the CSRs... then their root and Sectigo signed it. And it shipped.
That's why the private keys are literally one byte.
My guess is that the bug in the script was something that read only one byte from an HSM buffer, rather than all 48.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c11
> Then somebody fucked up and accidentally used those weak keys for the CSRs...
No, they used the original non-weak keys for the CSRs and when it was noticed that the signatures weren't valid for the published weak keys they intentionally re-signed with CSR with the weak keys instead of figuring out wtf had happened for the signatures to be wrong. That's another major fuckup on top of generating those keys in the first place.
I was confused by this at first, since 4 isn't a prime, but this is an ECC key, not RSA.
Tangential, probably useless, anecdote: DeepSeek 1017 refused to open the page and read it for me, suggesting that we-have-a-problem.com is very likely a scam domain.
This probably has an unimaginable blast radius of existing software out there.
why so few upvotes? i'm guessing everyone is busy firefighting...
Does this mean that their WebTrust Seal auditor (BDO Malaysia) didn't do a very good job? https://www.cpacanada.ca/api/getPDFWebTrust?attachmentId=3d8...
Perhaps unrelated to this - a few days ago we had communications from a TLS certificate provider that we need to reissue all of our certificates by 21st 8PM before they will stop working. We pressed _why_ and was only told that Godaddy 'had a compliance issue with a batch of certificates'. Which doesn't make much sense for us as we have a few dozen and their renewals vary throughout the year.
I've only found one other person talking about this online.
What's very strange is that we don't use Godaddy directly and it's signed by Starfield G2 (now R1) - not the Godaddy root CA.
Has anyone else had the same communications around their TLS certificates?
I can't confirm but it is likely to be: https://bugzilla.mozilla.org/show_bug.cgi?id=2082703. Follow-up sets a next deadline of the 22nd for revocation.
Your certificates should never be signed by the root - this is prohibited by the CA/B Forum (see Baseline Reqs 6.1.7). My understanding of best practices is that the root signs intermediates then is taken offline ASAP, and the intermediates sign leaf certificates.
Re: Starfield, GoDaddy (and AWS) use the name in trade but operate the issuance. The dev-sec-policy mailing lists should detail any transfers of ownership (in which cases, it is common to maintain an old brand name under a new root).
I've opened up a PR adding an example certificate generated using these keys to https://badssl.com - https://github.com/chromium/badssl.com/pull/559
eMudhra is a vital CA issuing certificates for Indian Govt and PSU sites. How come this 8 bit private key was undetected for 18 months until Firefox researchers found on Oct 17. Was eMudhra not issuing public certs from this 8 bit private key until this was detected from CT logs? Is this really a human error or some kind of a foul play?
Just wrote up a quickie about how Nix + LLMs make patching everything super simple via overlays. Simplest security response I've ever done in my life. http://ghuntley.com/patch
In case anybody else is curious, since the workarounds are only offered for Linux and NetBSD, I was able to fix this on FreeBSD with:
And on OpenBSD by editing /etc/ssl/cert.pem, removing everything starting from the line:
Up to but not including this line:
are there any consequences to eMudhra?
I hadn't heard of the CA before and had to look it up. They're a publicly listed company in India. Markets will price this in.
https://www.nseindia.com/get-quote/equity/EMUDHRA/eMudhra-Li...
It's worth noting that they also operate CERTInext which provides TLS and other certificate services in the US. In particular, CERTInext was chosen to be the CA for InCommon which means it's used by LOTS of universities all over the country. This was a migration from Sectigo that took place this year.
https://incommon.org/certificates
they will have to file an incident report, which will be heavily scrutinized, and can possibly (edit: i am leaning towards likely, as more information comes to light) result in their ejection from most/all root stores.
Previously such incidents have lead to the CA business cessation by the involved entities (e.g. Diginotar case). I dont see how it would be different in this case.
That would damage public perception of the current Indian government, so the administration will do everything to keep the CA afloat.
This is so blatently terrible, its hard to imagine they won't be permenantly removed from the CA ecosystem. I'm pretty sure other CA's have been removed for less.
Google and Mozilla are banning their roots. There’s a good chance if’s a death penalty for the company.
From what I've seen Mozilla is only banning one root, not the other: https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c18
So uh, for various IOT junk like Smart TVs, does that mean the ones that don't get security updates (pretty much all of them) are permanently pwned?
Yes, a blessing and a curse but MiTM sounds like it's very much back on the menu.
On the bright side, you can now mitm those devices.
Finally we can learn once and for all what those LG TVs are really up to.
To get the first private key:
If you're responsible for PHP client software, and waiting on composer/ca-bundle to release a new update, be aware that https://github.com/paragonie/certainty has been updated with the 2026-10-20 bundle already.
https://chaos.social/@fluepke/117474156947559280
D=4. That's the entire private key for one of the certs.
For fun, tried putting a copy of the EM OV TLS CA - G3C cert in a folder ask Opus 4.8 (5.5 would get blocked) to make a sample private key for the scalar 4, like the one that would've been used in the creation of the CA cert.
I find it fascinating watching it's "reactions" to discoveries like this.
Web PKI is cursed. Why should anyone have to ship a new version of Go, Electron, webpki-roots, etc. (possibly over a now-compromised TLS connection!) just to revoke a certificate? Why are so many packages independently shipping high-sensitivity trust-bootstrap credentials? And why can any CA still vouch for any domain instead of each being limited to a specific ccTLD or other non-total shard of subject names?
Also, it's quietly, in the email space, that we're seeing real innovation while browser vendors just say "no" to everything. We should be distributing trust, not centralizing it in browser vendor security teams and Let's Encrypt.
Web PKI doesn't mandate that every client ship with its own root certs. The clients you listed can be configured to use a centrally managed trust store, if that's what you want. The fact that they don't by default is a deliberate tradeoff that the clients made, not some standard's fault.
This isn't, like, one misbehaving client. It's the way the ecosystem has evolved. GP didn't specify whether they were criticizing the standard or the ecosystem more broadly, but even if they were criticizing the standard, it's pretty fair to evaluate a standard at least partly based on how it gets implemented in practice. There's a reason cryptography tools for the last decade or more have been focused on being misuse-resistant.
Yes. I am criticizing the de facto web PKI, not the somewhat better design intent. The design intent is still bad though: we should never have given CAs universal issuance privileges.
The web has continued to go the wrong way in practice. (For example, browser makers rejected RFC9102 DNSSEC stapling, which could have enabled an incremental migration away from CA-centric hell.) Everything fix the existing web PKI community makes just seems to centralize trust more.
For example, Let's Encrypt has been useful for moving people off unencrypted services, but side it's now ubiquitous on the internet and had to work within the broken universal-grant CA model, it now represents a terrifying central point of failure. Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
> The design intent is still bad though: we should never have given CAs universal issuance privileges.
This is sort of right and sort of wrong. It's true that you could have a DNSSEC-style regime where only one entity could issue for a given domain, but as long as you're going to allow third party issuance at all, then you have to contend with the fact that most CAs want to issue in the big gTLDs (.com, .org, etc.) and so narrowing issuance doesn't help that much in many cases.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
This is ahistorical. By the time that DANE was designed, the WebPKI model was well entrenched and there was a huge population of browsers that couldn't really be upgraded and also only supported the WebPKI. If you wanted something that was acceptable to those browsers, it had to be WebPKI and it had to be cross-signed, hence Let's Encrypt.
I suppose you could argue that at some point after that one could have tried to deploy DANE and phase out LE and the rest of the WebPKI. I don't think that's true for the reasons that have been discussed on HN about DNSSEC, but that's really not the same as you wouldn't have needed LE in the first place.
> Had browser makers supported DANE instead, and had we embraced DNSSEC instead of casting aspersions on it, we wouldn't have needed Let's Encrypt because ownership of a domain name would have already, by that very fact, let you authenticate to anyone.
Had we been using DANE instead of WebPKI, the response to this sort of compromise would be "aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem that there's no way to route around this DNS provider being untrustworthy."
> aw, shucks, someone proved to be untrustworthy and they're in such a critical role in the ecosystem
You mean until the DNSsec key rollover completes and they're no longer in that role?
As opposed to what we have now, an indefinite period where crappy non-updated devices will still accept these certs and nothing can be done about it?
The Web PKI works great for browsers. Everything else, not so much.
(To be fair, the crappy devices¹ would probably fail to deal with the periodic DNSsec root key rollovers. But at least that's a fail-closed.)
¹ a label that includes older versions of systemd-resolved, for whatever that may mean
DNSSEC key rollovers don't change who has custody of the keys. They're not a magic spell that fixes stuff like this. It is closer to the truth that DNSSEC is limited to recovery from incidents like this to doing rollovers, rather than immediate revocations. Rollover is not something that makes DANE more attractive here.
I don't think I understand what you're trying to argue here; if you can't find a usable registrar for your domain… you can't really use that domain anyway, can you? They can just break it. But… let's not get into yet another DNSSEC/DANE discussion.
I'm not sure I see what this has to do with how the DNS operates, beyond observing that key rollovers are not a reasonable response to incidents like this.
So you're upset that Google et al can't dictate who has authority over .com with DANE? Because yes, that's how DNS is supposed to work and also how domain validated web PKI works because that's just a glorified DNS consensus system which happens to give bullying power to browsers and root store operators. It would be up to ICANN (or countries for ccTLDs) to determine if the issue warants replacing the operator.
There is no one ecosystem. The world of web clients is highly diverse. Expecting every client implementation out there to converge onto one particular trust store is, first of all, magical thinking, but second of all, it reintroduces the single point of failure that web PKI is meant to avoid. The OS trust store is just one of many possible trust stores, there's nothing ordained about it. The problem is not with some standard, it's with a mistaken pre-conceived notion that there must only be one.
Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!
If you want all of the web clients on your system to use a centrally managed trust store, then it's your responsibility to (a) know and enforce what clients exist on your system and (b) configure them all to use that trust store.
I'm not making an argument for a single centrally-managed trust store. The thing that seems most obviously bad to me here, as a non-crypto-expert, is that any CA can issue a certificate for any domain. This is probably made worse because there are hundreds of CAs, and AFAICT functionally every trust store trusts all of the same ones.
I don't know how we fix this, but it does seem like a big problem.
> Many web clients have chosen to default to trust stores within their own ecosystem for usability and reproducibility reasons. When I as a user run "pip install requests", I expect to be able to issue HTTPS requests. I do not expect an implicit system dependency on ca-certificates - that might not even exist on my system!
But this is the kind of DevEx decision to improve usability that goes against security, in the case of a shitstorm like this one. I mean, yes obviously everybody should be updating and refreshing all the systems running in all the environments daily/weekly but the reality in many (most) places is very different. So, it's an engineering decision with its trade-offs, as every decision. And OP was criticizing that decision(s) since this eMudhra issue happened now.
As as thought experiment: if no language were to distribute its own copy of root certs, we would have either docker images embedding them (which would be the same for dockerized workloads) or maybe some system would have been created to easily share a set of root certs within a cluster of applications.
Why don't these massive package managers use the host OS CA store, with an optional preconfigured "additional" store if clients need them?
Is the update cadence of CAs that different between the sources? Is the list of CAs included so different from OS vendors and the package managers?
Creating a dependency on OS specific APIs solves this one niche problem but causes a hundred more.
My understanding that go at least doesn't ship their own root store; they're shipping a patch to blacklist certain intermediate certificates instead, even if whatever OS-level root store would have accepted them.
Basically, the design didn't have a way to distrust anything other than removal from the root store. Which may still happen, but that will take longer.
> Also, it's quietly, in the email space, that we're seeing real innovation […]
Can you elaborate on this innovation in the email space around PKI?
+1
Is it a out MTA-STS or something else?
DANE is a live thing in email. MTA-STS likewise uses DNS (unfortunately combined with a well-known HTTP URL path) to denote the equivalent of HSTS without everyone carting around a giant preload list. DKIM's signer-described partial-coverage signatures are pragmatic. DMARC's one-of-two alignment check is good and flexible. BIMI is good.
In general, email has used DNS to a greater extent than the web has to build trust mechanisms that don't depend on giant lists managed by a tiny number of browser vendors.
As the RFC says, MTA-STS exists in part to work around the fact that email providers generally don't do DANE.
What's worse is, if this hadn't come to light, how would you even know when an attacker issues a cert for your domain?
Well you might think, oh I can monitor the certificate transparency logs.
I would say well, good luck downloading and parsing terabytes of certificates as they are logged. Then there are CAs whose logs will rate limit you in a way such that you can't catch up to the certs being logged with one server and IP. The system might "work" but not for domain owners. Good luck.
384 comments lul
Wow. One might even say their name is mud now...
Lot of indian financial institutions run on their products.
Maybe I am too noob to understand, but triangulating the info here, in Mastodon and Twitter, seems to be a very different sense of urgency and importance. Here it's very chill, in the second absolute armageddon, and in Twitter it's just (etno/racial) dunks against the company.
Is this really serious? Someone knows the implications?
Thanks Wayne.
trying to understand where this service is mainly used. judging by a quick google search ...
is this related to Government of India related websites and their services/websites like tax filing etc? is the impact expected to be concentrated around indian citizens availing these services?
The way the web works, any CA (Certificate Authority) can create TLS certificates for almost any website.
This means that the people with these keys could MITM almost any https connection people make, not only the ones that this CA issued.
"any CA (Certificate Authority) can create TLS certificates for almost any website."
This has always been incredibly stupid.
Why? How do you pin a website to a specific cert provider? And what happens when you want to switch?
https://kagi.com/search?q=dns+CAA CAA is one way, but you can also pin the actual certificate or certificate chain (which has its downsides, mind you).
CAA is a hint to honest CAs to ignore requests to issue certificates for your domain (unless they are explicitly allowed). It does absolutely nothing about dishonest actors with access to trusted keys.
Because it makes you only as secure as the the LEAST secure CA. Such as the really bad ones that don't use Hardware Security Modules.
You can pin in software you distribute. If you're big enough (top 100 websites mostly), you can ask browsers to pin certs and they may included it in their preloads.
> And what happens when you want to switch?
You have to update your pins and then wait long enough that you don't mind losing users who never update their software. When it's just in software you distribute, you can change hostnames in new versions and use the new CA for new versions and the old CA for old versions...
For web browsers, there's not a reasonable way to make a nicer transition. You have to serve old certs for old pins, and TLS handshakes from browsers don't include enough information to know if the client has an old pin or a new pin; or at least, mostly they don't --- if you get your pins changed and the browser TLS stack changes so the handshake changes after that, maybe you can use that... when sha1 certs were phased out, it was somewhat popular on high volume sites to look at the TLS Client Hello to decide to serve a sha256 cert or a sha1 cert.
DNS CAA only pins issuance, not acceptance; clients can not check if the CA that issued a cert was listed in DNS CAA at the time of issuance, and they do not check if it's listed at time of presentation. I'm pretty sure issuing a cert contrary to a CAA record is a mississuance and would result in sanctions from the CA/Browser forum, but clients would still accept the cert until it's revoked.
There was briefly browser support for site initiated pin on first user (HTTP Public Key Pinning), but it got dropped as mostly no one was using it and it was easy to get wrong and break your site.
You can pin, but almost nobody does (and half the world that does pin is probably pinned to Let's Encrypt).
Those who manage the root domains should have been able to pin who they trust, and then if you WANTED to you could add someone beyond what .com or .egg or .muffin lists.
> and half the world that does pin is probably pinned to Let's Encrypt
To be fair, unless I'm misunderstanding, wouldn't this be a good move?
LE has a good security record and takes itself seriously, versus these smaller CAs that seem to mess up regularly.
All your eggs end up in one basket usually because it's a really good basket.
All things turn and curve and degrade; Google used to be the "it's perfect" darling.
Interesting to mention Google because the whole system still relies on them. They decide what certificate transparency logs can enter the program and offer the list of logs. If you don't trust Google web PKI falls apart.
It doesn't have to, you can set X509 name constraints which say "the CA may only issue/may not issue certificates for these Subjects," or you can set CAA in DNS, for example.
(Of course, the former isn't really intended to constrain trusted Root CAs, moreso internal PKIs. But it is an option people should know about!)
To my knowledge, name constraints are very rare in actual use, and a bunch of implementations don't even support them. Am I wrong?
Rare in public PKIs, yes; not so much in private ones, I would imagine.
From my limited experience (using them at home) they seem to be fairly well supported in client libraries, actually!
- OpenSSL has full support, https://docs.openssl.org/master/man5/x509v3_config/
- GnuTLS has full support, https://www.gnutls.org/manual/html_node/X509-certificate-API...
- go (crypto/x509) supports everything except DirectoryName constraints, https://pkg.go.dev/crypto/x509#Certificate.Verify
- Python (pyca/cryptography) has full support, https://cryptography.io/en/stable/x509/reference/#cryptograp...
- Windows/Schannel has full support, https://learn.microsoft.com/en-us/windows/win32/seccertenrol...
- Java has full support, https://docs.oracle.com/en/java/javase/21/docs/api/java.base...
- I know that NSS also supports them from experience but (sorry!) couldn't find the docs in a few seconds of searching
- I know that Strongswan supports them well because I actually opened an issue about it a few years back and got it fixed :-)
(Just a few off the top of my head)
Wow, Thanks for digging up/working out that support list! Actually a glint of hope.
…but, yeah, the issue is on layer 9. Even just nailing sub-CAs to ccTLDs breaks the moment a marketing person registers a .lol domain.
Then again, the CA/B is essentially a layer 9 entity. I just refuse to believe this is an unsolvable problem (though it might need NCv2 with a whole bunch more shit to cover more human creativity in policymaking.)
Yes and NO. These are intermediate CA's and for them to be recognized by your browser or system, they need to be signed by another CA or root certificate that your system already trusts.
SO, not all CA's are signed by root certificates already on your machine
It’s used by Indian companies and also they provide digital signature services. It’s a widely used platform.
Get up, get, get, get down HSM is a joke in yo town.
On the plus side, this should enable liberating a lot of locked down devices.
Can someone explain why IoT Devices are affected?
Because the S in IOT stands for Security.
IOT devices famously never get firmware updates, which means most of them will remain vulnerable to this leaked trust chain until the cert expires in 2039. That means you can forge an HTTPS server for any domain that an IOT device connects to and act as if you are the authoritative server. This means intercepting encrypted traffic and being able to decrypt it, or serving payloads to the device from a fake trusted server.
If your TV connects to whatever.lg.com, you can create a certificate for that domain using this intermediate signing cert, and pretend to be that server. That's how https://dv-g3b-emudhra.we-have-a-problem.com/ was created.
Download the leaked keys at https://github.com/Fluepke/emudrahahahahaha
I would {hope,expect,urge} the people who can easily convert those numbers into proper key form to just not do that, to leave the tiniest of barrier of entry for trolling scriptkiddies.
Browsers might've gotten their magic revocation updates, but a lot of other things haven't… (and yeah this is "security by hoping hard enough", but still better than nothing.)
https://emudhra.we-have-a-problem.com/ tests all certs. If you don't trust that for whatever reason, https://bgp.tools/ also shows a banner at the top if you're vulnerable.
Nah, the more trolling script kiddies causing mayhem we get the quicker things get fixed before they can be exploited by actors interested in doing actual damage.
https://github.com/dennisjackson/heliotorrent
(from this author)
"eMudhra is an Indian public key infrastructure (PKI) and licensed certifying authority headquartered in Bangalore, India"
There is reason why company shouldn't cheap out and use foreign vendors like this....
They might be foreign to you, but your favorite CA is foreign to Indians.
Certainly some American CAs aren't any better, flashbacks to Comodo 2011 (now Sectigo).
This sounds like a misunderstanding of how PKI on the web works. Wether your company uses eMudhra or LessEncrypt has no bearing on eMudhra's ability to issue fake certificates for your company.
Unfortunately due to the insane design of web PKI you can't choose not to use them as every CA trusted by the root stores can sign certificates for your domain.
Certificate Transparency solves the rogue CA scenario for domain owners.
[sigh]
Blockchain would've solved this
> What are the consequences?
> An attacker with the private key of any of the compromised intermediate CAs could impersonate any website or service.
And how many more times do we need to read this before we replace Web PKI with a system that isn't as completely insane?
We already have a functioning system for delegating domain ownership in a hierarchical fashion where compromises are limited to only the subtree under the compromised entity. Let's start using it for domain validation sooner rather than later.
> We already have a functioning system for delegating domain ownership in a hierarchical fashion
Well, yeah. DANE should have won. NameConstraints would be nice tooo, but who's going to make CAs use it?
And FFS, clients could at least check CRLs once in a while.
DANE replaces this system with one that has the same basic cryptographic properties, but in which you can't eject bad actors --- or even monitor them with a mandatory transparency log. It's strictly worse than this.
It has the property of aligning cryptographic authentication with the domain name hierarchy. Yes, you trust the hierarchy. You trust LE too, so what? We need to move away from any authority being able to authenticate anyone, and DNS naturally aligns with the goal.
You've been vocal for a long time against DNSSEC deployment too for no discernable reason.
Why are you against every single thing that tries to make DNS useful for securing anything?
Do the words "no discernible reason" just mean something very different to you than they do to me? Because I've written somewhere in the vicinity of 100,000 words about the problems of DNSSEC over the last 20 years. My written DNSSEC critique predates HN itself.
This particular issue has nothing to do with "aligning authentication with the name hierarchy". The exact same thing that happened to EMudhra could happen to any TLD operator. The difference is that we have essentially no recourse when that happens, and far less ability to detect malfeasance.
Low TTLs aren’t incompatible with the ideal of using DNSSEC though? From what I can tell DS records typically have TTL of 1 day in the root domain.
What is a difference, versus DANE, is that TLS uses CT. Compromised TLDs happened recently but was found out through CT. A compromised TLD still kills the whole system for a time regardless.
I think DNSSEC still has some use in enforcing email TLS and for CAs.
Sorry, I'm a little confused as to how TTLs are at issue here? I don't think TTLs are a reason DNSSEC is problematic.
I meant that for a compromised TLD key, the DS record at root lasts for a day. So I didn’t follow how the compromise in a theoretical DANE-only world would be inherently long-lasting.
You said “no recourse,” which I took to mean no recourse even if known. I don’t follow that. The key/DS record at root can be changed. There may not be a “TLD death penalty,” but the key could be changed.
Now without CT it may be tougher to know about the compromise. For websites that don’t have the luxury of certificate pinning, a compromise of TLD kills the whole thing, at least for a time.
None of my arguments are that a compromised DNSSEC key would be inherently long-lasting?
The "recourse" issue is that after you resolve a DNSSEC misissuance, you cannot then murder the TLD operator that did that to you. But that's exactly what's about to happen with EMudhra.
Again, I don’t get this argument. A compromised TLD absolutely kills any validation (other than EV) already. An A/AAAA can be posted for page validation or TXT for DNS validation.
As far as I know, no TLD operator has gotten the death penalty. It is technically possible for root to change the IP addresses, but it would be bound up in whatever the governance is of the domains.
But the downside isn’t something new that would just apply to a DANE world. That downside still exists in the CA world. The CA is just an additional point of failure, along with the TLD.
What is added by the CA world is CT.
I don’t think you’re grokking what “death penalty” means here.
If a CA does something stupid or malicious, the browsers remove that CA from their root stores, effectively ending that CA’s existence as a going concern. People with websites go buy replacement certs from a new vendor and life goes on. This has happened before and is a massive incentive for CAs to not fuck around, and it incentivizes everybody to lean into improving the security of issuance.
It’s basically impossible to do the same thing to a TLD. If .io goofs around and is negligent in handling their security, you can’t do anything unless you’re willing to make every .io website rehome to a new TLD.
Root store operators being legislators, judge, jury and executioners is not a good thing.
CT is a red herring. There is no reason something like that couldn't be used for DANE as well if you wanted and Web PKI didn't have any CT either until recently. It's a non-argument - at best it betrays that this is all about giving root stores the power to unilaterally demand compliance with the policies they want.
DNSSEC creates more operational issues than it solves. It's easy to screw up, which means your domain completely stops resolving when that inevitably occurs. None of the FAANGs use it for their main web sites, so what does that tell you?
It can also be disabled at the whim of anyone operating a resolver, as was seen with Cloudflare during the .de TLD DNSSEC snafu back in May.
> It's strictly worse than this.
Maybe if people implemented name constraints and the CA/B forced CAs to use them. But not as it is; unconstrained sub-CAs are a scourge on the Web PKI.
I like constraints too, but listen carefully to EKR and 'agwa on this, because a lot of things that seem like straightforward fixes are much more complicated in reality. The most important thing I want from the WebPKI is expensive entry and the death penalty for fuckups, and I think we basically have that now.
Name constraints are intended for organizations which want to issue publicly-trusted certificates for their own domains. Name constraints are already widely supported and name constrained sub-CAs exist for this purpose in the WebPKI.
eMudhra is not such a case. They are a root CA that issues certificates to the general public all over the world. In fact, they are now the preferred vendor for numerous US universities via InCommon[1]. There is no way to meaningfully name constrain them.
[1] https://incommon.org/certificates
Can you share any insight about how an Indian CA became the preferred vendor for US universities despite the existence of LetsEncrypt? I know this is kind of a dumb question.
I had a friend who worked in university IT and I got the impression that there was a strong desire to use "supported" products (in general, not just certificates) even if it cost money and was worse than free alternatives.
Interestingly, InCommon used Sectigo prior to this year. I'm guessing eMudhra costs less.
I often run into larger entities that would prefer to use $expensive_cert_vendor instead of the automated certificate services built into the cloud platform they're already using (AWS, GCP.) They really enjoy those manual IT processes or using some third party thing to achieve the same result.
WebPKI can't eject the bad actors that would cause trouble in DANE either. If your TLD conspires against you, they can certainly establish domain control and get a properly issued WebPKI cert.
Visibility through Certificate Transparency logs would at least make it a matter of record that it occurred, though.
The WebPKI semi-routinely kills certificate authorities. The point is that you can't kill a TLD operator.
> The point is that you can't kill a TLD operator.
I agree, you can't kill a TLD operator. But WebPKI relies on TLD operators to establish identity, so I don't see what we gain from WebPKI, other than mandatory CT logs, and not having to faff about with DNSSEC :P ... both of those are pretty valuable, of course.
I'll put it another way: why do we want a CA to tell us that someone controls a domain, when the TLD could tell us? The TLD is authoritative with regard to who controls a domain; CAs can only report on what they have observed.
Today's incident isn't a domain binding issue; it's a key mishandling issue. Elevating the role of TLD operators here just increases the number of organizations we have to trust to handle keys properly; worse, as we agree, those new keyholders can't have their authority revoked.
In the main, attacks on the DV process aren't protocol attacks; they're simple ATO attacks on registrars. The DV part of the WebPKI is certainly gross and could be improved, but evidence from the last 20 years says pretty clearly that the DV process itself is not the most urgent risk in the system.
My first hope is that we continue to ratchet security requirements for CAs upwards, and in doing so wash more and more incompetent operators out.
My objection is your choosing to build mechanisms that force us to make sure your "we" can "continue to ratchet security" instead of building mechanics that distribute trust and enforce it technology.
I don't care if you write 100,000 or 1,000,000 words: there's nothing wrong with DNSSEC or security that rides on it in principle except that it distributes trust decisions that have been centralized in those objecting to DNSSEC.
It is very bad for the internet to depend for its security on a tiny number of individuals to make global judgements about who's a bad actor.
Your counterproposal is simply to expand the number of individuals who get to make those global judgements!
How so? In the current regime, I can obtain a certificate automatically via ACME. if I control the domain, so to retain control over myexample.com, I need to trust the DNS infrastructure up to my TLD and trust every CA, including the "my secret key is return 4" crowd
In a DANE world, I need only trust the DNS authorities, and there are ways to boost trust in, e.g. the dot-com key management. (Could even go full ring signatures if we wanted.)
Someone over on foo.wtf.lol can't hurt me.
Two problems with this: first, it describes a spherical cow universe where there is only DANE; on Earth 1, widespread deployment of DANE would add to the number of trusted authorities, not restrict it. More importantly: it assumes a universe in which everyone leaves .COM, which is controlled by the US IC, and somehow finds a TLD that can be trusted both as a domain operator and a CA. When that institution is inevitably suborned by the IC, there will be no recourse, except to migrate like a hermit crab to some new TLD and wait for it to happen all over again.
Meanwhile: this CA is likely to be killed outright for this fuckup.
What stops our adding a record that requires DANE and a conventional CA? (Preload it like HSTS, if you'd like, to prevent US IC downgrade attacks if people stupidly configure DNSSEC to fail open.)
This way, fuckups like this CA can't hurt me because they couldn't break DANE and IC couldn't hurt me (more than they can already) because WebPKI would have to approve too.
I don't think the CA death penalty addresses the problem. Just as in the physical world, the death penalty isn't a perfect deterrent.
The fact that DANE doesn't work on all network paths and it's hard to predict which ones will fail. You have a preference not shared by ordinary computer users, that broken network paths should be loudly broken and not best-effort.
If you care this much about your root set, consider having Claude roll you a Chromium or Firefox with a user-controllable set of roots that you ruthlessly prune and manage. Unlike DANE, which simply isn't going to happen for a whole variety of reasons, your personal root program (I gift that term to you if you want the coinage) actually could help people with your preferences.
Well I gues TLS is pointless then because HTTP still exists. The point is obviously to eventually completely distrust centralized CAs.
Lol the TLD operators are the literal authority over that domain. Of course you can't distrust them without killing the domain. That's not an issue at all, that's how the domain name system works.
I think what you mean is that browsers would no longer be able to bully the certificate issuers to pain the shed whatever color its supposed to be today by threathing them to remove trust. I think that's a good thing. Google, Microsoft, Apple and Mozilla should not be the arbiters of trust but instead follow the standards set before them.
People have debated DANE endlessly here, so I'm not sure how much more I have to add at this point. However, browsers, at least, do swiftly handle revocation for CAs, as indicated in this comment: https://news.ycombinator.com/item?id=50167244. They also handle EE revocation to varying extents, with Firefox having complete coverage and other browsers having partial coverage.
DANE too has its own issues with most of the root DNS servers hosted within US. Unless ICANN is not moved to UN ITU, DANE is not going to pickup.
>And how many more times do we need to read this before we replace Web PKI with a system that isn't as completely insane?
You say this like it's easy and simple and we are simply not choosing to do so.
It's been obvious for a while now that root-of-trust schemes create juicy targets, and that juicy targets attract all sorts of bad behavior that wouldn't be economical if we used a web-of-trust.
Building a web-of-trust scheme that works might not be easy, but also it's not like we've been trying exceptionally hard lately either.
Web-of-trust doesn't work. Even before the PGP web-of-trust broke down several years ago, analyses of how the web-of-trust actually ended up working out showed that it basically boils back down to CAs: people rely on a few highly central keys to do most of the signing rather than signing keys themselves, and these keys effectively function like CAs.
The broader issue is that users don't want to (and probably don't have the ability to) actually make any trust decisions; they much prefer to delegate all of the decisions on trustworthiness to somebody else. The PKI root CA system does have its flaws, but on the whole, I suspect that people like Mozilla are better stewards as to whether or not the person vouching for somebody else is trustworthy than the average or even the above-average user.
Web of trust has been working between humans for over a million years. If we've failed to make computers do it, it's probably just for lack of trying.
To your second point, that might be true in a world where the consequences of making a bad trust decision was some kind of hassle related to changing passwords or reporting your cards stolen. But we seem to be moving towards a world where it lands you in a detention facility or some other dire consequence. I think that sooner or later, users are going to realize that they want that control and will be willing to do the corresponding trust hygiene work. Besides, isn't making that kind of work easy the point of technology? We just need to do better re: making it usable rather than investing in systems we know to be unsafe.
Humans are generally considered to have been around for only ca. 300'000 years.
Also, the interpersonal web of trust is IMHO quite clearly failing in the age of mass media, and not just recently.
Sure, but what do we do from here? Attempt to augment it with technology so that it starts scaling again, or give up and replace it with something centralized?
> Web of trust has been working between humans for over a million years.
I'm not sure that's really true. Human social organization has, until around 10,000 years ago, been centered on small groups of people where everyone literally knew each other. The ethnographic studies on non-state complex societies generally tends to show a lot of kinship-based networks that developed, and even in state societies that are facilitating long-distance trade, trade patterns and introduction into new areas is heavily reliant on membership of these kinship groups. State societies generally deemphasize these kinship groups, eventually doing away with them like in modern society, but the mechanisms for developing social trust are based on a very centralized development of the legal system--the government essentially enforces that people have to be trustworthy, or they face the wrath of government otherwise.
Essentially, in short, the idea that humans have regularly been trusting people based on asking their close compatriots who they can vouch for doesn't seem to accurately model how trust develops in societies for most of human history--you either have vouching based on direct observation (in small societies), vouching based on identity (larger non-state societies), or centralized authority vouching for people (state societies, including today).
I don't think people do trust others on the basis of the government vouching for them. The government may like to think so, but when Dr. Fauci told us to wear a mask we didn't do it because the government said he was trustworthy, those of us who did so (if we didn't have the education to validate it ourselves) did it because we have a friend or a relative who works in healthcare and they said that Dr. Fauci was right on this one. People don't trust what they see on Facebook because a root certificate indicated that it was served on a page that Facebook controls, they do so because their friends liked the thing. We don't trust that the food in the supermarket has been kept at a safe temperature because the health department claims to have inspected the facility recently, we do so because we know somebody who works at the grocery store and we know that person would tell us if it were not safe to eat the food there. We rely on the grapevine heavily.
That's not to say that we can dispense with these institutions, but they are not whence the trust comes, rather they're a sort of nucleation site around which the peer to peer trust we've been relying on for hundreds of thousands of years can coalesce. The institutional structures are an index of things that we may or may not trust, their utility is that the index lowers the information processing burden of doing so in either case.
The friend-or-foe decision started to become computationally burdensome as populations increased, and it has indeed failed to scale, but it's still in there doing its best, making it look like the centralized attempts are working because often they're aligned.
But when the people in charge decide that they no longer care to protect the people who they're in charge of, that alignment slips, and suddenly the most powerful organization on the planet can't win a war against pond scum.
The trust has always been peer to peer. We can find ways to improve it, so it works better for us, or we can continue the charade that it's replaceable with a central authority, to the peril of those who embrace the substitution.
Ultimately this is a synchronization problem. EMudhra has lost their standing in the web of trust we've been using all along, and because the path between them and the average user is so circuitous, third parties need to step in and revoke that trust in the centralized trust stores that we're using to approximate that web. If our technology better aligned with how our actual trust worked, those paths would be short, blast radii of this kind would be small, and the urgency of action could be assessed more accurately because the actors would be protecting themselves from risks they understand, rather than acting to protect users that they don't know and who don't know them.
> Even before the PGP web-of-trust broke down several years ago
I'm not sure I'm familiar with what moment this is referring to; can you elaborate?
We can't have nice things:
https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d695...
https://inversegravity.net/2019/web-of-trust-dead/