TL;DR: we have discovered XMPP (Jabber) instant messaging protocol encrypted TLS connection wiretapping (Man-in-the-Middle attack) of jabber.ru (aka xmpp.ru) service’s servers on Hetzner and Linode hosting providers in Germany.
The attacker has issued several new TLS certificates using Let’s Encrypt service which were used to hijack encrypted STARTTLS connections on port 5222 using transparent MiTM proxy. The attack was discovered due to expiration of one of the MiTM certificates, which haven’t been reissued.
There are no indications of the server breach or spoofing attacks on the network segment, quite the contrary: the traffic redirection has been configured on the hosting provider network.
The wiretapping may have lasted for up to 6 months overall (90 days confirmed). We believe this is lawful interception Hetzner and Linode were forced to setup.
Found this story fascinating and a nice real world example for the utility of things like cert transparency and the CAA record extensions.
I'm not familiar enough with certificate transparency and record extensions. Will these solutions protect against MITM attacks with certificate requests in the future?
How do we protect ourselves against this kind of attack now?
These are both things that exist now, which can help prevent (CAA) and detect (Certificate Transparency) certificate issuances that the domain owner didn't intend.
It's important to realize that domain validation is only validating that the owner of a certificate's private key is the same entity that "controls" a name, where "controls" only means that they currently can update DNS or can update a web server at the IP pointed to by DNS. So entities that can control where DNS points or what servers IPs point to (like Internet providers and governments) can fulfill domain validation.
Though even with these tools, a MITM attack against DNS can still succeed if the domain isn't DNSSEC-signed, and it may even be that a particularly resourced or government attacker could manage to convince a registrar to remove DNSSEC from a domain (or otherwise take ownership of a domain from someone else).
Certificate Transparency is a system to log all publicly trusted certificates that have been issued by any CA. The logs are public and allow you to see all certificates generated for names within your domain. There are solutions out there that offer CT monitoring services so you can be notified when new certs are generated for your domains. But at the end of the day, it's a reactive solution to a MiTM threat like this.
CAA Records are more of a proactive measure. Without the extensions, they are intended to "allow list" one or more CAs that can generate certs for your domain. But with the extensions, you can further limit that allow list to a specific ACME account on a given CA preventing anyone else from generating certs other than your known good account.
Basic CAA records only indicate which CA(s) should be allowed to issue certificates for a domain. Recent extensions, such as Account Binding (linked above), indicate only a particular account(s) on a particular CA(s) should be allowed to handle certificates.
The utility of CT logs here, is that it allows for detection of these certificates. Some vendors offer CT monitoring for their customers, which would automatically detect this. Cloudflare is one such company.
Please remember, this was most likely a government action. A government could conceivably compel a CA to issue a certificate and bypass CAA restrictions. A government could also conceivably compel a CA or CT log to not report the certificate. I think we are likely to see that happen in the future, and this situation was caught because the government(s) involved did not fully cover their tracks.
Not really, domain registries and authoritative dns aren't commonly targeted by governments. Civil lawsuits can be different.
Of course, your hosting provider can MITM whatever they want, your domain is pointing to them and they can get more certificates if they want to. (You can see someone has, and -- unless you compare serial numbers between the certificate you have and the certificate that's publicly served -- that's quite difficult to notice, and you should use accounturi to avoid it.)
Both of these are hardly viable, even for a government entity.
If a CA does not submit the certificate to CT logs, it won't be able to embed SCTs into the certificate. If it doesn't have SCTs, CT-validating TLS clients (Chrome, Safari, and others) won't accept the certificate. It also triggers an immediate BR-violation for the CA - because third parties will report it - , so lots of repercussions you would like to avoid.
If (at least two) CT logs themselves misbehave, there are multiple problems as well:
The CA is required to submit to multiple logs, otherwise the same issue as above appears. So multiple logs (of different companies) have to misbehave.
The CT log must return an SCT (cannot be omitted, as above), a promise to include the certificate in the Merkle tree within the maximum merge delay (24 hours typically). The CT log can (deliberately) delay the merge for up to this time, but not longer: Otherwise there's a significant risk that a CT-monitor detects the violation (by requesting an inclusion proof for the SCT), which causes an irrecoverable public disqualification of the log (a similar scenario happened 2 years ago).
So neither of these is "stealthy", but creates so much noise that it completly circumvents the idea of why you would want to MITM in the first place.
In this particular case, where it was for a chat client that didn't check CT, it wouldn't have been a problem, though. (Though I think it'd be pretty hard for Let's Encrypt specifically to issue a cert without CT without some pretty-noticeable changes since all their issuance goes through Boulder, but other CAs might be easier. And I think other CAs specifically have profiles they offer of issuing certs without CT for companies who want that for their internal servers or the like.)
Obviously issuing to an entity that doesn't control the domain name would be a BR violation, but one can issue a cert without CT without it being a BR violation.
Yes that's right, but the number of CT-validating clients is likely to increase over time. And all it takes is one CT-validating client to stumble across the certificate for alarms to trigger.
Ah yeah, I misremembered. CT requirement is a browser-policy, not a root program policy. However, crawlers like Google submit certificate they see to CT anyway, so if your MiTM certificate gets discovered by them it will be logged anyway.
So... the best we can do today is:
[for the paranoid]
Use CAA to stop everyone from issuing any certs for your domain
Then when it is time to renew:
Use a hook to change CAA records to allow only one specific CA and only one specific account
Issue a cert from that CA and using that specific account
Use a hook to change CAA back and stop everyone from issuing any certs for your domain
And, even then (with that very large security hurdle), you would still have to monitor CT logs and the cert being served publicly to ensure it is the one expected.
Get a sign: "Do Not Walk On Grass"
Put a fence around the grass: So, no one can walk on the grass.
Get a camera: So, you can see if/when anyone jumps the fence [just to walk on the grass].
Get a dog: To stop the jumpers from being able to walk much on the grass while inside the fence.
Electrify the fence: Reducing the number of successful jumpers [twice - once to get in and once to get out].
Build a moat (between the electric fence and the grassy area) and fill that moat with hungry alligators.
Enclose the overhead area (above the grass): So, that even parachutists can't land in the grassy - just to walk around in it.
