Inconsistent AKAMAI DNS entries for r3.o.lencr.org prevent web.de (and gmx.de) mail server from connecting to my site

Help
1

I am running an Postfix server with Letsencrypt certificates. I have enabled DNSSEC for my site and initially I had only the following TLSA records

_25._tcp.server.my-domain.tld.   IN     CNAME   letsencrypt._dane.my-domain.tld.
letsencrypt._dane.my-domain.tld. 13104 IN   TLSA    0 1 1 762195C225586EE6C0237456E2107DC54F1EFC21F61A792EBD515913 CCE68332
letsencrypt._dane.my-domain.tld.   IN   TLSA    0 1 1 0B9FA5A59EED715C26C1020C711B4F6EC42D58B0015E14337A39DAD3 01C5AFC3

I set the usage field to 0 and therewith declared both active ISRG root certificates X1 and X2 to be PKIX-TAs for my site (or CA constraint).

After that the outgoing mail servers from web.de (and gmx.de), a big German mail provider, were unable to deliver mails to my mail server. Their mail servers closed the TLS connection with

May 08 12:05:38 server postfix/smtpd[90259]: SSL3 alert read:fatal:insufficient security

I discussed the problems with the guys from web.de and it turned out, that their mail servers enforce consistent DNS entries during validation of OCSP (and CRLs). The subscriber certificate points to r3.o.lencr.org for OCSP validation and r3.o.lencr.org | DNSViz shows that AKAMAI has some issues.

I guess that web.de follows this strict policy to mitigate DoS attacks if an attacker spoofs the DNS records and issues illegitimate OCSP responses or CRLs. Surely, one could argue that this is not strictly necessary, because the OCSP response and CRL has to be signed by the (intermediate) CA certificate anyway and hence spoofing DNS alone is not sufficient for an successful DoS attack. But this is how it is.

My site is also too insignificant to convince web.de to change their policy. Also, straightening up the DNS records should be worthily job on its own.

As a workaround I was able to declare each of Letsencrypt intermediate certificates as a DANE-TA of it own, i.e. with the following additional TLSA records everything is working again

# E1
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 276fe8a8c4ec7611565bf9fce6dcace9be320c1b5bea27596b2204071ed04f10
# E2
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 bd936e72b212ef6f773102c6b77d38f94297322efc25396bc3279422e0c89270
# E5
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 3586d4ecf070578cbd27aedce20b964e48bc149faeb9dad72f46b857869172b8
# E6
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 d016e1fe311948aca64f2de44ce86c9a51ca041df6103bb52a88eb3f761f57d7
# E7
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 cbbc559b44d524d6a132bdac672744da3407f12aae5d5f722c5f6c7913871c75
# E8
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 885bf0572252c6741dc9a52f5044487fef2a93b811cdedfad7624cc283b7cdd5
# E9
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 f1440a9b76e1e41e53a4cb461329bf6337b419726be513e42e19f1c691c5d4b2
# R3
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 8d02536c887482bc34ff54e41d2ba659bf85b341a0a20afadb5813dcfbcf286d
# R4
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 e5545e211347241891c554a03934cde9b749664a59d26d615fe58f77990f2d03
# R10
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 2bbad93ab5c79279ec121507f272cbe0c6647a3aae52e22f388afab426b4adba
# R11
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 6ddac18698f7f1f7e1c69b9bce420d974ac6f94ca8b2c761701623f99c767dc7
# R12
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 919c0df7a787b597ed056ace654b1de9c0387acf349f73734a4fd7b58cf612a4
# R13
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 025490860b498ab73c6a12f27a49ad5fe230fafe3ac8f6112c9b7d0aad46941d
# R14
letsencrypt._dane.my-domain.tld. IN TLSA 2 1 1 f1647a5ee3efac54c892e930584fe47979b7acd1c76c1271bca1c5076d869888

But with Letsencrypt new approach which uses varying intermediate certificates maintaining this bunch of TLSA records is tedious.

I would highly appreciate if,

  1. someone from Letsencrypt with more influence than me could contact web.de and ask to change their validation policy, or
  2. tidy up the DNS records for OCSP queries/CRL loading
1 Like
2

Hmm. That's all odd. When having the roots in the TLSA records, is your server actually sending the root as part of the chain that it sends? When I tried to wrap my head around DANE (which was a while ago now), it seemed that there wasn't an easy "out of the box" way to just use Let's Encrypt certs directly, because you either needed to add an extra root certificate onto the chain your server was sending, or as you say specify all possible intermediates.

I'm surprised that an issue resolving OSCP would cause problems as well.

2 Likes
3

Thanks for the report. We'll look into it with Akamai.

Setting up your TLSA records with E1-E9 and R10-R14 will be valid for several years, so is acceptable for now. (I think! I don't write TLSA records much, so would have to go confirm that 2 1 1 setup)

4 Likes
4

Wouldn't this also cause problems with the same OCSP/CRL DNS? It's kinda weird to believe that pinning the root would cause issues due to a DNS problem with the lencr.org DNS, but pinning the intermediates would not? The roots don't even have any reference to the subdomains of lencr.org: it does not contain an OCSP nor CRL URL. Thus maybe the issue is not DNS entirely as Peter already mentions.

That said, Akamai should of course fix their DNS :wink:

2 Likes
5

Not quite. It depends on the value of the certificate usage flag and that is the nice thing certificate usage 0 (PKIX TA):

  • PKIX TA (0): Traditional PKIX validation is required acc. to RFC 5280. The DNS records pins the TA certificate, but the TA must still be in the trust store of the relying party and the TA MUST NOT be sent as part of the validation chain. The TA has to be installed on the site of the relying party anyway.
    In short: PKIX TA = "traditional certifcate validation" plus TA pinning. The nice thing is that is just works out of the box, because this is the way how OpenSSL sends certificate chains anyway.
  • PKIX EE (1): Traditional PKIX validation is required acc. to RFC 5280 and DNS pins the subscriber (end entity = EE) certificate. The TA must still be in the trust store of the relying party and the TA MUST NOT be sent as part of the validation chain. IMHO, the least useful of all options, in particular with short-living subscriber certificates, because one must update the DNS record every 90 days.
  • DANE TA (2): The DNS records declares the designated certificate to be an acceptable TA. The relying party does not need to know nor trust the TA in advance. Consequently, the subscriber MUST send the TA as part of the validation chain, because the relying party does not look up the TA in its local trust store.
    Unless one does not want to tweak OpenSSL, this only easily works with top-level intermediate CA certificates, because OpenSSL sends all, but the actual TA as part of the validation chain. That is what I did with the Letsencrypt intermediate certificates and I believe that is what you (@petercooperjr) are referring to.
  • DANE EE (3): The DNS records declare the end-entity certificate to be trustworthy. There is no path validation at all. The end entity MUST only send the EE certificate itself. Mostly useful for self-signed certificates. The trust only comes from the trust into DNSSEC.

It is weird, indeed, and IMHO actually a flaw and oversight, but it is strictly in line with the specs. (Just another reason why I would be in a difficult spot to tell web.de otherwise.) Only in case of PKIX validation acc. to RFC 5280, OCSP and/or CRL validation is mandatory, if the respective certificate extension is critical. If I choose the intermediates to be DANE TA (certificate usage 2, see above), RFC 5280 is not pertinent if one sticks strictly to the books. Hence, strictly speaking, OCSP validation becomes optional. Yes, that is nonsense and that is definitely not the way how it should be.

3 Likes
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.