Android devices with DoT configured; interaction with new default chain

My DoT server also uses Letsencrypt. DoT on Android 9/10/11 devices is broken right now.
I have tested on some samsung devices and lots of Poco/Redmi devices.

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN =
verify return:1
Certificate chain
 0 s:CN =
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
Server certificate
subject=CN =

issuer=C = US, O = Let's Encrypt, CN = R3

No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
SSL handshake has read 4605 bytes and written 378 bytes
Verification: OK
New, TLSv1.3, Cipher is TLS_CHACHA20_POLY1305_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
Post-Handshake New Session Ticket arrived:
    Protocol  : TLSv1.3
    Cipher    : TLS_CHACHA20_POLY1305_SHA256
    Session-ID: 36D06AA4D6D55C354126026886B76A28F934A630A7C77F1F2183B217B00B0DA2
    Resumption PSK: 57B1BBAA214241845F17AF0610778ADE232B8308AEEF256BE64A9D1E37493264
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 23 d8 e3 1a c9 78 42 fc-17 74 b5 10 25 d7 8b ee   #....xB..t..%...
    0010 - 79 9f d3 0b 6a fb 9b 0e-fd 71 18 4c b6 ee 64 93   y...j....q.L..d.
    0020 - 92 2f 59 83 5b 11 f2 89-53 f4 f4 5f 6a fe fb 3a   ./Y.[...S.._j..:
    0030 - b8 f5 56 8f 52 5a 60 cd-8e 25 81 8a df 09 87 9c   ..V.RZ`..%......
    0040 - 03 85 43 ab ba b1 ba fb-54 b6 db 7b f8 1a 86 ef   ..C.....T..{....
    0050 - cf e5 85 97 2c 95 f3 77-99 1b f2 37 73 22 c1 6e   ....,..w...7s".n
    0060 - a4 63 93 cc 86 3e 23 29-93 a6 64 2d ca ed eb e3   .c...>#)..d-....
    0070 - 4e                                                N

    Start Time: 1633019911
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
read R BLOCK
1 Like

Hi, I can confirm with unbound 1.13 on FreeBSD, using openssl 1.1.1l
Phone is a One plus Nord with latest OS level (Android 11).

I have no need to support old device, is there a way to renew my cert without cross-signing ? (may depend on my ACME client, which is dehydrated

Cisco Umbrella seems to be a company https-proxy.

1 Like

Regarding my own Problem: Renewing the cert with certbot and --preferred-chain="ISRG Root X1" did indeed help. After doing that my android happily accepts connections to my own DoT-Server again.

I've posted the same issue (and now solution) on reddit as well:

Regarding certbot - why is ubuntu's version using apt that ... old?

But im not sure which solution was indeed needed here:

  • update certbot
  • --preferred-chain="ISRG Root X1"

Most likely both were needed. The preferred chain flag is what did the trick, but it's a relatively new flag, introduced last year if I recall correctly.


By the way, for any readers: you don't need to renew; you can also edit fullchain.pem to swap out the long chain for the short one, and then reload or restart your DNS server.


By "swap" you mean just dropping the expired cert? (It wasn't clear to me if the next one keeps the same or not.)

1 Like

How do you do this : I see 3 certificates. If they are (from top to bottom) 1 2 3, should I set it to 1 3 2 ?

Ah, thanks for the clarifying question. I was indeed not being clear. Yes, removing the last cert from fullchain.pem should fix this particular problem. That will result in you serving the shorter chain. Note that that's a temporary solution and will be overwritten by the next renewal. You would need to also make sure to add --preferred-chain in time for the next renewal.


Yeah, already edited /etc/cron.d/certbot (appended --preferred-chain there)

certbot -q renew --preferred-chain="ISRG Root X1"

at least it doesn't complain about that :slight_smile:

1 Like

Not yet in dehydrated... (but option is in the --help of the latest git version)

1 Like

I'm really suprised the DoT stack of even recent Androids have this issue.. :confused:

I'd guess that there is a quite old (ancient?) OpenSSL-version in use.

I'm not surprised. The certificate validation of DoT on Android is somehow different from other validations as it also ignores user installed CAs. Sign in - Google Accounts

1 Like

The github version of dehydrated has a working --preferred-chain option

1 Like

Just edit crontab:

certbot renew; sed -i --follow-symlinks '61,$ d' /etc/letsencrypt/live/XXXX.XXX/fullchain.pem

Unfortunately, this solution is very fragile, since it depends on the cross-signed certificate starting at a very specific line offset in the file. That won't always be the case.

1 Like

And the cut might actually come after certbot is all done which it might have already restarted the web server which would use the longer file.
[and then the file would get cut]

Is this really needed now when the old DST root has expired?

There are still two possible trust paths to choose from. [there were three]
If both work for you, then it won't matter,
But if only one does work, then you may still need to specify that one path until such time that is the default path or the only path left.

1 Like