Android devices with DoT configured; interaction with new default chain

My DoT server also uses Letsencrypt. DoT on Android 9/10/11 devices is broken right now.
I have tested on some samsung devices and lots of Poco/Redmi devices.

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ishanjain.me
verify return:1
---
Certificate chain
 0 s:CN = ishanjain.me
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFVTCCBD2gAwIBAgISAzLY9S1SlUIgeJ7uK7CKjLDNMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MzAxNDQyMjZaFw0yMTEyMjkxNDQyMjVaMBcxFTATBgNVBAMT
DGlzaGFuamFpbi5tZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMh
SyKvGCtadmYjjndhLLC+n9hbskbmTTP0+LFWWMTr61ViUeTio8mkATv4N/7emvRw
SwP+mYs3CXEjdy3lB94XwE2abTS0YcjFi6loJocmGExxuZUwoWP9eymkYQGOGKCS
SDFeovVQR6EHwLol4rEX/yvWTI56aRibNEGAThYOY9OrZH6SNJTlJw4yQh3A2M/I
uagCXc20iSGj4i1X5+ASaH1SIbbCQFa1gAKIdlMEgMBj57qvC92jxp4oPziku5Uh
ACcnbiv18KegiMsOXkOFZubyDykkNgC8W1ws48UTIm+73MxltiIQi5GQGBlZPftT
yg9l5yyTV6+bVzyLKzMCAwEAAaOCAn4wggJ6MA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4E
FgQUNjcYFWt2fONni2YDHOyVqFsBqZ4wHwYDVR0jBBgwFoAUFC6zF7dYVsuuUAlA
5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRwOi8vcjMu
by5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNyLm9yZy8w
TQYDVR0RBEYwRIIOKi5kbnMuaXNoYW4ucHeCCiouaXNoYW4ucHeCDiouaXNoYW5q
YWluLm1lgghpc2hhbi5wd4IMaXNoYW5qYWluLm1lMEwGA1UdIARFMEMwCAYGZ4EM
AQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0
c2VuY3J5cHQub3JnMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHYAXNxDkv7mq0VE
sV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF8N18EZQAABAMARzBFAiEA7s02NVlm
7dywJL5Vat/rJyxk7YJqEu02LAGVfmJjie0CIBVQgxagCeYFgcRp8WDHF/AjaZNl
vbvJCTQMKTkA0ovlAHcA9lyUL9F3MCIUVBgIMJRWjuNNExkzv98MLyALzE7xZOMA
AAF8N18EXQAABAMASDBGAiEAp3a9GM+xC6yPAu+H/A5ng7vv/C4zNj82zfPrSNP8
034CIQDw9j3T6lPADGrrCOHQSlmackQ4wOtj0f8T/2kS/kCEyDANBgkqhkiG9w0B
AQsFAAOCAQEAiiaV/7ZI7BHPz7ABj/8tOMCw/IHqnogFeOz0JM7LphI8a8r9qteP
oagzO4FVKcdXIhmGQpG3bX6XZAV1PeBN/ukR4sNMjrYD/aT35P1BWeHjglApLlW7
qywGE1wOzo+c5yjRIIJpFCTeqXf6rIqBijZcrPjY+1HQs3zgQLC13f5p3TAkm45K
MIDeODEZ12YASmWJe3HPL4oFV1e/XA6txKGoK0rm7TkiWBscsugw+jU+hRPpfirw
oZ7QZZHJygkTpO312N2FtHyawQf/8MSzNmSXgX0BYs80OCm3AWysguDdO7BKpnDA
cSro07jWl7LwpcCpz/JH2gGRJfdydAcabw==
-----END CERTIFICATE-----
subject=CN = ishanjain.me

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4605 bytes and written 378 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_CHACHA20_POLY1305_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_CHACHA20_POLY1305_SHA256
    Session-ID: 36D06AA4D6D55C354126026886B76A28F934A630A7C77F1F2183B217B00B0DA2
    Session-ID-ctx:
    Resumption PSK: 57B1BBAA214241845F17AF0610778ADE232B8308AEEF256BE64A9D1E37493264
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 23 d8 e3 1a c9 78 42 fc-17 74 b5 10 25 d7 8b ee   #....xB..t..%...
    0010 - 79 9f d3 0b 6a fb 9b 0e-fd 71 18 4c b6 ee 64 93   y...j....q.L..d.
    0020 - 92 2f 59 83 5b 11 f2 89-53 f4 f4 5f 6a fe fb 3a   ./Y.[...S.._j..:
    0030 - b8 f5 56 8f 52 5a 60 cd-8e 25 81 8a df 09 87 9c   ..V.RZ`..%......
    0040 - 03 85 43 ab ba b1 ba fb-54 b6 db 7b f8 1a 86 ef   ..C.....T..{....
    0050 - cf e5 85 97 2c 95 f3 77-99 1b f2 37 73 22 c1 6e   ....,..w...7s".n
    0060 - a4 63 93 cc 86 3e 23 29-93 a6 64 2d ca ed eb e3   .c...>#)..d-....
    0070 - 4e                                                N

    Start Time: 1633019911
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed
1 Like

Hi, I can confirm with unbound 1.13 on FreeBSD, using openssl 1.1.1l
Phone is a One plus Nord with latest OS level (Android 11).

I have no need to support old device, is there a way to renew my cert without cross-signing ? (may depend on my ACME client, which is dehydrated

Cisco Umbrella seems to be a company https-proxy.

1 Like

Regarding my own Problem: Renewing the cert with certbot and --preferred-chain="ISRG Root X1" did indeed help. After doing that my android happily accepts connections to my own DoT-Server again.

I've posted the same issue (and now solution) on reddit as well:

Regarding certbot - why is ubuntu's version using apt that ... old?

But im not sure which solution was indeed needed here:

  • update certbot
  • --preferred-chain="ISRG Root X1"
5 Likes

Most likely both were needed. The preferred chain flag is what did the trick, but it's a relatively new flag, introduced last year if I recall correctly.

2 Likes

By the way, for any readers: you don't need to renew; you can also edit fullchain.pem to swap out the long chain for the short one, and then reload or restart your DNS server.

5 Likes

By "swap" you mean just dropping the expired cert? (It wasn't clear to me if the next one keeps the same or not.)

1 Like

How do you do this : I see 3 certificates. If they are (from top to bottom) 1 2 3, should I set it to 1 3 2 ?

Ah, thanks for the clarifying question. I was indeed not being clear. Yes, removing the last cert from fullchain.pem should fix this particular problem. That will result in you serving the shorter chain. Note that that's a temporary solution and will be overwritten by the next renewal. You would need to also make sure to add --preferred-chain in time for the next renewal.

4 Likes

Yeah, already edited /etc/cron.d/certbot (appended --preferred-chain there)

certbot -q renew --preferred-chain="ISRG Root X1"

at least it doesn't complain about that :slight_smile:

1 Like

Not yet in dehydrated... (but option is in the --help of the latest git version)

1 Like

I'm really suprised the DoT stack of even recent Androids have this issue.. :confused:

I'd guess that there is a quite old (ancient?) OpenSSL-version in use.

I'm not surprised. The certificate validation of DoT on Android is somehow different from other validations as it also ignores user installed CAs. Sign in - Google Accounts

1 Like

The github version of dehydrated has a working --preferred-chain option

1 Like

Just edit crontab:

certbot renew; sed -i --follow-symlinks '61,$ d' /etc/letsencrypt/live/XXXX.XXX/fullchain.pem

Unfortunately, this solution is very fragile, since it depends on the cross-signed certificate starting at a very specific line offset in the file. That won't always be the case.

1 Like

And the cut might actually come after certbot is all done which it might have already restarted the web server which would use the longer file.
[and then the file would get cut]

Is this really needed now when the old DST root has expired?

@Forza
There are still two possible trust paths to choose from. [there were three]
If both work for you, then it won't matter,
But if only one does work, then you may still need to specify that one path until such time that is the default path or the only path left.

1 Like