Android devices with DoT configured; interaction with new default chain

Another service for reference: odvr.nic.cz:853 (should work without SNI, too)

My DoT server also uses Letsencrypt. DoT on Android 9/10/11 devices is broken right now.
I have tested on some samsung devices and lots of Poco/Redmi devices.

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ishanjain.me
verify return:1
---
Certificate chain
 0 s:CN = ishanjain.me
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFVTCCBD2gAwIBAgISAzLY9S1SlUIgeJ7uK7CKjLDNMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MzAxNDQyMjZaFw0yMTEyMjkxNDQyMjVaMBcxFTATBgNVBAMT
DGlzaGFuamFpbi5tZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMh
SyKvGCtadmYjjndhLLC+n9hbskbmTTP0+LFWWMTr61ViUeTio8mkATv4N/7emvRw
SwP+mYs3CXEjdy3lB94XwE2abTS0YcjFi6loJocmGExxuZUwoWP9eymkYQGOGKCS
SDFeovVQR6EHwLol4rEX/yvWTI56aRibNEGAThYOY9OrZH6SNJTlJw4yQh3A2M/I
uagCXc20iSGj4i1X5+ASaH1SIbbCQFa1gAKIdlMEgMBj57qvC92jxp4oPziku5Uh
ACcnbiv18KegiMsOXkOFZubyDykkNgC8W1ws48UTIm+73MxltiIQi5GQGBlZPftT
yg9l5yyTV6+bVzyLKzMCAwEAAaOCAn4wggJ6MA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4E
FgQUNjcYFWt2fONni2YDHOyVqFsBqZ4wHwYDVR0jBBgwFoAUFC6zF7dYVsuuUAlA
5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRwOi8vcjMu
by5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNyLm9yZy8w
TQYDVR0RBEYwRIIOKi5kbnMuaXNoYW4ucHeCCiouaXNoYW4ucHeCDiouaXNoYW5q
YWluLm1lgghpc2hhbi5wd4IMaXNoYW5qYWluLm1lMEwGA1UdIARFMEMwCAYGZ4EM
AQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0
c2VuY3J5cHQub3JnMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHYAXNxDkv7mq0VE
sV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF8N18EZQAABAMARzBFAiEA7s02NVlm
7dywJL5Vat/rJyxk7YJqEu02LAGVfmJjie0CIBVQgxagCeYFgcRp8WDHF/AjaZNl
vbvJCTQMKTkA0ovlAHcA9lyUL9F3MCIUVBgIMJRWjuNNExkzv98MLyALzE7xZOMA
AAF8N18EXQAABAMASDBGAiEAp3a9GM+xC6yPAu+H/A5ng7vv/C4zNj82zfPrSNP8
034CIQDw9j3T6lPADGrrCOHQSlmackQ4wOtj0f8T/2kS/kCEyDANBgkqhkiG9w0B
AQsFAAOCAQEAiiaV/7ZI7BHPz7ABj/8tOMCw/IHqnogFeOz0JM7LphI8a8r9qteP
oagzO4FVKcdXIhmGQpG3bX6XZAV1PeBN/ukR4sNMjrYD/aT35P1BWeHjglApLlW7
qywGE1wOzo+c5yjRIIJpFCTeqXf6rIqBijZcrPjY+1HQs3zgQLC13f5p3TAkm45K
MIDeODEZ12YASmWJe3HPL4oFV1e/XA6txKGoK0rm7TkiWBscsugw+jU+hRPpfirw
oZ7QZZHJygkTpO312N2FtHyawQf/8MSzNmSXgX0BYs80OCm3AWysguDdO7BKpnDA
cSro07jWl7LwpcCpz/JH2gGRJfdydAcabw==
-----END CERTIFICATE-----
subject=CN = ishanjain.me

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4605 bytes and written 378 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_CHACHA20_POLY1305_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_CHACHA20_POLY1305_SHA256
    Session-ID: 36D06AA4D6D55C354126026886B76A28F934A630A7C77F1F2183B217B00B0DA2
    Session-ID-ctx:
    Resumption PSK: 57B1BBAA214241845F17AF0610778ADE232B8308AEEF256BE64A9D1E37493264
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 23 d8 e3 1a c9 78 42 fc-17 74 b5 10 25 d7 8b ee   #....xB..t..%...
    0010 - 79 9f d3 0b 6a fb 9b 0e-fd 71 18 4c b6 ee 64 93   y...j....q.L..d.
    0020 - 92 2f 59 83 5b 11 f2 89-53 f4 f4 5f 6a fe fb 3a   ./Y.[...S.._j..:
    0030 - b8 f5 56 8f 52 5a 60 cd-8e 25 81 8a df 09 87 9c   ..V.RZ`..%......
    0040 - 03 85 43 ab ba b1 ba fb-54 b6 db 7b f8 1a 86 ef   ..C.....T..{....
    0050 - cf e5 85 97 2c 95 f3 77-99 1b f2 37 73 22 c1 6e   ....,..w...7s".n
    0060 - a4 63 93 cc 86 3e 23 29-93 a6 64 2d ca ed eb e3   .c...>#)..d-....
    0070 - 4e                                                N

    Start Time: 1633019911
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed
1 Like

My DoT server also uses Letsencrypt. DoT on Android 9/10/11 devices is broken right now.
I have tested on some samsung devices and lots of Poco/Redmi devices.

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ishanjain.me
verify return:1
---
Certificate chain
 0 s:CN = ishanjain.me
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFVTCCBD2gAwIBAgISAzLY9S1SlUIgeJ7uK7CKjLDNMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MzAxNDQyMjZaFw0yMTEyMjkxNDQyMjVaMBcxFTATBgNVBAMT
DGlzaGFuamFpbi5tZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMh
SyKvGCtadmYjjndhLLC+n9hbskbmTTP0+LFWWMTr61ViUeTio8mkATv4N/7emvRw
SwP+mYs3CXEjdy3lB94XwE2abTS0YcjFi6loJocmGExxuZUwoWP9eymkYQGOGKCS
SDFeovVQR6EHwLol4rEX/yvWTI56aRibNEGAThYOY9OrZH6SNJTlJw4yQh3A2M/I
uagCXc20iSGj4i1X5+ASaH1SIbbCQFa1gAKIdlMEgMBj57qvC92jxp4oPziku5Uh
ACcnbiv18KegiMsOXkOFZubyDykkNgC8W1ws48UTIm+73MxltiIQi5GQGBlZPftT
yg9l5yyTV6+bVzyLKzMCAwEAAaOCAn4wggJ6MA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4E
FgQUNjcYFWt2fONni2YDHOyVqFsBqZ4wHwYDVR0jBBgwFoAUFC6zF7dYVsuuUAlA
5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRwOi8vcjMu
by5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNyLm9yZy8w
TQYDVR0RBEYwRIIOKi5kbnMuaXNoYW4ucHeCCiouaXNoYW4ucHeCDiouaXNoYW5q
YWluLm1lgghpc2hhbi5wd4IMaXNoYW5qYWluLm1lMEwGA1UdIARFMEMwCAYGZ4EM
AQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0
c2VuY3J5cHQub3JnMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHYAXNxDkv7mq0VE
sV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF8N18EZQAABAMARzBFAiEA7s02NVlm
7dywJL5Vat/rJyxk7YJqEu02LAGVfmJjie0CIBVQgxagCeYFgcRp8WDHF/AjaZNl
vbvJCTQMKTkA0ovlAHcA9lyUL9F3MCIUVBgIMJRWjuNNExkzv98MLyALzE7xZOMA
AAF8N18EXQAABAMASDBGAiEAp3a9GM+xC6yPAu+H/A5ng7vv/C4zNj82zfPrSNP8
034CIQDw9j3T6lPADGrrCOHQSlmackQ4wOtj0f8T/2kS/kCEyDANBgkqhkiG9w0B
AQsFAAOCAQEAiiaV/7ZI7BHPz7ABj/8tOMCw/IHqnogFeOz0JM7LphI8a8r9qteP
oagzO4FVKcdXIhmGQpG3bX6XZAV1PeBN/ukR4sNMjrYD/aT35P1BWeHjglApLlW7
qywGE1wOzo+c5yjRIIJpFCTeqXf6rIqBijZcrPjY+1HQs3zgQLC13f5p3TAkm45K
MIDeODEZ12YASmWJe3HPL4oFV1e/XA6txKGoK0rm7TkiWBscsugw+jU+hRPpfirw
oZ7QZZHJygkTpO312N2FtHyawQf/8MSzNmSXgX0BYs80OCm3AWysguDdO7BKpnDA
cSro07jWl7LwpcCpz/JH2gGRJfdydAcabw==
-----END CERTIFICATE-----
subject=CN = ishanjain.me

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4605 bytes and written 378 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_CHACHA20_POLY1305_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_CHACHA20_POLY1305_SHA256
    Session-ID: 36D06AA4D6D55C354126026886B76A28F934A630A7C77F1F2183B217B00B0DA2
    Session-ID-ctx:
    Resumption PSK: 57B1BBAA214241845F17AF0610778ADE232B8308AEEF256BE64A9D1E37493264
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 23 d8 e3 1a c9 78 42 fc-17 74 b5 10 25 d7 8b ee   #....xB..t..%...
    0010 - 79 9f d3 0b 6a fb 9b 0e-fd 71 18 4c b6 ee 64 93   y...j....q.L..d.
    0020 - 92 2f 59 83 5b 11 f2 89-53 f4 f4 5f 6a fe fb 3a   ./Y.[...S.._j..:
    0030 - b8 f5 56 8f 52 5a 60 cd-8e 25 81 8a df 09 87 9c   ..V.RZ`..%......
    0040 - 03 85 43 ab ba b1 ba fb-54 b6 db 7b f8 1a 86 ef   ..C.....T..{....
    0050 - cf e5 85 97 2c 95 f3 77-99 1b f2 37 73 22 c1 6e   ....,..w...7s".n
    0060 - a4 63 93 cc 86 3e 23 29-93 a6 64 2d ca ed eb e3   .c...>#)..d-....
    0070 - 4e                                                N

    Start Time: 1633019911
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed
1 Like

Hi, I can confirm with unbound 1.13 on FreeBSD, using openssl 1.1.1l
Phone is a One plus Nord with latest OS level (Android 11).

I have no need to support old device, is there a way to renew my cert without cross-signing ? (may depend on my ACME client, which is dehydrated

Cisco Umbrella seems to be a company https-proxy.

1 Like

Regarding my own Problem: Renewing the cert with certbot and --preferred-chain="ISRG Root X1" did indeed help. After doing that my android happily accepts connections to my own DoT-Server again.

I've posted the same issue (and now solution) on reddit as well:

Regarding certbot - why is ubuntu's version using apt that ... old?

But im not sure which solution was indeed needed here:

  • update certbot
  • --preferred-chain="ISRG Root X1"
5 Likes

Most likely both were needed. The preferred chain flag is what did the trick, but it's a relatively new flag, introduced last year if I recall correctly.

2 Likes

By the way, for any readers: you don't need to renew; you can also edit fullchain.pem to swap out the long chain for the short one, and then reload or restart your DNS server.

5 Likes

By "swap" you mean just dropping the expired cert? (It wasn't clear to me if the next one keeps the same or not.)

1 Like

How do you do this : I see 3 certificates. If they are (from top to bottom) 1 2 3, should I set it to 1 3 2 ?

Ah, thanks for the clarifying question. I was indeed not being clear. Yes, removing the last cert from fullchain.pem should fix this particular problem. That will result in you serving the shorter chain. Note that that's a temporary solution and will be overwritten by the next renewal. You would need to also make sure to add --preferred-chain in time for the next renewal.

4 Likes

Yeah, already edited /etc/cron.d/certbot (appended --preferred-chain there)

certbot -q renew --preferred-chain="ISRG Root X1"

at least it doesn't complain about that :slight_smile:

1 Like

Not yet in dehydrated... (but option is in the --help of the latest git version)

1 Like

I'm really suprised the DoT stack of even recent Androids have this issue.. :confused:

I'd guess that there is a quite old (ancient?) OpenSSL-version in use.

I'm not surprised. The certificate validation of DoT on Android is somehow different from other validations as it also ignores user installed CAs. Sign in - Google Accounts

1 Like

The github version of dehydrated has a working --preferred-chain option

1 Like

Just edit crontab:

certbot renew; sed -i --follow-symlinks '61,$ d' /etc/letsencrypt/live/XXXX.XXX/fullchain.pem

Unfortunately, this solution is very fragile, since it depends on the cross-signed certificate starting at a very specific line offset in the file. That won't always be the case.

1 Like

And the cut might actually come after certbot is all done which it might have already restarted the web server which would use the longer file.
[and then the file would get cut]