Hi.
I'm hosting a DoT-Server on ubuntu. Since today, almost 3 Minutes after the X3-expiration Android won't connect to that DoT-Server anymore.
This also happens with dot1.applied-privacy.net (a public DoT-Server using LE for the TLS-Cert).
This behaviour was repoted to me from ppl using following Phones on current OS-Versions availible:
My unbound-server is running version 1.12.0, if that helps.
Not from what I can see:
---
Certificate chain
0 s:/C=US/ST=California/L=San Francisco/O=OpenDNS, Inc./CN=dot1.applied-privacy.net
i:/O=Cisco/CN=Cisco Umbrella Secondary SubCA mia-SG
1 s:/O=Cisco/CN=Cisco Umbrella Secondary SubCA mia-SG
i:/C=US/ST=California/L=San Francisco/O=Cisco/CN=Cisco Umbrella Primary SubCA
2 s:/C=US/ST=California/L=San Francisco/O=Cisco/CN=Cisco Umbrella Primary SubCA
i:/O=Cisco/CN=Cisco Umbrella Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDWjCCAkKgAwIBAgIEYVa5dDANBgkqhkiG9w0BAQsFADBAMQ4wDAYDVQQKDAVD
aXNjbzEuMCwGA1UEAwwlQ2lzY28gVW1icmVsbGEgU2Vjb25kYXJ5IFN1YkNBIG1p
YS1TRzAeFw0yMTA5MjgxNjA4MjhaFw0yMTEwMDMxNjA4MjhaMHUxCzAJBgNVBAYT
AlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2Nv
MRYwFAYDVQQKDA1PcGVuRE5TLCBJbmMuMSEwHwYDVQQDDBhkb3QxLmFwcGxpZWQt
cHJpdmFjeS5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDUrkpD
mIWc5smQMBrrQa75Z05k7lFx4A5LJYMuREhNT7crUNuaOCahwKIU8cBlltmcwdfz
u/vEib/D4LeFSG3wGpmSY4U/YvOEjly+2Xf0McW/jbHKpA3EQ6S/tgOpSx5A0WxT
p1SAycG/6/1MXnclao5xzL/U5tBaEp9YnTTBK1ylHAFkfujMqw07pyNYaNBz1pwf
OQZyK1SraG8nUIPofE3wYsZusZxCIZHV1cucY+JhFGxkn5JYnHtc5jqvm5ws57F6
PK4z94meJVJfCZ4qQBDtkFLD5aFUAE04v4agcPz07a8IePQZeIpzhFXKrDir6nZW
g8bS2u6wOIQgngPpAgMBAAGjJzAlMCMGA1UdEQQcMBqCGGRvdDEuYXBwbGllZC1w
cml2YWN5Lm5ldDANBgkqhkiG9w0BAQsFAAOCAQEAFPj6+yntXkew4DAmtNvv3a93
4aEXVSLKStynyuRliWJ8rHDXqfccScTXjlj3qvQPYQ4Lr5Zh9O4+2npe73u6Hkrr
nvGtSCIY5ShSlZo6sC7QndeJOnkIHEyUjbWogxEfCt4JtGmdqO3NW3D9qNvZt2od
pO4QoZPxvLCBrWl/L9Kgeg4q8cPysV1K/5z/GceAVL1yvU14luivs1BV8kdnLicw
tsK3/+fpj3vyC4XPxb+2vxjR4ikGqsGOUF6+xlREfMGMF48b4xhI7UaEqJuFBcOy
xfd/uJNvKAKV9Mo4uDIuFDEnMChiNVmd+4ByG6J+V7rb4USirllcBmb2LIA5PA==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=San Francisco/O=OpenDNS, Inc./CN=dot1.applied-privacy.net
issuer=/O=Cisco/CN=Cisco Umbrella Secondary SubCA mia-SG
---
openssl s_client -showcerts -servername dot1.applied-privacy.net -connect dot1.applied-privacy.net:853
This says:
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = doh.applied-privacy.net
verify return:1
CONNECTED(00000003)
Android 11's DoT does not accept the "New Default Chain", apparently. We now cross-tried two other services with at least two other phones. No other client seems to have an issue with the same services.
I wonder if the "Alternative Chain" would help. Very old Androids shouldn't be an issue.
@rg305: That's interesting, I get the same result as @jsuelwald from where I'm sitting:
$ openssl s_client -connect dot1.applied-privacy.net:443 | egrep 'subject|issuer'
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = doh.applied-privacy.net
verify return:1
subject=CN = doh.applied-privacy.net
issuer=C = US, O = Let's Encrypt, CN = R3
@vcunat, thanks for sharing. That's very interesting to hear that there might be an issue with DoT on Android specifically. I wonder if it uses a different, or differently configured, TLS stack.
And so does SSL Labs...
I don't use anything CISCO nor Umbrellas (indoors) so I know it's not form any of my systems 
Another service for reference: odvr.nic.cz:853 (should work without SNI, too)
My DoT server also uses Letsencrypt. DoT on Android 9/10/11 devices is broken right now.
I have tested on some samsung devices and lots of Poco/Redmi devices.
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ishanjain.me
verify return:1
---
Certificate chain
0 s:CN = ishanjain.me
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFVTCCBD2gAwIBAgISAzLY9S1SlUIgeJ7uK7CKjLDNMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MzAxNDQyMjZaFw0yMTEyMjkxNDQyMjVaMBcxFTATBgNVBAMT
DGlzaGFuamFpbi5tZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMh
SyKvGCtadmYjjndhLLC+n9hbskbmTTP0+LFWWMTr61ViUeTio8mkATv4N/7emvRw
SwP+mYs3CXEjdy3lB94XwE2abTS0YcjFi6loJocmGExxuZUwoWP9eymkYQGOGKCS
SDFeovVQR6EHwLol4rEX/yvWTI56aRibNEGAThYOY9OrZH6SNJTlJw4yQh3A2M/I
uagCXc20iSGj4i1X5+ASaH1SIbbCQFa1gAKIdlMEgMBj57qvC92jxp4oPziku5Uh
ACcnbiv18KegiMsOXkOFZubyDykkNgC8W1ws48UTIm+73MxltiIQi5GQGBlZPftT
yg9l5yyTV6+bVzyLKzMCAwEAAaOCAn4wggJ6MA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4E
FgQUNjcYFWt2fONni2YDHOyVqFsBqZ4wHwYDVR0jBBgwFoAUFC6zF7dYVsuuUAlA
5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRwOi8vcjMu
by5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNyLm9yZy8w
TQYDVR0RBEYwRIIOKi5kbnMuaXNoYW4ucHeCCiouaXNoYW4ucHeCDiouaXNoYW5q
YWluLm1lgghpc2hhbi5wd4IMaXNoYW5qYWluLm1lMEwGA1UdIARFMEMwCAYGZ4EM
AQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0
c2VuY3J5cHQub3JnMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHYAXNxDkv7mq0VE
sV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF8N18EZQAABAMARzBFAiEA7s02NVlm
7dywJL5Vat/rJyxk7YJqEu02LAGVfmJjie0CIBVQgxagCeYFgcRp8WDHF/AjaZNl
vbvJCTQMKTkA0ovlAHcA9lyUL9F3MCIUVBgIMJRWjuNNExkzv98MLyALzE7xZOMA
AAF8N18EXQAABAMASDBGAiEAp3a9GM+xC6yPAu+H/A5ng7vv/C4zNj82zfPrSNP8
034CIQDw9j3T6lPADGrrCOHQSlmackQ4wOtj0f8T/2kS/kCEyDANBgkqhkiG9w0B
AQsFAAOCAQEAiiaV/7ZI7BHPz7ABj/8tOMCw/IHqnogFeOz0JM7LphI8a8r9qteP
oagzO4FVKcdXIhmGQpG3bX6XZAV1PeBN/ukR4sNMjrYD/aT35P1BWeHjglApLlW7
qywGE1wOzo+c5yjRIIJpFCTeqXf6rIqBijZcrPjY+1HQs3zgQLC13f5p3TAkm45K
MIDeODEZ12YASmWJe3HPL4oFV1e/XA6txKGoK0rm7TkiWBscsugw+jU+hRPpfirw
oZ7QZZHJygkTpO312N2FtHyawQf/8MSzNmSXgX0BYs80OCm3AWysguDdO7BKpnDA
cSro07jWl7LwpcCpz/JH2gGRJfdydAcabw==
-----END CERTIFICATE-----
subject=CN = ishanjain.me
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4605 bytes and written 378 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_CHACHA20_POLY1305_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_CHACHA20_POLY1305_SHA256
Session-ID: 36D06AA4D6D55C354126026886B76A28F934A630A7C77F1F2183B217B00B0DA2
Session-ID-ctx:
Resumption PSK: 57B1BBAA214241845F17AF0610778ADE232B8308AEEF256BE64A9D1E37493264
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 604800 (seconds)
TLS session ticket:
0000 - 23 d8 e3 1a c9 78 42 fc-17 74 b5 10 25 d7 8b ee #....xB..t..%...
0010 - 79 9f d3 0b 6a fb 9b 0e-fd 71 18 4c b6 ee 64 93 y...j....q.L..d.
0020 - 92 2f 59 83 5b 11 f2 89-53 f4 f4 5f 6a fe fb 3a ./Y.[...S.._j..:
0030 - b8 f5 56 8f 52 5a 60 cd-8e 25 81 8a df 09 87 9c ..V.RZ`..%......
0040 - 03 85 43 ab ba b1 ba fb-54 b6 db 7b f8 1a 86 ef ..C.....T..{....
0050 - cf e5 85 97 2c 95 f3 77-99 1b f2 37 73 22 c1 6e ....,..w...7s".n
0060 - a4 63 93 cc 86 3e 23 29-93 a6 64 2d ca ed eb e3 .c...>#)..d-....
0070 - 4e N
Start Time: 1633019911
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
closed
My DoT server also uses Letsencrypt. DoT on Android 9/10/11 devices is broken right now.
I have tested on some samsung devices and lots of Poco/Redmi devices.
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ishanjain.me
verify return:1
---
Certificate chain
0 s:CN = ishanjain.me
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFVTCCBD2gAwIBAgISAzLY9S1SlUIgeJ7uK7CKjLDNMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MzAxNDQyMjZaFw0yMTEyMjkxNDQyMjVaMBcxFTATBgNVBAMT
DGlzaGFuamFpbi5tZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMh
SyKvGCtadmYjjndhLLC+n9hbskbmTTP0+LFWWMTr61ViUeTio8mkATv4N/7emvRw
SwP+mYs3CXEjdy3lB94XwE2abTS0YcjFi6loJocmGExxuZUwoWP9eymkYQGOGKCS
SDFeovVQR6EHwLol4rEX/yvWTI56aRibNEGAThYOY9OrZH6SNJTlJw4yQh3A2M/I
uagCXc20iSGj4i1X5+ASaH1SIbbCQFa1gAKIdlMEgMBj57qvC92jxp4oPziku5Uh
ACcnbiv18KegiMsOXkOFZubyDykkNgC8W1ws48UTIm+73MxltiIQi5GQGBlZPftT
yg9l5yyTV6+bVzyLKzMCAwEAAaOCAn4wggJ6MA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4E
FgQUNjcYFWt2fONni2YDHOyVqFsBqZ4wHwYDVR0jBBgwFoAUFC6zF7dYVsuuUAlA
5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRwOi8vcjMu
by5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNyLm9yZy8w
TQYDVR0RBEYwRIIOKi5kbnMuaXNoYW4ucHeCCiouaXNoYW4ucHeCDiouaXNoYW5q
YWluLm1lgghpc2hhbi5wd4IMaXNoYW5qYWluLm1lMEwGA1UdIARFMEMwCAYGZ4EM
AQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0
c2VuY3J5cHQub3JnMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHYAXNxDkv7mq0VE
sV6a1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF8N18EZQAABAMARzBFAiEA7s02NVlm
7dywJL5Vat/rJyxk7YJqEu02LAGVfmJjie0CIBVQgxagCeYFgcRp8WDHF/AjaZNl
vbvJCTQMKTkA0ovlAHcA9lyUL9F3MCIUVBgIMJRWjuNNExkzv98MLyALzE7xZOMA
AAF8N18EXQAABAMASDBGAiEAp3a9GM+xC6yPAu+H/A5ng7vv/C4zNj82zfPrSNP8
034CIQDw9j3T6lPADGrrCOHQSlmackQ4wOtj0f8T/2kS/kCEyDANBgkqhkiG9w0B
AQsFAAOCAQEAiiaV/7ZI7BHPz7ABj/8tOMCw/IHqnogFeOz0JM7LphI8a8r9qteP
oagzO4FVKcdXIhmGQpG3bX6XZAV1PeBN/ukR4sNMjrYD/aT35P1BWeHjglApLlW7
qywGE1wOzo+c5yjRIIJpFCTeqXf6rIqBijZcrPjY+1HQs3zgQLC13f5p3TAkm45K
MIDeODEZ12YASmWJe3HPL4oFV1e/XA6txKGoK0rm7TkiWBscsugw+jU+hRPpfirw
oZ7QZZHJygkTpO312N2FtHyawQf/8MSzNmSXgX0BYs80OCm3AWysguDdO7BKpnDA
cSro07jWl7LwpcCpz/JH2gGRJfdydAcabw==
-----END CERTIFICATE-----
subject=CN = ishanjain.me
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4605 bytes and written 378 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_CHACHA20_POLY1305_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_CHACHA20_POLY1305_SHA256
Session-ID: 36D06AA4D6D55C354126026886B76A28F934A630A7C77F1F2183B217B00B0DA2
Session-ID-ctx:
Resumption PSK: 57B1BBAA214241845F17AF0610778ADE232B8308AEEF256BE64A9D1E37493264
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 604800 (seconds)
TLS session ticket:
0000 - 23 d8 e3 1a c9 78 42 fc-17 74 b5 10 25 d7 8b ee #....xB..t..%...
0010 - 79 9f d3 0b 6a fb 9b 0e-fd 71 18 4c b6 ee 64 93 y...j....q.L..d.
0020 - 92 2f 59 83 5b 11 f2 89-53 f4 f4 5f 6a fe fb 3a ./Y.[...S.._j..:
0030 - b8 f5 56 8f 52 5a 60 cd-8e 25 81 8a df 09 87 9c ..V.RZ`..%......
0040 - 03 85 43 ab ba b1 ba fb-54 b6 db 7b f8 1a 86 ef ..C.....T..{....
0050 - cf e5 85 97 2c 95 f3 77-99 1b f2 37 73 22 c1 6e ....,..w...7s".n
0060 - a4 63 93 cc 86 3e 23 29-93 a6 64 2d ca ed eb e3 .c...>#)..d-....
0070 - 4e N
Start Time: 1633019911
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
closed
Hi, I can confirm with unbound 1.13 on FreeBSD, using openssl 1.1.1l
Phone is a One plus Nord with latest OS level (Android 11).
I have no need to support old device, is there a way to renew my cert without cross-signing ? (may depend on my ACME client, which is dehydrated
Cisco Umbrella seems to be a company https-proxy.
Regarding my own Problem: Renewing the cert with certbot and --preferred-chain="ISRG Root X1" did indeed help. After doing that my android happily accepts connections to my own DoT-Server again.
I've posted the same issue (and now solution) on reddit as well:
Regarding certbot - why is ubuntu's version using apt that ... old?
But im not sure which solution was indeed needed here:
Most likely both were needed. The preferred chain flag is what did the trick, but it's a relatively new flag, introduced last year if I recall correctly.
By the way, for any readers: you don't need to renew; you can also edit fullchain.pem to swap out the long chain for the short one, and then reload or restart your DNS server.
By "swap" you mean just dropping the expired cert? (It wasn't clear to me if the next one keeps the same or not.)
How do you do this : I see 3 certificates. If they are (from top to bottom) 1 2 3, should I set it to 1 3 2 ?
Ah, thanks for the clarifying question. I was indeed not being clear. Yes, removing the last cert from fullchain.pem should fix this particular problem. That will result in you serving the shorter chain. Note that that's a temporary solution and will be overwritten by the next renewal. You would need to also make sure to add --preferred-chain in time for the next renewal.
Yeah, already edited /etc/cron.d/certbot (appended --preferred-chain there)
certbot -q renew --preferred-chain="ISRG Root X1"
at least it doesn't complain about that 