Help thread for DST Root CA X3 expiration (September 2021)

Wait, I don't understand. DST CA X3 is the retired root cert, right? How is that preferred? I need help understanding this. ISRG Root X1 is the one we need to add to some services that can't connect anymore...

1 Like

@matthkarl
Wait!
Wasn't 465 was already encrypted?
How can 25 not answer?
How do you get inbound emails?
[via relay only?]

2 Likes

As I represent some websites, and we can't just tell potential website visitors to upgrade software that simply isn't upgrade-able (mac mavericks will never go past chrome 67, for example)
so, we must use a different certificate with a root certificate that is universally trusted... (any ideas?)

if you are using certbot as an ACME client to get LE certificates, you can change the ACME server to use a different provider, I'm still in the process of looking into this (sslforfree.com via zerossl.com)
I believe the ACME protocol requires free 90 day certs, and zerossl does offer that, apparently...)
User Guide β€” Certbot 1.19.0.dev0 documentation

So, there are several ACME service providers
https://www.xf.is/2020/06/30/list-of-free-acme-ssl-providers/

So, after some examination, you only need to use the server flag on the certbot command
--server 'https://api.buypass.com/acme/directory' for example

  1. register with a new ACME service provider
  2. get a certificate from them, and do your usual stuff after.
2 Likes

Thank you for your help.
No. I developed on windows (hence I have to use pfx, otherwise I cannot try), but production server is Ubuntu 18.04, where I commented DST CA X3 using
sudo dpkg-reconfigure ca-certificates
I am not using IIS nor ngnix, but embedded web server kestrel.

Now I executed
openssl s_client -connect hub1.cet.cloud:443 -servername hub1.cet.cloud
on an Ubuntu 20.04 PC, with same result (here /etc/ca-certificates.conf contains DST CA X3 commented, and ISRG Root X1 listed and enabled).
Perhaps I am misunderstanding: when requesting certificates using "openssl s_client", does server answers with complete chain (teorically)? Can we see complete response some way?

2 Likes

@rg305

I was too inpatient. After waiting longer I got this.

140421228041536:error:0200206E:system library:connect:Connection timed out:../crypto/bio/b_sock2.c:110:
140421228041536:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:111:
connect:errno=110

So it seems, STARTTLS is not working.

1 Like

I see:

curl -Iki hub1.cet.cloud
HTTP/1.1 307 Temporary Redirect
Date: Fri, 01 Oct 2021 14:41:48 GMT
Server: Kestrel
Location: https://hub1.cet.cloud/

KESTREL?
Is it using the fullchian.pem?

2 Likes

It's NOT a requirement.

2 Likes

No, it uses another file, freecert.pfx, produced with command
openssl pkcs12 -inkey /etc/letsencrypt/live/hub1.cet.cloud/privkey.pem -in /etc/letsencrypt/live/hub1.cet.cloud/fullchain.pem -export -out freecert.pfx -password pass:[mypassword]

1 Like

@criton2000
Well then is is having issues with the PFX file (or contents therein) or is unable to match the chain entries to whichever root store it uses.
It fails to serve the provided chain.
[mind of its' own]

---
Certificate chain
 0 s:/CN=hub1.cet.cloud
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
1 Like

Hi,

Our domains have been affected by the DST Root CA X3 expiration.

The only solution we found was to update certbot to at least 1.12 to have access to the --prefered-chain command to force ISRG Root X1.

The problem is that certbot only goes up to 0.31 with apt.

We tried snapd but it seems to not be compatible with our Ubuntu 16.04 (error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount: unknown filesystem type 'squashfs')

Same goes for pip installation, which shows errors that seems to be linked to our python version.

Any help would be appreciated to update our certbot version, like many others, our system is down and clients are waiting.

Thanks !

1 Like

I thought the same, but
openssl pkcs12 -info -in freecert.pfx
produces (see original help request for complete message)

Certificate bag
Bag Attributes: <No Attributes>
subject=/C=US/O=Let's Encrypt/CN=R3
issuer=/C=US/O=Internet Security Research Group/CN=ISRG Root X1

Where ***** does it find reference to DST CA X3?

1 Like

Drop it (for now) and use another ACME client - like: acme.sh

2 Likes

Welcome to the community forum! Yes, there are several working and specific solutions that many posters have adopted to resolve issues client side. However the specific solution depends on which of the two expirations are causing you problems and what your set-up is like.

In this thread, many users have discovered they are sending an incorrect chain or no chain at all. The solution is to update the configuration specific to your server to serve the correct the chain. You can use the search field on this thread or in the forum generally to see if information has posted and resolved for your server.

The other problem is around which chain you are sending particularly for the roots. There are trade-offs for each one and you can read about them here under the RSA changes for May 4th Production Chain Changes

Those are the main server side options you can take to still use Let’s Encrypt. There are some client side problems related to cached certificates and chain building behavior that have specific solutions as well.

Please begin by searching on the forum to see if your specific setup has been asked about. If you cannot find it, you will need to include some basic information like what is running server-side, what problem you are seeing, and your domain.

4 Likes

The toe bone is connected to the foot bone...
The foot bone is connected to the heal bone...

End-leaf cert connects to R3
R3 connects to X1
X1 connects to X3

Even if you don't explicitly provide that info - it is in there - like DNA!

3 Likes

But that chain contains

-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

that is https://letsencrypt.org/certs/lets-encrypt-r3.pem, which refers to X1 self signed, or not (I do not know how to link to original help request)?

1 Like

? ? ?

I'm not arguing about what that file contains or not.
What I've said, and the test show, is that the system is NOT serving a trusted chain.
This is all of what I get directly from it:

-----BEGIN CERTIFICATE-----
MIIFIjCCBAqgAwIBAgISA6bJd9Q/QePvDBKsI+penHAdMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MzAxMDQwMzVaFw0yMTEyMjkxMDQwMzRaMBkxFzAVBgNVBAMT
Dmh1YjEuY2V0LmNsb3VkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
2QKa90ruzeJvuDpsz3xLGZecF8xIwOhF2WCgWpj1b0rk7m9eWHJ/CnyvL2bkPYgj
xwIJ1SnhoIhRoUg2acVmw7HeP8/2ixJ4w+B4SBcsfnHk+yF/W0PY11hSCyikDmoH
FAFygZPKjsW6pcvoYvkaH5TjkoKLoAatCELPgA8+bDGvbus4KwKFuqclvHxH4602
2m1IVgdUrm0Yod4bhVzJVjz8Mbn/q9E9lEUZAZNH5nDOdF9Xgjb1YdQJdNFwUfaH
hm01XFv1huYTb6IhhYXyOU8xUxuK4pKGSFIhVdmlnjVZwsWNzm01vgy5/F13mnGa
IwNiSlsHFKcF46dJ1YXPGwIDAQABo4ICSTCCAkUwDgYDVR0PAQH/BAQDAgWgMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1Ud
DgQWBBScx2SaR2DlHI8kNlJzXHTkSkIkVDAfBgNVHSMEGDAWgBQULrMXt1hWy65Q
CUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9y
My5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3Jn
LzAZBgNVHREEEjAQgg5odWIxLmNldC5jbG91ZDBMBgNVHSAERTBDMAgGBmeBDAEC
ATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNl
bmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2AH0+8viP/4hVaCTC
wMqeUol5K8UOeAl/LmqXaJl+IvDXAAABfDaBmJgAAAQDAEcwRQIhAJY0gQ/vQEWR
1fwn+ML/RA1BKvqTtis7pVUnK/JMtVlZAiAGgtsPJ/IvB66eidubtJK5yHM9aIoo
CZsB3bLNUx0KxwB2AG9Tdqwx8DEZ2JkApFEV/3cVHBHZAsEAKQaNsgiaN9kTAAAB
fDaBmUIAAAQDAEcwRQIhAP9nXqL7HtpVBYBSbcPb2kiWyDpGzJe1rSubYd/r/pHA
AiA0bI92IbJx4wW3P6lPrs/c1feK4P7qH2pDnZelCboz8zANBgkqhkiG9w0BAQsF
AAOCAQEApasjJEtVS+k/xon8uzzY2r9e3ogwKOShEX1y+O/8q8y2adaVNh6/R8rz
/A7Fja4EwnMfEBIlew6SzLWBKWoJ3oF2TXee2/E0Z6/gBC0imlBG/po1RwVrldck
Zh7TSAAlqjNZfPsz+CSuMXqZ4/D30lgLpB0nrqL1rNxJrhLphNSOzkOmNRF3xokM
dzCsOl5+9k2MXWQqAZSEXXKdt/qVgc2CVzD3Q5KODsPv8nNNsQcq0+Bj5VAdO3Nv
nrD5lTVs31Ykf4BXslw8vEma5SDxnhNXOoW7TJOtMLE+f2mdLa+6mNZazyBv8Uza
tGgku+XQSB2XAEH0xY6+2uEbGCxQxQ==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
/xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
-----END CERTIFICATE-----

That second cert expired yesterday!

2 Likes

Hi there,

we are using IIS 10.0 and have to support old Android devices, so it is important that our server provides both paths, including the "New Default Chain" with IdenTrust DST Root CA X3.

Sadly, the IIS removes the path including the expired IdentTrust Certificate.

The Workaround we found was putting the "ISGR Root X1" in the Untrusted Store of the Windows Certificate Manager as described here and reboot the server.

Is there a better way to do this? This way we are unable to renew the certificates using win-acme.

Thanks!

1 Like

It is astounding, but explains perfectly what happens. Please, can you tell me how you have obtained that response (two certificates, second being probably the obsolete R3)?

1 Like

openssl s_client -connect EXAMPLE.COM:443 -servername EXAMPLE.COM -showcerts
[simply copy out the certs shown]

2 Likes

Hello, we are using centos 7. I updated the ca-certificates package and checked to make sure it's removed trust list | grep -C3 'DST Root CA X3', however, I'm still seeing the root certificate listed when I check our sites:

Is a server reboot required, or is there another way to clear this certificate from the chain?

99% of our users can access the site without any issue. We are having issues with users that have older device (mostly old MacBooks)

1 Like