After my latest renewal with certbot the SSL connection broke as Issuer changed from R3 to R11

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
platform.rgsgames.com
I ran this command:
echo | openssl s_client -connect platform.rgsgames.com:443
It produced this output:

$ echo | openssl s_client -connect platform.rgsgames.com:443
CONNECTED(00000004)
depth=0 CN = platform.rgsgames.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = platform.rgsgames.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = platform.rgsgames.com
verify return:1

Certificate chain
0 s:CN = platform.rgsgames.com
i:C = US, O = Let's Encrypt, CN = R11

Server certificate
-----BEGIN CERTIFICATE-----
MIIE+jCCA+KgAwIBAgISBOE143C4HBxkSEXhy3QBdCAqMA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTEwHhcNMjQwNjA4MDYzMzAzWhcNMjQwOTA2MDYzMzAyWjAgMR4wHAYDVQQD
ExVwbGF0Zm9ybS5yZ3NnYW1lcy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQC67ohB1elN3Dnkli3ynzjhv8HB+yVB+PVubnWDo3F+KQx2wuj3f069
LpQA3ELm5nPZbTCggobE1TcwCWb1SMClE/XL+nHFkrmX0ns3smMDZvQjb3GcLOMe
HiLnkwF0JifbE7DZQxL6dFdIxm9DRqy8rWV2TUIqGV96ro1VkyMWLM4LNCE7/+sF
Y96Lpf9w/mgeieF0pYTSmlbwmcycVjsUCDBauElC0nCf19YIdlubK5kFi6rTWLBs
9Fm5YMxlzgBLPxjTAG5xObC7vAjml3z4ie/WUyKa6tu36+lOtGLQmGEDtGhFjnH0
GViiECTyaWsxjF5P5CwwyTKamNFcKxzbAgMBAAGjggIZMIICFTAOBgNVHQ8BAf8E
BAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQC
MAAwHQYDVR0OBBYEFMEsTUPEDiwtxKiVfdw0dDC22KHNMB8GA1UdIwQYMBaAFMXP
RqTq9MPAemyVxC2wXpIvJuO5MFcGCCsGAQUFBwEBBEswSTAiBggrBgEFBQcwAYYW
aHR0cDovL3IxMS5vLmxlbmNyLm9yZzAjBggrBgEFBQcwAoYXaHR0cDovL3IxMS5p
LmxlbmNyLm9yZy8wIAYDVR0RBBkwF4IVcGxhdGZvcm0ucmdzZ2FtZXMuY29tMBMG
A1UdIAQMMAowCAYGZ4EMAQIBMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDwAHUAPxdL
T9ciR1iUHWUchL4NEu2QN38fhWrrwb8ohez4ZG4AAAGP9sPg3gAABAMARjBEAiBD
Fkgs21uKs6Wo449KQ8SbYcjupAcyPJjcZNr/spVSjgIgDjO+tFFce3V3BTmEteLK
ZIaDb1Z1IoniUbznWZXYrMYAdwAZmBBxCfDWUi4wgNKeP2S7g24ozPkPUo7u385K
Pxa0ygAAAY/2w+FsAAAEAwBIMEYCIQD1IwPQHiRiVnTMRtaX1Lgh5jqzg6xR4VJV
Kh/NqwnKfAIhALbybo2WAE+DE6AU2h7dGJ5gm1Ct1V/UTuZauQOEkvRaMA0GCSqG
SIb3DQEBCwUAA4IBAQAsJMl8GY/4Jpu70ygAQ4KEOLnEAG0FV3A0/URGemCIktNE
cjhdA04MadM6nohqBTVHa6yCcNc0FLyaVRPvLo5U0afCfCknvzvxbMOEpLt8o5za
CSoN8oogdIhukXrxuJ5S0t18il6wcYK4bNgJgTCV2QNXwo3rPfdnRRIj5fvAEIAo
kwOKrPPDsYP9IbPlhpgr2cM1Y1J6DX/PKwb1fqKM820fxxFrsxMrHoafCB4qpWYu
4ie+SwtwVnqUhXFRZt8KhRBkaZCURaE3vSMTTv9A72kWWL22EpAa8pJvS75CIqQm
EUyGw9ddaAXYIlnQp1/MNef7EuOXreeBgXAWgpBz
-----END CERTIFICATE-----
subject=CN = platform.rgsgames.com

issuer=C = US, O = Let's Encrypt, CN = R11


No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 1773 bytes and written 459 bytes
Verification error: unable to verify the first certificate

New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 623F629AA660DCD686BA62C6A351665FB31C61F2C9B96EB38212F4D910B31CED
Session-ID-ctx:
Master-Key: C1C1910E80399FAFEF9B7131FB3BE787A611C65049181913304B66DA66425B8C865C1AA6C8B04654D4511D43E7E5DB67
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1717936695
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended master secret: no

DONE

My web server is (include version):

apache but the renewed certificate is loaded on Citrix load balancer set as SSL Offload

The operating system my web server runs on is (include version):

CentOS7

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

The problem I have is when the client is an application server or an AWS CloudFront they read the output I posted above and they elaborate the certificate as "untrusted" due to the message present in the output:
"
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = platform.rgsgames.com
verify error:num=21:unable to verify the first certificate
verify return:1
"

The only one thing I can see here is that something changed in the chain, specifically with the CN; it was R3 before the last renewal and now it's R11.

How can this be fixed?
Please help.

1 Like

Actually send the entire chain instead of just the end leaf certificate, which is what the webserver (or I guess Citrix load balancer in your case) currently is doing.

If the webserver/Citrix load balancer didn't send the chain previously, the openssl command should have failed then too.

4 Likes

The renewal has been doing on since a long while automatically via certbot, performing the two main steps: renewal + upload the new cert to the LB.
Therefore, as per your suggestion, we need to change the way the certbot script upload the new certificate on citrix. Citrix just provide what we upload there. Do you think this will be fixed by verifying that on the citrix will be uploaded the fullchain.pem ?

those are the files generated and treated everytime the certificate gets renewed

privkey.pem -> ../../archive/platform.rgsgames.com/privkey25.pem
fullchain.pem -> ../../archive/platform.rgsgames.com/fullchain25.pem
chain.pem -> ../../archive/platform.rgsgames.com/chain25.pem
cert.pem -> ../../archive/platform.rgsgames.com/cert25.pem

Uploading fullchain.pem usually does the trick, but some systems require uploading the end leaf certificate (cert.pem) and the chain (chain.pem) separately. I'm not familiar with Citrix load balancer, so I don't know which method you require.

4 Likes

I don't know either Citrix techology I only know that this was working before and now it is not fully working. browsering the URL this error gets ingnored whereas via app server it is not due to the errors:

verify error:num=20:unable to get local issuer certificate
verify return:1
verify error:num=21:unable to verify the first certificate
verify return:1

Yes, because only the end leaf certificate without any intermediates are send:

---
Certificate chain
 0 s:CN = platform.rgsgames.com
   i:C = US, O = Let's Encrypt, CN = R11
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun  8 06:33:03 2024 GMT; NotAfter: Sep  6 06:33:02 2024 GMT
---

Usually, that's only because the browser has cached the intermediate from some other connection. Although some browsers can also use the AIA CA Issuers information to retrieve the intermediate and use it to build the chain to the root.

But in the end you must make sure whatever is sending the certificate also sends the intermediate.

The following is what you SHOULD see (as an example):

---
Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R11
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun  6 16:57:10 2024 GMT; NotAfter: Sep  4 16:57:09 2024 GMT
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
4 Likes

The below Citrix page describes how to setup the intermediate chain. I don't know anything about your device other than this. If you have any questions please refer them to Citrix.

I suggest this only as a good starting point

4 Likes

One reason this can suddenly be a problem is some browsers like Firefox (used to?) have the R3 intermediate bundled in their own certificate store so that meant servers got away with not serving the full chain (including intermediates).

Other browsers and client tools variously either needed to know the intermediate in advance or resolve it via the OS or other path resolution and some things may need so/software updates to bundle the new intermediates, but serving the full chain should solve that and is the more flexible solution.

2 Likes

Just to keep all of you posted on this issue. As most of you highlighted, the problem was on Citrix because the LB wasn't showed properly the certificate as with the renewed one LetsEncrypt changed the CA root from R3 to R11 and this wasn't done on LB side as well. So we had to go through the citrix console, install the new CA Root and link it to the domain where it changed: platform.rgsgames.com. I understood that from the Authority side, LetsEncrypt, this is something usual that may happen so now what I need to fix is the automation scripts to keep track of the CA Root change when it happens and have the new chain updated on Citrix as soon as it gets modified from LE, of course along with the new certificate.
I will look for this here in the LE community, maybe there was someone else who had to fix the same problem: "how to check certificate CA root on citrix and update it with certbot".
Thanks all for your precious help.

1 Like

R3/R11 are NOT roots.

Please note that at renewal, the CA will randomly issue from R11 or R10!! (Or E5/E6 for ECDSA certs.)

So if you're hardcoding the intermediate, you don't have a durable solution.

You should develop a method where at renewal the configured intermediate will also be updated.

7 Likes

lmaio,

Have you found a solution to this issue? We are currently facing a similar problem.

Our server, which is NodeJS-based, is hosted on an Azure VM and protected by Azure Front Door. We have been using Certbot to generate and renew our certificates without any issues for years, up until the most recent renewal last week. We are still trying to determine exactly what went wrong. We have not made any changes to the configurations of either Azure Front Door or our NodeJS server. The only difference we've noticed so far is a change in the certificate issuer from R3 to R11, as you also pointed out.

We would greatly appreciate any assistance in resolving this issue.

@alexshao regarding Azure Front Door (current edition or legacy 'classic'?)

  • Are you uploading your full chain or just your leaf/end-entity certificate?
  • Have you tried uploading your full chain to keyvault and referencing the cert from there?
  • Is there a reason you're not using Azure provided certificates?

[Edit: you pay azure for support, you should definitely use that facility]

6 Likes

The CA is also expected and guaranteed to change these to R12/R13[, etc...] and E7/E8[, etc ...] at any time in the future - by both the ACME spec and LetsEncrypt's Documentation. These Intermediate certificates should never be hardcoded into a deployment/integration.

5 Likes

This is how often it changes:
[root@rgs-auth01 letsencrypt]# for i in $(seq 1 27); do echo -n | openssl x509 -in /etc/letsencrypt/archive/platform.gi.rgsgames.com/cert$i.pem -issuer | grep 'issuer='; done
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
issuer= /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Let's Encrypt/CN=R11

the certbot scripts we use, they only push to the LB (Citrix or F5) cert and key files and if the intermediate randomly (not so randomly actually as I verified on my system) gets changed it will be left out. Do you know if there is a way in the certbot command invocation to take also this situation into account and push the fullchain as well when intermediate changes?

The intermediates will be changing more often from here on.

In your scripting, you should be using the chain.pem (for just the intermediate), or fullchain.pem (for the leaf plus intermediates), as needed to load into your systems. You should be loading them each and every time, since they might change every time.

4 Likes