Server missing R11 certificate?

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: campus.apafiq.org

I ran this command: echo | openssl s_client -connect campus.apafiq.org:443

It produced this output:

Connecting to 149.50.149.35
CONNECTED(00000003)
depth=0 CN=campus.apafiq.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=campus.apafiq.org
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN=campus.apafiq.org
verify return:1
---
Certificate chain
 0 s:CN=campus.apafiq.org
   i:C=US, O=Let's Encrypt, CN=R11
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 28 22:37:12 2025 GMT; NotAfter: Aug 26 22:37:11 2025 GMT
 1 s:C=US, O=Let's Encrypt, CN=R10
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIF/jCCBOagAwIBAgISBm6iM8D9jQTBRAig6vx1oOgJMA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTEwHhcNMjUwNTI4MjIzNzEyWhcNMjUwODI2MjIzNzExWjAcMRowGAYDVQQD
ExFjYW1wdXMuYXBhZmlxLm9yZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoC
ggIBALUOROZl7t+xt7njFGVR8U5PPZWnZUbP2zAmo2G/c3GvrFkAnqmKY6lroDJB
qBS8YsqhHSC2A1bk4QvjW3ch55+vGa/iEOvN9ftpo39o90/1qc5iYSIkIe7gHqy5
+PRXtJD/g3eSYz+9vN2F5dcaMFHxJHyWR07HgltO5FtmyFDgqdoU2Sitrz6nz18l
8qsZs5I+ahNutfsOfu6aqTM8jYaaS0y9Lcw84FEQWfcUgJup3s2A6SBW5qzoJsJF
IiKO94XZUUYGpElfhghd64Xw96DVGF4yHJZ50qsUR9mtk9D9APKQlH/BINfwsHIM
6OEhWo9pGG5B8QgLOKrJcoZt3j5yzpTRZaKRD0dXIqY5wWhQsJxxq9yq0s+clj1q
FYPVQkljnrSPNPgkS6kBJk82ZcIAkfBCOIFup1fUnBtgIXy/6SV+ieyvUa+0HwTH
SO8fbFAqdtELCSXxJKOwdwhzfY/4UksSGoLPpeOQ0iQHEEU4PF0Q6mKJX8PwGpck
B/naEPa5ZGvP0h3YD3zss5zDY9/BJniW7e+86s3gqjRA4uBzOgTO6x+4gYyl0sAF
nzZu+OnxPu+wuiWFfTy2Urv/oTBdD8hVwkqnAckQtKcplAW1CXtk0BdvpoYYZW8P
hCJLgde5LOCLzeduajCMnX6q0fJfxn5LbEPz5vm7dzI7Hg5FAgMBAAGjggIhMIIC
HTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
MAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFBYdgZeh73Zdh77p9NpUtvIMyeJbMB8G
A1UdIwQYMBaAFMXPRqTq9MPAemyVxC2wXpIvJuO5MDMGCCsGAQUFBwEBBCcwJTAj
BggrBgEFBQcwAoYXaHR0cDovL3IxMS5pLmxlbmNyLm9yZy8wHAYDVR0RBBUwE4IR
Y2FtcHVzLmFwYWZpcS5vcmcwEwYDVR0gBAwwCjAIBgZngQwBAgEwLgYDVR0fBCcw
JTAjoCGgH4YdaHR0cDovL3IxMS5jLmxlbmNyLm9yZy8xMC5jcmwwggEEBgorBgEE
AdZ5AgQCBIH1BIHyAPAAdgAS8U40vVNyTIQGGcOPP3oT+Oe1YoeInG0wBYTr5YYm
OgAAAZcZQG3+AAAEAwBHMEUCIEbAnItCud1ttRFJeUV7jZhoATmu+1p30tTN7hwm
9wz+AiEAwtlvJ2SXa+f9dJVCc7q8SM/yjNgl6diBnURU5TqrAC0AdgDtPEvW6AbC
pKIAV9vLJOI4Ad9RL+3EhsVwDyDdtz4/4AAAAZcZQHW7AAAEAwBHMEUCIQCXMuHh
uaIJQHwTQw9orqMNDsAurOrXLDsHP1XR1LItOQIgGqXP4tUkRfH/QB6nuFRZFQIu
2bptZmNrCu8WUVOWOUwwDQYJKoZIhvcNAQELBQADggEBAFWRPdRjF9z296vlz++h
js7W768OkdANnztfyeO9tYKq0SsLM1IWemISniMZ8AevI2ZDgbaGNt3rPUAqbZdT
8XVWD1vwgyDA7qOhhrihjQHXbsTOAtEJeXBY8fcrpnJsp8DuhQRAJ4YdySCyLUAM
HAGSgwubbOk9m+KZhuXXKfNAW/jPO+NRojLSB+iqW+VC/+auHAsH5JBNh8lwE3UT
FElTCmL+kcSu566g1fLvHJnv6mAd0mebBNHaQE+gqhvIQSxzSBl6T4ZKNvXPEBro
P/zyMGp7LatHPpsPfjKIIk59GbR8Me9tstEp/kDpebKWhjnhmzms1eq99ckU470h
ty0=
-----END CERTIFICATE-----
subject=CN=campus.apafiq.org
issuer=C=US, O=Let's Encrypt, CN=R11
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3648 bytes and written 417 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Server public key is 4096 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
DONE

My web server is (include version): httpd-2.4.62-1 (custom fork)

The operating system my web server runs on is (include version):
AlmaLinux 8x

My hosting provider, if applicable, is: ---

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
Provider's control panel

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): ---

Hi everyone, hope you're all doing well.

I having this situation where a server is presenting an incomplete chain when certs are issued by R11, but no errors when the issuer is R10.

For what I could gather it is the server who is at fault, not a Letsencrypt problem, but this is a VPS and the provider has not been helpful... since the site loads on most browsers it seems to be good enough for them.

But I do want to fix it and also learn how to do it.

So, to the point, this is actually a subdomain I am hosting, it has a proprietary control panel with a LE function that is not working in this case (I assume due to the main domain hosted elsewere) and thus I am using sslfree.io to manually generate the cerificate.

The one I got was issued by R11 and when browsed the cert chain is incomplete.
My very little knowledge on this makes me think the server is missing some R11 cert in it's vault.
Could it be so? And could someone explain to me how should I import it?

PS: In the past, with this same VPS I had the same issue when I created a couple of new sites, but I worked around that by deleting and recreating them and the certs where re-issued by R10 instead of R11 (the accounts were still empty so it was an option).

Thanks in advance!

1 Like
2

Welcome to the Let's Encrypt Community! :slightly_smiling_face:

This is my guess as well. In my experience, most "good" implementations directly use the certificate chain issued by Let's Encrypt during the certificate-acquisition process rather than storing or "pinning" intermediates, which leads to problems like this.

https://letsencrypt.org/certs/2024/r11.pem

Here in this community we strongly discourage using third-party services like this due to their potentially-questionable implementations that could violate Let's Encrypt's subscriber agreement due to the way account and certificate keys are handled (not to mention lack of automation). Even for "one-off" situations, I recommend using a vetted ACME client (I author :man_mage: CertSage as an example).

5 Likes
3

No, that's not it. At least, not related to any root certificate store if that's what you meant by "vault".

When a certificate is issued using ACME (such as with LE), next to the end leaf certificate it's also send the signing intermediate.

Usually an ACME client will install both at the same time, so the webserver automatically has the correct intermediate with the end leaf certificate.

Now, in this case it seems somebody manually messed up and hard-coded the R10 intermediate certificate, which is a WRONG thing to do entirely. Never hardcode intermediates.

If it's a VPS, what does the provider have to do with this? As far as I know, with a VPS the sysop of that VPS is in controle of the webserver, not the provider.

6 Likes
4

Ensure you configure your service to use the fullchain.pem instead of the cert.pem

3 Likes
5

I'm also confused about this part..

How is the "proprietary control panel" of your provider related to getting and installing a LE certificate on your VPS where you have root access? I don't see any issue with running e.g. Certbot or any other Linux based ACME client on your VPS to do this..

3 Likes
6

Hey, thanks for all the feedback and welcoming!

Ok, I'll elaborate more on the details.

It is a VPS that includes a control panel similar to cPanel but developed by the provider. It also includes a level of support and when the automated system failed to properly generate the certificates I reached out to them for assistance, as they rather have me using their control panel tools for this instead of doing it manually.

As far as I see their tool is a wrapper for certbot, but there's (IMHO) something clearly not working right because this particular issue has hit me several times now.

Maybe the issue is not completely on their tool, maybe it happens when the host is almalinux and not, say debian, but the thing is the problem is there.

The way their tools is sopposed to work is you set up the domain, request the certificate through and once it is succesfully created certbots will renew it by itself.
But it seems it won't allow me to request a cert for a subdomain so I decided to work around this using sslfree.io which has worked well for me in the past, in order to buy time and look for a permanent solution...

But alas, contrary to other certs this one got issued by R11 too. (I know that is expected and not an issue in itself)

About why the requests show R10 AND R11, AFAIK nobody hardcoded anything, my best guess is that maybe it's the result of my first request through the propietary tool (which was unable to get the cert) and my later installation of the one I got from sslfree.io.

I might also add that in a previous equivalent situation I did run certbot manually but couldn't fix the issue that way. That was another domain in this same server, but I didn't want to bring that up and make this more complex than it needs to be.

@webprofusion

Ensure you configure your service to use the fullchain.pem instead of the cert.pem

I sort of tried that in the past but couldn't figure it out. Any pointers you could give me?

Thank you all.

7

How does your Apache server get configured?

The problem is not so much in getting a cert but in what your Apache is configured to use

echo|openssl s_client -connect campus.apafiq.org:443 | head -18
depth=0 CN = campus.apafiq.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = campus.apafiq.org
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = campus.apafiq.org
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:CN = campus.apafiq.org
   i:C = US, O = Let's Encrypt, CN = R11
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 28 22:37:12 2025 GMT; NotAfter: Aug 26 22:37:11 2025 GMT
 1 s:C = US, O = Let's Encrypt, CN = R10
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT

Notice the first cert is your domain issued by R11. All fine and good.

But, the second one is R10. It should be the intermediate as given to your ACME client by Let's Encrypt. In this case it should be R11. It could be others.

It looks like your Apache refers to just the new leaf but has the (long deprecated) SSLCertificateChainFile naming a wrong file that uses R10 (as if it was hardcoded and not using the one given to you). That's my best guess as to how it got this way.

Do you have access to the Apache config?

5 Likes
8

when I connected to it with raw IP address I meet a landing page of a ferozo webpanel (https://ferozo.com/) so not sure if client have a raw apache config option

3 Likes
9

ok, I'll dig the apache config and post it here.

I can't recall if when I had issues before the second certificate in the chain was from R10 or was just missing, I think the latter.
If that was the case then maybe what is happening is that the automated process fails to properly deal with the second cert and registers either an old one or leaves the slot empty.