Certificate verify failed during renewal

Thanks in advance for any assistance or advice

My domain is: map.moodassessment.com.au

I ran this command:certbot renew --pre-hook "systemctl stop httpd" --post-hook "systemctl start httpd"

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/map.moodassessment.com.au.conf


Cert is due for renewal, auto-renewing...

Plugins selected: Authenticator apache, Installer apache

Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org

Attempting to renew cert (map.moodassessment.com.au) from /etc/letsencrypt/renewal/map.moodassessment.com.au.conf produced an unexpected error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579). Skipping.


Processing /etc/letsencrypt/renewal/test-map.moodassessment.com.au.conf


Cert is due for renewal, auto-renewing...

Plugins selected: Authenticator apache, Installer apache

Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org

Attempting to renew cert (test-map.moodassessment.com.au) from /etc/letsencrypt/renewal/test-map.moodassessment.com.au.conf produced an unexpected error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579). Skipping.

All renewal attempts failed. The following certs could not be renewed:

/etc/letsencrypt/live/map.moodassessment.com.au/fullchain.pem (failure)

/etc/letsencrypt/live/test-map.moodassessment.com.au/fullchain.pem (failure)


All renewal attempts failed. The following certs could not be renewed:

/etc/letsencrypt/live/map.moodassessment.com.au/fullchain.pem (failure)

/etc/letsencrypt/live/test-map.moodassessment.com.au/fullchain.pem (failure)


2 renew failure(s), 0 parse failure(s)

My web server is (include version): Apache/2.4.6

The operating system my web server runs on is (include version): RHELS 7.3

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):certbot 0.29.1

1 Like

Testing a connection to https://acme-v02.api.letsencrypt.org/directory - it appears to not recognize the cert at https://acme-v02.api.letsencrypt.org

[root@map httpd]# curl https://acme-v02.api.letsencrypt.org/directory
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

Checking via openssl

[root@map httpd]# openssl s_client -connect acme-v02.api.letsencrypt.org:443
CONNECTED(00000003)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
0 s:/CN=acme-v01.api.letsencrypt.org
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGozCCBYugAwIBAgISAzeib02BoUeWjPfdMYGlC63bMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEwMTcyMDIyMTZaFw0yMjAxMTUyMDIyMTVaMCcxJTAjBgNVBAMT
HGFjbWUtdjAxLmFwaS5sZXRzZW5jcnlwdC5vcmcwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDH1QCccnLrFbct3ZLuo1wfdyOKGD7W4dTJzvUEnSfClO2J
uCM4e/qAB2XVz86QrwDgwXJJkRbqK+H1yj1vQtHezO9KayY3Ew1ezhVHFBbptmiY
idM7J6DdqNJrJwksNtn4rAXlr47LaG7vMmAw8a2QYkx47kL44UHbFVpe0UCS6h4B
lzJwaftO4LCEHo8o4YyDQNDAyw/pIDB1j8mMYDNj5NKeplYDbZIqbzQ7aquLYH+8
kRAX856BX2UzcTVTEKRzgi/I5tA3aqiM7XqvqJ8I6TnGutlJ5AdrnJkIqPoGUYKY
LENUR/PVy26m1VltvFcIp9ExQ0OOKPlWb+DxOIGBAgMBAAGjggO8MIIDuDAOBgNV
HQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1Ud
EwEB/wQCMAAwHQYDVR0OBBYEFLj6iZ7Rv1yycd4P5AEIwFOJs8WqMB8GA1UdIwQY
MBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEF
BQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8v
cjMuaS5sZW5jci5vcmcvMIIBiQYDVR0RBIIBgDCCAXyCHmFjbWUtdjAxLTEuYXBp
LmxldHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDEtMi5hcGkubGV0c2VuY3J5cHQub3Jn
gh5hY21lLXYwMS0zLmFwaS5sZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAxLTQuYXBp
LmxldHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDEtNS5hcGkubGV0c2VuY3J5cHQub3Jn
ghxhY21lLXYwMS5hcGkubGV0c2VuY3J5cHQub3Jngh5hY21lLXYwMi0xLmFwaS5s
ZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAyLTIuYXBpLmxldHNlbmNyeXB0Lm9yZ4Ie
YWNtZS12MDItMy5hcGkubGV0c2VuY3J5cHQub3Jngh5hY21lLXYwMi00LmFwaS5s
ZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAyLTUuYXBpLmxldHNlbmNyeXB0Lm9yZ4Ic
YWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZzBMBgNVHSAERTBDMAgGBmeBDAEC
ATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNl
bmNyeXB0Lm9yZzCCAQUGCisGAQQB1nkCBAIEgfYEgfMA8QB2ACl5vvCeOTkh8FZz
n2Old+W+V32cYAr4+U1dJlwlXceEAAABfJAiQaMAAAQDAEcwRQIhAIXY+LEo5ikc
Ku9bEPcME4HkEMsNin8U3ZF8eL1il5L5AiBAzx+UynE0EOMT0azKoBm5/UFfHwIO
ELZVoY8y+60PggB3AG9Tdqwx8DEZ2JkApFEV/3cVHBHZAsEAKQaNsgiaN9kTAAAB
fJAiQpoAAAQDAEgwRgIhAOw4apJUTir/jPE/IZWetTyHnrv2Wb6KeVtT15xnWU0R
AiEAym23iIItXxMScz2QGMkk0PW2TMwMB0wBNqVGRmA7IAIwDQYJKoZIhvcNAQEL
BQADggEBAFGIPlF6nWL0CN93HuSRptGSSZBF+/ZEUYJCXD584uyPBCHIOXUXxVri
+B5gE5GC4iM6F1R5STdJwvseMNrkAZWrnxhs73SLAukP8zqQ4eWcdUaPHzPKaw55
GOyFigXX52MS6V7siwrb9r6vLMFGdsqsE6lSIhcL6ecutIhRK/mPALvK5HieA1XF
21DAxm7zuzEJTy0zjwznB3VXw9Bo3LP6ysQ/eTb5Oe5mlqSzSy+0hrzYfgKClfds
XNzeqdOvpUX8dvWlgVoCIDoiSEQRz+YhD0V7/FbIXDzeMGYtiV8+8OMlmKNnZdtF
G5PtvjyOJ7DU6k784LmkQV3eV0Dvt+w=
-----END CERTIFICATE-----
subject=/CN=acme-v01.api.letsencrypt.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3519 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: B82BF0FBE343104060065E0270E4A5D4E9C4DA589837D487ED4D7E64E32149D6
Session-ID-ctx:
Master-Key: 694ECBFBB64597DFD9F9DFAA619405A723DE6DE31BB42F7F5C29FE1B5E32DFCE45EB91E6328098D00A94677501604E3F
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1635941310
Timeout : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)
---

@stuartp Welcome and good report.

Yes, starting Sept 30 the Lets Encrypt servers starting using a certificate chain that ends with ISRG Root X1. This change was in response to the prior DST Root CA X3 expiring.

There are many, many posts about this here.

You need to add ISRG Root X1 to your CA trust store. Right off-hand I do not know how to do this for your system. I just wanted to help you understand what has happened.

Here is some good background info on these chains:

Note the Lets Encrypt ACME API server uses the "short chain" while this website and the other LE sites use the "long chain"

5 Likes

That is quiet old version, full with bugs (including security), the recent is 7.9. You may want to update that:

yum upgrade

No risk, just patches, it stays on the same major release 7. The bonus is that involves update on the ca-certificates, so your original problem supposed to disappear automatically.

5 Likes

Thanks Mike

I have just run
yum --showduplicates list 'ca-certificates' which showed my installed version was out of date.

I then ran
yum update ca-certificates which installed the latest (2021) version and then subsequently re-ran the certbot renew command . This seemed to work for my domain.

Some browsers I've tested since see the ISRG Root X1 as the root cert, but others still show DST Root CA X3 - not sure why that is - possibly some caching going on

3 Likes

Thanks bruncsak. Yum update ca-certificates seemed to clear up the main issue, but I will proceed to do a full update

cheers
Stuart

4 Likes

Yes, likely caching. Generally, browsers see the chain you send but then build their own so they may show something different. They do this to adapt to poorly configured servers and other reasons. To confirm what chain you send, best to use openssl or sites like this:

4 Likes

Interesting link - thanks for that.

Noted that it flags on "Trusted" (? doesn't explain why) and "Chain Issues" despite having an intermediate (R3)

1 Like

That is just your leaf - your domain signed by (issued by) R3. The "long chain" default is described in that earlier "long / short chain" link I provided.

Starting with Apache 2.4.8, you send the full chain using this style of conf:

SSLCertificateFile      /etc/letsencrypt/live/EXAMPLE/fullchain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/EXAMPLE/privkey.pem

You are at 2.4.6 but I assume that will get updated when you update the rest. The SSLCertificateChainFile was deprecated in 2.4.8

4 Likes

Ok - thanks for that - great info

2 Likes

No, the apache stays in 2.4.6. RedHat only backports patches if required to the original version snapshot at the start of the major release. So the OP is supposed to use the SSLCertificateChainFile statement in the apache configuration file.

Here is the version upgrade history from one of the server I am maintaining (It is CentOS, but that is the same as RHEL):

2019 Oct 03 11:44:17 Installed: httpd-2.4.6-90.el7.centos.x86_64
2020 Apr 28 21:03:15 Updated: httpd-2.4.6-93.el7.centos.x86_64
2020 Nov 24 21:03:27 Updated: httpd-2.4.6-97.el7.centos.x86_64
2021 Oct 22 05:31:15 Updated: httpd-2.4.6-97.el7.centos.1.x86_64
5 Likes

Well, ok then, thanks @bruncsak :slight_smile: Try something like:

SSLCertificateFile      /etc/letsencrypt/live/(path)/cert.pem
SSLCertificateChainFile /etc/letsencrypt/live/(path)/chain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/(path)/privkey.pem

@stuartp You do not seem to be using the chain at all. Any questions post the SSLCertificate... lines from your conf.

5 Likes

@MikeMcQ - here is the section

 # best practice ssl settings from letsencrypt options
   include /etc/letsencrypt/options-ssl-apache.conf

   #Server Certificate:
   SSLCertificateFile /etc/letsencrypt/live/map.moodassessment.com.au/fullchain.pem

   #Server Private Key:
   SSLCertificateKeyFile /etc/letsencrypt/live/map.moodassessment.com.au/privkey.pem

   #Server Certificate Chain:
   # not required with letsencrypt fullchain above

@stuartp Ah, your version of Apache does not support using fullchain for the CertificateFile like in my post #9 (it would use just the leaf cert from it). Your version requires the chain to be in ChainFile like I just showed in post #12 with the leaf only (cert.pem) in CertificateFile.

Change to that, restart Apache and check with
https://decoder.link/sslchecker/

3 Likes

Ok - I have added the reference to SSLCertificateChainFile and all browsers tested are now pulling it correctly.

@MikeMcQ & @bruncsak
Thanks v much to both of you - especially for persisting after I thought it had been solved, and thought I was just experiencing a caching issue with some clients, when in fact there was still some configuration to be updated in order to address the issue completely.

cheers
Stuart

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.