[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: test.tanf.pro

I ran this command: certbot renew --dry-run

It produced this output: certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/test.tanf.pro.conf

Failed to renew certificate test.tanf.pro with error: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/test.tanf.pro/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): Apache 2.4.37 43.module_el8.5.0+2597+c4b14997.alma43.module_el8.5.0+2597+c4b14997.alma

The operating system my web server runs on is (include version): AlmaLinux release 8.5 (Arctic Sphynx)

My hosting provider, if applicable, is: Self-hosted

I can login to a root shell on my machine (yes or no, or I don't know): Yes I can.

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): I am not.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.22.0

I have gone through pretty much anything that I can find today. What is odd is that we have other server that run on Nginx that updates with no issues. I've verified that the DNS and dig return what I would expect. There are no errant entries in the /etc/hosts file. We looked on the firewall - we were not seeing any dropped packets to/from the LE
address. For giggles we opened http/https temporarily and still receive the same results. I've uninstalled/reinstalled certbot via dnf. Looked at the Python3 libraries and none have been updated. I want to say it is a provider certificate issue, but I just cannot prove it at this time. Tried adding the LE Acme certs to the trust store and that did not help at all. Any help is very appreciated!

A small list of what I have done:

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> acme-v02.api.letsencrypt.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4326
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;acme-v02.api.letsencrypt.org. IN A

;; ANSWER SECTION:
acme-v02.api.letsencrypt.org. 6781 IN CNAME prod.api.letsencrypt.org.
prod.api.letsencrypt.org. 231 IN CNAME ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com. 231 IN A 172.65.32.248

;; Query time: 32 msec
;; SERVER: 10.209.69.44#53(10.209.69.44)
;; WHEN: Thu Jan 20 21:12:18 UTC 2022
;; MSG SIZE rcvd: 155

  • curl -v https://acme-v02.api.letsencrypt.org/directory
  • Trying 172.65.32.248...
  • TCP_NODELAY set
  • Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, [no content] (0):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, [no content] (0):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, [no content] (0):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, [no content] (0):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, [no content] (0):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=acme-v01.api.letsencrypt.org
  • start date: Dec 18 04:18:20 2021 GMT
  • expire date: Mar 18 04:18:19 2022 GMT
  • subjectAltName: host "acme-v02.api.letsencrypt.org" matched cert's "acme-v02.api.letsencrypt.org"
  • issuer: C=US; O=Let's Encrypt; CN=R3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • TLSv1.3 (OUT), TLS app data, [no content] (0):
  • TLSv1.3 (OUT), TLS app data, [no content] (0):
  • TLSv1.3 (OUT), TLS app data, [no content] (0):
  • Using Stream ID: 1 (easy handle 0x557020bbc6b0)
  • TLSv1.3 (OUT), TLS app data, [no content] (0):

GET /directory HTTP/2
Host: acme-v02.api.letsencrypt.org
User-Agent: curl/7.61.1
Accept: /

Certificate chain
0 s:CN = acme-staging.api.letsencrypt.org
i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
MIIFbDCCBFSgAwIBAgISBMsfznqt2F+klAkx0pA8jj8gMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEyMTcwMzU2NTVaFw0yMjAzMTcwMzU2NTRaMCsxKTAnBgNVBAMT
IGFjbWUtc3RhZ2luZy5hcGkubGV0c2VuY3J5cHQub3JnMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEAyWnf6MyPSVvSU6v7uLaC3/spVg+lAizVWRMllSjv
NOm/LDjUEoMxN9Fii/2JdXZi0EWxaF3eHgIOq04V/zeG7tKr7UZG0H4MvLh1GdIq
ffPNsZQDCDwVoT2DHfb2WpwwBe01cH2xUeE6YwZeRGkXNBEYgK9dNciWX7Tir4jm
bJ8YuIhH04Pbbqqbnql/igbfCGFfRrks55Sb+Q87gDUh45yso6tGp2sqZeotdk/s
F+N2GoKOTF46VBgvFu4KgwS6vLUHl+MFbdIeDBwVXWt3TFb9YhSAmzYpEvgqT9CM
CcJscS8GuzBiTEfEjBD0s8JsdQ9D8GboyihZn6k3d4Xu/QIDAQABo4ICgTCCAn0w
DgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAM
BgNVHRMBAf8EAjAAMB0GA1UdDgQWBBR2cS9CSwQwZrpFwFWd2vyeEs6nnTAfBgNV
HSMEGDAWgBQULrMXt1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYI
KwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0
cDovL3IzLmkubGVuY3Iub3JnLzBRBgNVHREESjBIgiRhY21lLXN0YWdpbmctdjAy
LmFwaS5sZXRzZW5jcnlwdC5vcmeCIGFjbWUtc3RhZ2luZy5hcGkubGV0c2VuY3J5
cHQub3JnMEwGA1UdIARFMEMwCAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYI
KwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0c2VuY3J5cHQub3JnMIIBBAYKKwYBBAHW
eQIEAgSB9QSB8gDwAHYARqVV63X6kSAwtaKJafTzfREsQXS+/Um4havy/HD+bUcA
AAF9xsAPzwAABAMARzBFAiAeljg7pnESTVM3IjVhDHScJskI7s2C1pbejNVo4E3r
FgIhANn9CMj3NjRCZ38ZndOspy9gcRkWvsggyk3VzBVtMm5vAHYAb1N2rDHwMRnY
mQCkURX/dxUcEdkCwQApBo2yCJo32RMAAAF9xsAQ/QAABAMARzBFAiBnWzYKHTw0
iPviseW2We9go9GKSdmpJC+320q3BkQ7pQIhALj8RlDQH0THOn6gwMgsSlQPGdna
eOsxoM74yHMuJCkoMA0GCSqGSIb3DQEBCwUAA4IBAQA1x7D5wRlDkqXbMx53vZ2+
h6/odXIhjDykKl9PlTrf8tNDr3QhGxlRjvle8E9qYF1zx9QUib4SDo0ql6FJRW+X
YOY0GzezRZMK8L+DhXPy2HrtK9/UN/pwo1bgf2Mkg7hG/DG9SZPfwtOII+hYVsrY
xoCPrH5Jt7CI4ge506CeA3mrEZ5XdNmekgdocf2AmHHFvlPrAlMapo0PLnZXTI1w
To1oxYTVEWDzIu/cAnNWWbBWySKEL1/mGH4WfNM4CmzwIpH0xmvfTjhueSdUs+sK
TWF5MHIp+RKlCdlDjlSv91RILXHBF22eGJOUd7B5gTyjYtBYKIqA+9SBJ3NqINr2
-----END CERTIFICATE-----
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

Server certificate
subject=CN = acme-staging.api.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = R3

No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits

SSL handshake has read 3263 bytes and written 414 bytes
Verification error: certificate has expired

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 10 (certificate has expired)

DONE

  • dnf install ca-certificates - Package ca-certificates-2021.2.50-80.0.el8_4.noarch is already installed.
1 Like
2

In your tests/examples, you are switching between production and staging.

2 Likes
3

Apologies. Too many commands run trying to get this resolved...

openssl s_client -showcerts -verify 5 -connect acme-v02.api.letsencrypt.org:443 < /dev/null
verify depth is 5
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
notAfter=Sep 30 18:14:03 2024 GMT
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
notAfter=Sep 15 16:00:00 2025 GMT
verify return:1
depth=0 CN = acme-v01.api.letsencrypt.org
notAfter=Mar 17 03:44:27 2022 GMT
verify return:1
---
Certificate chain
 0 s:CN = acme-v01.api.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
MIIGojCCBYqgAwIBAgISBLMnruYqIvO73OA+zqzhMzAZMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEyMTcwMzQ0MjhaFw0yMjAzMTcwMzQ0MjdaMCcxJTAjBgNVBAMT
HGFjbWUtdjAxLmFwaS5sZXRzZW5jcnlwdC5vcmcwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDeZKZE8bu58CBC+moWFuIEQjonTUccqRqSbcjz7tuiD9a5
evj/49sOxOOKW6DmtFGW9VOFBWP733g2h4qZrVKCyWLE7xA+0L0b2RagX+8kx8Da
AxVhc/b2l0vpozDMniCvpm5HNQLXnWtZkAckqtpQg3DHs7NGH9LYuUqxtz7ghGRs
0HX/16MdfNmD8FkSocjd2i+e/iDz9sFe/3VU46/DgQCBFVOOMGnMIXgZ9Rx1sWx+
ytT4jNLU1JEZzyzA7M13C41RxDnXLAmMqlZbRQObn/gfYMb60RDFd2lDikIkl7p8
BRhk/BmVXYzBurrTMFB+DYuTShegqp8sxpws8PlhAgMBAAGjggO7MIIDtzAOBgNV
HQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1Ud
EwEB/wQCMAAwHQYDVR0OBBYEFNVrgxI9R0VntpglWIlwqq/qw6PnMB8GA1UdIwQY
MBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEF
BQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8v
cjMuaS5sZW5jci5vcmcvMIIBiQYDVR0RBIIBgDCCAXyCHmFjbWUtdjAxLTEuYXBp
LmxldHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDEtMi5hcGkubGV0c2VuY3J5cHQub3Jn
gh5hY21lLXYwMS0zLmFwaS5sZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAxLTQuYXBp
LmxldHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDEtNS5hcGkubGV0c2VuY3J5cHQub3Jn
ghxhY21lLXYwMS5hcGkubGV0c2VuY3J5cHQub3Jngh5hY21lLXYwMi0xLmFwaS5s
ZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAyLTIuYXBpLmxldHNlbmNyeXB0Lm9yZ4Ie
YWNtZS12MDItMy5hcGkubGV0c2VuY3J5cHQub3Jngh5hY21lLXYwMi00LmFwaS5s
ZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAyLTUuYXBpLmxldHNlbmNyeXB0Lm9yZ4Ic
YWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZzBMBgNVHSAERTBDMAgGBmeBDAEC
ATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNl
bmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2AEHIyrHfIkZKEMah
OglCh15OMYsbA+vrS8do8JBilgb2AAABfca0qu8AAAQDAEcwRQIhAIXGiF+/yGHK
SBfMsuC0lgjR+4+OUlIxR+h4VmrTED4ZAiAyDQY5SVmkTuudYvCrNxfIy9Ka7DY5
Az1hdW9DnQSlOAB2AEalVet1+pEgMLWiiWn0830RLEF0vv1JuIWr8vxw/m1HAAAB
fca0qwwAAAQDAEcwRQIgdqptBOaSGPIJbqhmbYwhB8DgSrDAjMvJsIaQFk2cVIYC
IQCDNh9ryodZnwpOt61S6nBsOl8Q4XJ2lhLYyVgmUVJXYDANBgkqhkiG9w0BAQsF
AAOCAQEAk7xyFXdU/izL1FzQwSRg2SiusUgXlqAFIBX9vgpDIpuCDgMAY6CzN5li
PUOVi85WjZBUStBIOdHjB70FpSUxKduz8zM5abpiEND9Zas2EGw4ETlBIH3XNB8w
06nkmQbrEfT9GEaOenZzjxyDOdFf7ZDX/tMe/Mwrkw9JoQIPxBDCfwTw20N216tM
iecZmfbTNegElf/5UtKgBpzLhoPqLIHdYTaF4FR+nF8P3v3Ur6t3HFiUmPYid4f7
Pt3e4nRNJvkzHTAONz6N+Q6IAG5aYqgQpBsu00K7nyFnnyzLZyZTB0IgEwO2aZcW
jDjxp9fZVwiRTiyI/gcjPiDY5wLCyQ==
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

Server certificate
subject=CN = acme-v01.api.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = R3

No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits

SSL handshake has read 3573 bytes and written 406 bytes
Verification error: certificate has expired

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 10 (certificate has expired)

DONE

1 Like
4

Very odd. The chain showed by your openssl is not the one sent by the Let's Encrypt ACME server - production or staging. It should look like the one below but yours has a copy of the expired DST Root CA X3. Do you have a proxy or firewall that is intercepting your requests?

echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = acme-v01.api.letsencrypt.org
verify return:1

If you do not believe me :slight_smile: you can see the chain using a website such as this

Tip: place three backticks before and after output so it formats nice. Like:
```
output
```

3 Likes
5

It is very odd. We do have Forcepoint in between everything. But, we have another server in the same DC as this one, that uses LE and updates with no issues; the only difference is it runs Nginx, is CentOS 7, and has been in place for several years. The output looks similar to yours. To makes things even weirder, this worked at some point in the past 3 months as we were able to get the initial certs. I guess the question is: How do I get the correct chain?

From our website app-test.thementornetwork.com, that works properly:

echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = acme-v01.api.letsencrypt.org
verify return:1
DONE
CONNECTED(00000003)
---
Certificate chain
 0 s:/CN=acme-v01.api.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
1 Like
6

I am assuming that just copying the chain.pem and fullchain.pem from the known-good machine is not really an option?

1 Like
7

Well, you could copy them if it covered the same domain name. But all I see with that name is here and expired several days ago.

And, I think you mean privkey.pem and fullchain.pem - you would not need chain.pem with that version of Apache

I don't have an explanation for why you see the obsolete chain. LE has not sent that format since Sept30 2021.

What are the contents of this file? Show output of:

cat /etc/letsencrypt/renewal/test.tanf.pro.conf

Also, what does your root cert store look like on this failing server? Show this:

grep -Ei 'ISRG|DST|R3' /etc/pki/tls/certs/ca-bundle.crt | grep '#'

I don't think the above items are causing the wrong chain to be seen - just curious to look for odd items. I think Forcepoint is most likely culprit but I do not know it to ask pointed questions about it.

3 Likes
8

Here is my test.tanf.pro.conf contents:

 cat /etc/letsencrypt/renewal/test.tanf.pro.conf
# renew_before_expiry = 30 days
version = 1.19.0
archive_dir = /etc/letsencrypt/archive/test.tanf.pro
cert = /etc/letsencrypt/live/test.tanf.pro/cert.pem
privkey = /etc/letsencrypt/live/test.tanf.pro/privkey.pem
chain = /etc/letsencrypt/live/test.tanf.pro/chain.pem
fullchain = /etc/letsencrypt/live/test.tanf.pro/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = REDACTED
authenticator = apache
installer = apache

And results from the cert bundle:

grep -Ei 'ISRG|DST|R3' /etc/pki/tls/certs/ca-bundle.crt | grep '#'
# R3
# ISRG Root X1
# DST Root CA X3
# GTS Root R3
# GlobalSign Root CA - R3
# ISRG Root X1

I would think that FP would inject the same cert to the other server/websites that are working.

2 Likes
9

You know better than me how FP is configured.

Your cert store is, well, a mess. Not sure how to fix that with Alma off-hand but perhaps you do since you mentioned changing it.

That command output should look like this:

# GTS Root R3
# GlobalSign Root CA - R3
# ISRG Root X1

There should not be two ISRG Root X1 and there should not be an R3. The DST Root CA X3 is also not necessary but should not be harmful.

The extra R3 could be causing the problem by itself but I would remove all 3 if you can.

4 Likes
10

Got it working @MikeMcQ! I ended up grabbing the copy of the ca-bundle.crt from the know good server and copying it over to the affected server (well /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem after making a copy of it). Saw the changes in ca-bundle.crt:

 grep -Ei 'ISRG|DST|R3' /etc/pki/tls/certs/ca-bundle.crt | grep '#'
# DST Root CA X3
# GTS Root R3
# GlobalSign Root CA - R3
# ISRG Root X1

The test ran successfully:

certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/test.tanf.pro.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for test.tanf.pro

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all simulated renewals succeeded:
  /etc/letsencrypt/live/test.tanf.pro/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

And the actual update was successful. Ran dnf update and it did not update anything, so I think I am set. Thanks for all of the assistance!

2 Likes
11

Excellent! Glad you saw that ca-bundle.crt is a link to the other. I just used that name as that was what showed in the curl examples you had earlier. Different distros use diff names so easiest to use names I see :slight_smile: Cheers!

3 Likes
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.