I can never renew my certificate

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: reckeng.com

I ran this command: # certbot renew

This is the output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/reckeng.com.conf

Failed to renew certificate reckeng.com with error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1006)')))

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/reckeng.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

I ran this command: # certbot certonly --standalone --preferred-challenges http -d recking.com

This is the output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
An unexpected error occurred:
requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1006)')))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): I do not use one.

The operating system my web server runs on is (include version): Arch Linux (up to date)

My hosting provider, if applicable, is: Linode

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.6.0

My only way around all this is to wipe the whole server and start again. :frowning:

2

Are you able to connect to other domains from that same system? What do these show?

curl -I https://google.com
curl -I https://cloudflare.com
4 Likes
3

$ curl -I https://google.com

HTTP/2 204
date: Mon, 15 Jan 2024 22:32:52 GMT
content-type: text/html
server: HTTP server (unknown)
x-xss-protection: 0
set-cookie: CONSENT=PENDING+451; expires=Wed, 14-Jan-2026 22:32:52 GMT; path=/; domain=.google.com; Secure
p3p: CP="This is not a P3P policy! See P3P and Google's cookies - Google Account Help for more info."
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

$ curl -I https://cloudflare.com

HTTP/2 301
date: Mon, 15 Jan 2024 22:33:24 GMT
location: https://www.cloudflare.com/
cache-control: max-age=3600
expires: Mon, 15 Jan 2024 23:33:24 GMT
set-cookie: __cf_bm=JzoRqKC073OY5I4p2jLXxcMJtQNArHUDoTa7lbWticE-1705358004-1-AbkNpUZZcFXAcHCTyBsBEzPYZvGCDmJdxe/kknN+X7GsAIeaN4UBZ91v2BWf61YR98MpP/bWr7pEa+U1THxlB5w=; path=/; expires=Mon, 15-Jan-24 23:03:24 GMT; domain=.cloudflare.com; HttpOnly; Secure; SameSite=None
report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=UL9n8GgO7AzlDpFF62LsikoHoE49pplxb0qee9qT8FnJm6NcjsIdMCuXmyZZ12cDSXQVlww19o8cXaQf0eG97zLaz%2B%2BX3KkTr9hybzmok%2FwxRdOtj9xO0I%2FAyKrYewvYVwUXPFj8DAmtdsb2"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
strict-transport-security: max-age=15780000; includeSubDomains
server: cloudflare
cf-ray: 84619486389ad184-LHR
alt-svc: h3=":443"; ma=86400

4

That's an odd response from google. Might not be important.

Can you retry the google request and also show these two new commands

curl -I https://acme-v02.api.letsencrypt.org/directory
echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head

curl -I https://google.com

Google should be responding with an http 301 and a different "Server:" response header

5 Likes
5

$ curl -I https://acme-v02.api.letsencrypt.org/directory

HTTP/2 200 
server: nginx
date: Tue, 16 Jan 2024 08:21:27 GMT
content-type: application/json
content-length: 752
cache-control: public, max-age=0, no-cache
replay-nonce: wFJtXv4dpA5sHrc67b73x2TqZXdLSfkDBDsxGpy8cb4Pf9mT-rA
x-frame-options: DENY
strict-transport-security: max-age=604800

$ echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head

depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify error:num=2:unable to get issuer certificate
issuer= O=Digital Signature Trust Co., CN=DST Root CA X3
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=R3
issuer= C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=0 CN=acme-v02.api.letsencrypt.org
issuer= C=US, O=Let's Encrypt, CN=R3
verify return:1
Connecting to 2606:4700:60:0:f53d:5624:85c7:3a2c
CONNECTED(00000003)
---
Certificate chain
 0 s:CN=acme-v02.api.letsencrypt.org
   i:C=US, O=Let's Encrypt, CN=R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec 28 04:53:39 2023 GMT; NotAfter: Mar 27 04:53:38 2024 GMT
 1 s:C=US, O=Let's Encrypt, CN=R3
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
DONE

$ curl -I https://google.com

HTTP/2 204 
date: Tue, 16 Jan 2024 08:23:14 GMT
content-type: text/html
server: HTTP server (unknown)
x-xss-protection: 0
set-cookie: CONSENT=PENDING+263; expires=Thu, 15-Jan-2026 08:23:14 GMT; path=/; domain=.google.com; Secure
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
6

That openssl result looks wrong. I think the google is not important (maybe related to authenticating you as a valid searcher).

Can you show the entire output of this

openssl s_client -connect acme-v02.api.letsencrypt.org:443 -showcerts < /dev/null
3 Likes
7

I guess the certificate " C=US, O=Internet Security Research Group, CN=ISRG Root X1" signed via " O=Digital Signature Trust Co., CN=DST Root CA X3" is incorrectly put in the trust anchor store.

2 Likes
8

$ openssl s_client -connect acme-v02.api.letsencrypt.org:443 -showcerts < /dev/null

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = acme-v02.api.letsencrypt.org
verify return:1
---
Certificate chain
 0 s:CN = acme-v02.api.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec 27 19:27:36 2023 GMT; NotAfter: Mar 26 19:27:35 2024 GMT
-----BEGIN CERTIFICATE-----
MIIFwjCCBKqgAwIBAgISBIlJd5yCVQnxihNgWJMCtC/lMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMzEyMjcxOTI3MzZaFw0yNDAzMjYxOTI3MzVaMCcxJTAjBgNVBAMT
HGFjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQCvWRt2GnxATtxWPasb1OU16NT+tl6F9srr27ZF/STMKgty
HG+TeyRkL+MYay2nOXr1T5uG0BPbwPoUbsshPlmLEJRbYdztVqJu9sPu+QuFNEps
EJkV6pzkNd5n5QrF0nsYR04rRUCobLyLww+ftSI/9elBJx9adi+oheRmRWXTtAIE
6lZuupbi8bpRXhHROTVPtVAu4UuG/1T0a5nN5HbDTwEVPFPL8qZolT9NgoAZODli
1uQg01tua75lP8SIqAXrXcXWIKxGRZEv97+SA1qbf03Lqsnu4hOP+//KM+W4pkl6
bot+8XllTISq/8IuKDFJP13kV4wazGMf2r8eHOYrAgMBAAGjggLbMIIC1zAOBgNV
HQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1Ud
EwEB/wQCMAAwHQYDVR0OBBYEFFH/kv5RVwcBGm+6l1/+9B/N67ryMB8GA1UdIwQY
MBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEF
BQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8v
cjMuaS5sZW5jci5vcmcvMIHjBgNVHREEgdswgdiCHmFjbWUtdjAyLTEuYXBpLmxl
dHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItMi5hcGkubGV0c2VuY3J5cHQub3Jngh5h
Y21lLXYwMi0zLmFwaS5sZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAyLTQuYXBpLmxl
dHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItNS5hcGkubGV0c2VuY3J5cHQub3Jnghxh
Y21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnghhpbmNpZGVudC5sZXRzZW5jcnlw
dC5vcmcwEwYDVR0gBAwwCjAIBgZngQwBAgEwggEEBgorBgEEAdZ5AgQCBIH1BIHy
APAAdgBIsONr2qZHNA/lagL6nTDrHFIBy1bdLIHZu7+rOdiEcwAAAYys9hHTAAAE
AwBHMEUCIGQ6hmd8/TKQHKSqaaJp89b4+L5oOFMG3ORxbzJ0ki5YAiEA2ImScdgS
L0D00WeOyHLMJDqsP1fjY3RLmytefJmjCyoAdgAp0DobtnSqcRzTA1tlV8FPiqeL
T+g4lEnspFP5RL0kaAAAAYys9hKRAAAEAwBHMEUCIFh1zN5bCC8tNO+u0g4WS0kt
tPc02anVFWadNrCG9d5JAiEA4WXpN44okgoIszeEheApJC1AM0jI1iLRmaKtRffl
q+YwDQYJKoZIhvcNAQELBQADggEBAEBGk9/+x01rZ/WpiF+rQ88JWjFVpk+AukP/
xiQAayMoEcbijHprs/jeP6B3vldd47BO63Hu/AKUKi3XB247Ho2Lgsv655FcpiF6
zUO5j29T1CfMDUsOLOJ8ua5rIhfd6JMsP9UujvglUv03o8YQcrkK6k6NlT+MIbYT
H4S8tEpE8RcmPVnLEKXhbcZy1aD6AqrqW8lchzbHI8PeOROQXtP7rfnaw8kCeYUH
BeVdlC6rZMYCFeskMIN3v0Q5zpWxY3ydQQMl/ZJ5/PIXN3HNfT3b9+9sHqXjNIkp
QWtaSFKWxf9cE6ciOvQHyX+nCvnDcGa7Xqu6NS/hcuhF4lpYzwE=
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = acme-v02.api.letsencrypt.org
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3349 bytes and written 410 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE
9

Agree. A little odd that curl worked.

In any case, Certbot and openssl likely share same CA Root Store.
Can you show output of this

openssl version -d
4 Likes
10

openssl version -d

OPENSSLDIR: "/etc/ssl"

11

Now show

ls -l /etc/ssl/certs

If it is a long list just show the first 10 or so

3 Likes
12

$ ls -l /etc/ssl/certs

total 1408
lrwxrwxrwx 1 root root   61 Jan 15 22:24 002c0b4f.0 -> ../../ca-certificates/extracted/cadir/GlobalSign_Root_R46.pem
lrwxrwxrwx 1 root root   87 Jan 15 22:24 01419da9.0 -> ../../ca-certificates/extracted/cadir/Microsoft_ECC_Root_Certificate_Authority_2017.pem
lrwxrwxrwx 1 root root   62 Jan 15 22:24 0179095f.0 -> ../../ca-certificates/extracted/cadir/BJCA_Global_Root_CA1.pem
lrwxrwxrwx 1 root root   83 Jan 15 22:24 02265526.0 -> ../../ca-certificates/extracted/cadir/Entrust_Root_Certification_Authority_-_G2.pem
lrwxrwxrwx 1 root root   79 Jan 15 22:24 04f60c28.0 -> ../../ca-certificates/extracted/cadir/USERTrust_ECC_Certification_Authority.pem
lrwxrwxrwx 1 root root   65 Jan 15 22:24 062cdee6.0 -> ../../ca-certificates/extracted/cadir/GlobalSign_Root_CA_-_R3.pem
lrwxrwxrwx 1 root root   63 Jan 15 22:24 064e0aa9.0 -> ../../ca-certificates/extracted/cadir/QuoVadis_Root_CA_2_G3.pem
lrwxrwxrwx 1 root root   88 Jan 15 22:24 06dc52d5.0 -> ../../ca-certificates/extracted/cadir/SSL.com_EV_Root_Certification_Authority_RSA_R2.pem
lrwxrwxrwx 1 root root   69 Jan 15 22:24 073bfcc5.0 -> ../../ca-certificates/extracted/cadir/TrustAsia_Global_Root_CA_G4.pem
lrwxrwxrwx 1 root root   72 Jan 15 22:24 08063a00.0 -> ../../ca-certificates/extracted/cadir/Security_Communication_RootCA3.pem
lrwxrwxrwx 1 root root   92 Jan 15 22:24 09789157.0 -> ../../ca-certificates/extracted/cadir/Starfield_Services_Root_Certificate_Authority_-_G2.pem
lrwxrwxrwx 1 root root   53 Jan 15 22:24 0a775a30.0 -> ../../ca-certificates/extracted/cadir/GTS_Root_R3.pem
lrwxrwxrwx 1 root root   54 Jan 15 22:24 0b1b94ef.0 -> ../../ca-certificates/extracted/cadir/CFCA_EV_ROOT.pem
lrwxrwxrwx 1 root root   54 Jan 15 22:24 0b9bc432.0 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X2.pem
lrwxrwxrwx 1 root root   82 Jan 15 22:24 0bf05006.0 -> ../../ca-certificates/extracted/cadir/SSL.com_Root_Certification_Authority_ECC.pem
lrwxrwxrwx 1 root root   69 Jan 15 22:24 0d69c7e1.0 -> ../../ca-certificates/extracted/cadir/GlobalSign_ECC_Root_CA_-_R4.pem
lrwxrwxrwx 1 root root   70 Jan 15 22:24 0f5dc4f3.0 -> ../../ca-certificates/extracted/cadir/UCA_Extended_Validation_Root.pem
lrwxrwxrwx 1 root root   64 Jan 15 22:24 0f6fa695.0 -> ../../ca-certificates/extracted/cadir/GDCA_TrustAUTH_R5_ROOT.pem
lrwxrwxrwx 1 root root   53 Jan 15 22:24 1001acf7.0 -> ../../ca-certificates/extracted/cadir/GTS_Root_R1.pem
lrwxrwxrwx 1 root root   92 Jan 15 22:24 10531352.0 -> ../../ca-certificates/extracted/cadir/Starfield_Services_Root_Certificate_Authority_-_G2.pem
lrwxrwxrwx 1 root root   84 Jan 15 22:24 106f3e4d.0 -> ../../ca-certificates/extracted/cadir/Entrust_Root_Certification_Authority_-_EC1.pem
lrwxrwxrwx 1 root root   79 Jan 15 22:24 128f4b91.0 -> ../../ca-certificates/extracted/cadir/Atos_TrustedRoot_Root_CA_RSA_TLS_2021.pem
lrwxrwxrwx 1 root root   65 Jan 15 22:24 14bc7599.0 -> ../../ca-certificates/extracted/cadir/emSign_ECC_Root_CA_-_G3.pem
13

Alright, now this (sorry so many steps ... diff distros use different methods of storing these)

ls -l /etc/ssl/certs|grep -Ei 'ISRG|DST'
5 Likes
14

$ ls -l /etc/ssl/certs|grep -Ei 'ISRG|DST'1~

lrwxrwxrwx 1 root root   54 Jan 15 22:24 0b9bc432.0 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X2.pem
lrwxrwxrwx 1 root root   54 Jan 15 22:24 4042bcee.0 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.pem
lrwxrwxrwx 1 root root   56 Jan 15 22:24 4042bcee.1 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.1.pem
lrwxrwxrwx 1 root root   54 Jan 15 22:24 6187b673.0 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.pem
lrwxrwxrwx 1 root root   56 Jan 15 22:24 6187b673.1 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.1.pem
lrwxrwxrwx 1 root root   54 Jan 15 22:24 8794b4e3.0 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X2.pem
lrwxrwxrwx 1 root root   56 Jan 15 22:24 ISRG_Root_X1.1.pem -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.1.pem
lrwxrwxrwx 1 root root   54 Jan 15 22:24 ISRG_Root_X1.pem -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.pem
lrwxrwxrwx 1 root root   54 Jan 15 22:24 ISRG_Root_X2.pem -> ../../ca-certificates/extracted/cadir/ISRG_Root_X2.pem
15

The ISRG_Root_X1.1.pem is unusual. I don't know why you would have a second copy there. My guess is that is what is causing the SSL Verify problem. Have you made any manual changes to your root store?

Before we look into it further and before deleting it (using the correct tools) can you just use the standard Arch command to update your CA Root store?

There may be other steps using pacman to update the ca-certificates package but I don't know Arch well enough to describe those commands.

This may help anyway. Try

sudo update-ca-trust

Then, show the result of this again

ls -l /etc/ssl/certs|grep -Ei 'ISRG|DST'
4 Likes
16

I haven't tried to edit the file manually.

Have tried both:

$ sudo update-ca-trust
$ sudo trust extract-compat

Unfortunately, the result is the same:

$ sudo certbot -v renew

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/reckeng.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer None
Failed to renew certificate reckeng.com with error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1006)')))

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/reckeng.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

$ ls -l /etc/ssl/certs|grep -Ei 'ISRG|DST'

lrwxrwxrwx 1 root root   54 Jan 17 16:37 0b9bc432.0 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X2.pem
lrwxrwxrwx 1 root root   54 Jan 17 16:37 4042bcee.0 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.pem
lrwxrwxrwx 1 root root   56 Jan 17 16:37 4042bcee.1 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.1.pem
lrwxrwxrwx 1 root root   54 Jan 17 16:37 6187b673.0 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.pem
lrwxrwxrwx 1 root root   56 Jan 17 16:37 6187b673.1 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.1.pem
lrwxrwxrwx 1 root root   54 Jan 17 16:37 8794b4e3.0 -> ../../ca-certificates/extracted/cadir/ISRG_Root_X2.pem
lrwxrwxrwx 1 root root   56 Jan 17 16:37 ISRG_Root_X1.1.pem -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.1.pem
lrwxrwxrwx 1 root root   54 Jan 17 16:37 ISRG_Root_X1.pem -> ../../ca-certificates/extracted/cadir/ISRG_Root_X1.pem
lrwxrwxrwx 1 root root   54 Jan 17 16:37 ISRG_Root_X2.pem -> ../../ca-certificates/extracted/cadir/ISRG_Root_X2.pem
17

Can you post the contents of these 3 files. We can decode them to see what they are. These are not secret - they are on any Arch system and they come from Mozilla anyway as I understand Arch (which is not well).

5 Likes
18

We can keep looking at this as noted in my prior post.

But, what changed on your system since your last renewal on Oct7 (now expired)? Were you doing those renewals on this same Arch setup? Something must have changed to cause this SSL Verify error. Any major o/s changes?

You show Certbot as version 2.6. How did you install that? If snap method you should be at 2.8. Are you using pip install? Have you tried refreshing that?

While Certbot and openssl failed your curl command worked fine so this points to an inconsistent CA root store. Hard to know what other issues this may cause you but it should be resolved.

4 Likes
19

Side note: IPv4 and IPv6 do not respond the same; IPv4 is closed IPv6 is filtered

>nmap -4 -Pn -p80,443 reckeng.com
Starting Nmap 7.94 ( https://nmap.org ) at 2024-01-17 20:40 UTC
Nmap scan report for reckeng.com (176.58.119.51)
Host is up (0.13s latency).
Other addresses for reckeng.com (not scanned): 2a01:7e00::f03c:91ff:fe03:45ed
rDNS record for 176.58.119.51: 176-58-119-51.ip.linodeusercontent.com

PORT    STATE  SERVICE
80/tcp  closed http
443/tcp closed https

Nmap done: 1 IP address (1 host up) scanned in 0.58 seconds

>nmap -4 -Pn -p80,443 www.reckeng.com
Starting Nmap 7.94 ( https://nmap.org ) at 2024-01-17 20:40 UTC
Nmap scan report for www.reckeng.com (176.58.119.51)
Host is up (0.13s latency).
Other addresses for www.reckeng.com (not scanned): 2a01:7e00::f03c:91ff:fe03:45ed
rDNS record for 176.58.119.51: 176-58-119-51.ip.linodeusercontent.com

PORT    STATE  SERVICE
80/tcp  closed http
443/tcp closed https

Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds

>nmap -6 -Pn -p80,443 reckeng.com
Starting Nmap 7.94 ( https://nmap.org ) at 2024-01-17 20:40 UTC
Nmap scan report for reckeng.com (2a01:7e00::f03c:91ff:fe03:45ed)
Host is up.
Other addresses for reckeng.com (not scanned): 176.58.119.51
rDNS record for 2a01:7e00::f03c:91ff:fe03:45ed: imfatant.com

PORT    STATE    SERVICE
80/tcp  filtered http
443/tcp filtered https

Nmap done: 1 IP address (1 host up) scanned in 3.30 seconds

>nmap -6 -Pn -p80,443 www.reckeng.com
Starting Nmap 7.94 ( https://nmap.org ) at 2024-01-17 20:40 UTC
Nmap scan report for www.reckeng.com (2a01:7e00::f03c:91ff:fe03:45ed)
Host is up.
Other addresses for www.reckeng.com (not scanned): 176.58.119.51
rDNS record for 2a01:7e00::f03c:91ff:fe03:45ed: imfatant.com

PORT    STATE    SERVICE
80/tcp  filtered http
443/tcp filtered https

Nmap done: 1 IP address (1 host up) scanned in 3.12 seconds
3 Likes
20

Certbot should use the python package certifi. I think this is now via a dependency in josepy, but it could be declared somewhere else. certifi is the python port of Mozilla's CA bundle, and python should be using that.

IIRC, OpenSSL uses it's own CA store and Curl will use whatever CA store it was built against.

You can try using updatedb and locate to find all the potential openssl installs and trust stores.

I too think this issue may be caused by having either the cross-signed dst or something similar in the trust store. I've had machines affected by that in the past, and I think this was the error.

5 Likes