SSL Error when i hit certbot --apache

My domain is: all domains

I ran this command: sudo certbot --apache

It produced this output:

n unexpected error occurred:
requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)')))

My web server is (include version): Apache 2.4

The operating system my web server runs on is (include version): Arch Linux

My hosting provider, if applicable, is: init7

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

[morta@5erver ~]$ sudo certbot --version
certbot 1.19.0

[morta@5erver ~]$ cat /etc/hosts
#
127.0.0.1 localhost 5erver
::1 localhost 5erver

[morta@5erver ~]$ cat /etc/hostname
5erver

[morta@5erver ~]$ cat /etc/resolv.conf

Generated by NetworkManager

search home
nameserver 77.109.128.2
nameserver 213.144.129.20
nameserver 2001:1620:2777:1::10

NOTE: the libc resolver may not support more than 3 nameservers.

The nameservers listed below may not be recognized.

nameserver 2001:1620:2777:2::20

[morta@5erver ~]$ dig acme-v02.api.letsencrypt.org

; <<>> DiG 9.16.21 <<>> acme-v02.api.letsencrypt.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3289
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;acme-v02.api.letsencrypt.org. IN A

;; ANSWER SECTION:
acme-v02.api.letsencrypt.org. 2582 IN CNAME prod.api.letsencrypt.org.
prod.api.letsencrypt.org. 300 IN CNAME ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com. 34 IN A 172.65.32.248

;; Query time: 39 msec
;; SERVER: 77.109.128.2#53(77.109.128.2)
;; WHEN: Wed Oct 06 12:13:24 UTC 2021
;; MSG SIZE rcvd: 155

[morta@5erver ~]$ curl -v https://acme-v02.api.letsencrypt.org/

  • Trying 172.65.32.248:443...
  • Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=acme-v02.api.letsencrypt.org
  • start date: Sep 30 00:32:23 2021 GMT
  • expire date: Dec 29 00:32:22 2021 GMT
  • subjectAltName: host "acme-v02.api.letsencrypt.org" matched cert's "acme-v02.api.letsencrypt.org"
  • issuer: C=US; O=Let's Encrypt; CN=R3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multiplexing
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x55bb62d87ab0)

GET / HTTP/2
Host: acme-v02.api.letsencrypt.org
user-agent: curl/7.79.1
accept: /

  • Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
    < HTTP/2 200
    < server: nginx
    < date: Wed, 06 Oct 2021 11:18:45 GMT
    < content-type: text/html
    < content-length: 2174
    < last-modified: Wed, 18 Aug 2021 16:36:00 GMT
    < etag: "611d36f0-87e"
    < x-frame-options: DENY
    < strict-transport-security: max-age=604800
    <
Boulder: The Let's Encrypt CA

  <div class="col-xs-6 text-left">
    <h1>Boulder<br>
    <small>The Let's Encrypt CA</small></h1>
  </div>
</div>

<div class="row">
  <div class="col-xs-8 col-xs-offset-2 text-center">
    <h3>This is an <a href="https://github.com/letsencrypt/acme-spec/">ACME</a> Certificate Authority running <a href="https://github.com/letsencrypt/boulder">Boulder</a>.</h3>
    <p>This is a <em>programmatic</em> endpoint, an API for a computer to talk to. You should probably be using a specialized client to utilize the service, and not your web browser. See <a href="https://letsencrypt.org/"><tt>https://letsencrypt.org/</tt></a> for help.</p>
    <p>If you're trying to use this service, note that the starting point, <em>the directory</em>, is available at this URL: <a href="https://acme-v02.api.letsencrypt.org/directory"><tt>https://acme-v02.api.letsencrypt.org/directory</a></tt>.</p>
  </div>
</div>
<div class="row">
  <div class="col-xs-4 col-xs-offset-2 text-center">
    <p><a href="https://letsencrypt.status.io" title="Twitter">
      <i class="fa fa-area-chart"></i>
      Service Status (letsencrypt.status.io)
    </a></p>
  </div>
  <div class="col-xs-4 text-center">
    <p><a href="https://twitter.com/letsencrypt" title="Twitter">
      <i class="fa fa-twitter"></i>
      Check with us on Twitter
    </a></p>
  </div>
</div> <!-- row -->
* Connection #0 to host acme-v02.api.letsencrypt.org left intact [morta@5erver ~]$ openssl s_client -connect acme-v02.api.letsencrypt.org:443 -servername acme-v02.api.letsencrypt.org CONNECTED(00000003) depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3 verify error:num=10:certificate has expired notAfter=Sep 30 14:01:15 2021 GMT verify return:1 depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3 notAfter=Sep 30 14:01:15 2021 GMT verify return:1 depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 notAfter=Sep 30 18:14:03 2024 GMT verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 notAfter=Sep 15 16:00:00 2025 GMT verify return:1 depth=0 CN = acme-v02.api.letsencrypt.org notAfter=Dec 29 00:30:20 2021 GMT verify return:1 --- Certificate chain 0 s:CN = acme-v02.api.letsencrypt.org i:C = US, O = Let's Encrypt, CN = R3 1 s:C = US, O = Let's Encrypt, CN = R3 i:C = US, O = Internet Security Research Group, CN = ISRG Root X1 --- Server certificate -----BEGIN CERTIFICATE----- MIIF4TCCBMmgAwIBAgISA/zMltu06YsYqsggX2+boUuTMA0GCSqGSIb3DQEBCwUA MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD EwJSMzAeFw0yMTA5MzAwMDMwMjFaFw0yMTEyMjkwMDMwMjBaMCcxJTAjBgNVBAMT HGFjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcwggEiMA0GCSqGSIb3DQEBAQUA A4IBDwAwggEKAoIBAQDnMmZAK8PHpPwp0uJcAQ0TGIkqvPJXCoGvvIQjgzzT8rxQ fgURc+4/bcH/kis1VKHLqT2FKSuNHvnRp6D2Q7kXbT4qGo4MDiayiRGLqrPxKigr vKEXNAK++y2iW42Nd81SNXTIqtXvVZTeynx4uos8PIhq4kx6BJgXl21tHikwYPda wxbgBxIpFN8PmcRygq/ELhf42E3ndQ20XkxAXVGbjqDb3qIgpCdyDxbB5DQyWRiU GjnoOq6DshNRkRBKgBsCJyrzCfkgPr38NAQDq8BIfYYDsmaToFUVqtFmI0ghPi/2 25GE/0x2z0KqNdtKGFO3gXxmeW9ZWxUATLlnmEwlAgMBAAGjggL6MIIC9jAOBgNV HQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1Ud EwEB/wQCMAAwHQYDVR0OBBYEFBhX1UYWBmnSokSGx9r0FslCJxNoMB8GA1UdIwQY MBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEF BQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8v cjMuaS5sZW5jci5vcmcvMIHJBgNVHREEgcEwgb6CHmFjbWUtdjAyLTEuYXBpLmxl dHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItMi5hcGkubGV0c2VuY3J5cHQub3Jngh5h Y21lLXYwMi0zLmFwaS5sZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAyLTQuYXBpLmxl dHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItNS5hcGkubGV0c2VuY3J5cHQub3Jnghxh Y21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnMEwGA1UdIARFMEMwCAYGZ4EMAQIB MDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0c2Vu Y3J5cHQub3JnMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDwAHYAb1N2rDHwMRnYmQCk URX/dxUcEdkCwQApBo2yCJo32RMAAAF8NFLoUwAABAMARzBFAiBXr/5iDsWkiUWu IwPXPkvZukvrqm8whPvxmhY77O0OSgIhAI8W+0txsZCeciq44ktYlTYlQHVsswOl LilsIoj/YAZlAHYAfT7y+I//iFVoJMLAyp5SiXkrxQ54CX8uapdomX4i8NcAAAF8 NFLoSwAABAMARzBFAiBuuZpz9Ku6jkNePIIoN+jfznvYLLU2V2KKmHzFr4+/LwIh AN1A1yS2FnuR/GpzdUUCZGhPuQSZhnbirscf/HLfEAOpMA0GCSqGSIb3DQEBCwUA A4IBAQCJteyN2nPZfPq+s62FnB318NoOFWTI7Zlhp5S4roA7Z45qpHoWXrKolBvX jkCh4kuKqh1U2F5jK5z0crInPyQh9czHYTRWO0GGw53WQBFWEJSUCuvPLlPYz8Ih /Vf0HWzM9zsVQLAJTRBvBM9aOc6qAbtJSYVeK5tPUO7nX+mcNzVCxiGDAFCxmVVK zQAn9TPcgjzwox4F2NNAZ6vOdc5ctfebaLMAFKLkiHWuFMZmhwAjd+1yU50QcfEm 3JfZmykrSt0/9Nqe8Hi2x6OEy/xKzm/jv+UENZeKR1pmF6DoJVo+wNMBzKFWdWwL b1R6BaPSS1uYxw4APu0CBix7kCRJ -----END CERTIFICATE----- subject=CN = acme-v02.api.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = R3


No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits

SSL handshake has read 3300 bytes and written 423 bytes
Verification error: certificate has expired

New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: DD80E5D63A08FEF1DDAA946BA0841BAF850B87C9B7C125D9BE9B48CFDD9171AB
Session-ID-ctx:
Master-Key: CDC999491175E7A07D818F2BA4D00EA257B00CD0F8EE281F6DB789DFA73B1FAD286F2FFFF60B75B55A80266DA1E64DB7
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1633522429
Timeout : 7200 (sec)
Verify return code: 10 (certificate has expired)
Extended master secret: yes

No clue what the error is. It was running till today....

It doesn't make much sense to me that your OpenSSL command would return a chain up to DST Root CA X3, as acme-v02.api.letsencrypt.org does not send the cross-signed ISRG Root CA X1 intermediate, but just the R3 intermediate.

Maybe your system has the cross-signed ISRG Root X1 certificate stored somewhere?

And how i can fix it?
Under /etc/ssl/certs i have this two ISRG Root X1 Certs

ISRG_Root_X1.1.pem
ISRG_Root_X1.pem

I have to delete it?

Very annoying error...

Check the output of:
openssl x509 -in /etc/ssl/certs/ISRG_Root_X1.1.pem -noout -text

If the issuer is CN = DST Root CA X3, this probably the file you can get rid of.

[morta@5erver certs]$ openssl x509 -in /etc/ssl/certs/ISRG_Root_X1.1.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
82:10:cf:b0:d2:40:e3:59:44:63:e0:bb:63:82:8b:00
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Validity
Not Before: Jun 4 11:04:38 2015 GMT
Not After : Jun 4 11:04:38 2035 GMT
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (4096 bit)
Modulus:
00:ad:e8:24:73:f4:14:37:f3:9b:9e:2b:57:28:1c:
87:be:dc:b7:df:38:90:8c:6e:3c:e6:57:a0:78:f7:
75:c2:a2:fe:f5:6a:6e:f6:00:4f:28:db:de:68:86:
6c:44:93:b6:b1:63:fd:14:12:6b:bf:1f:d2:ea:31:
9b:21:7e:d1:33:3c:ba:48:f5:dd:79:df:b3:b8:ff:
12:f1:21:9a:4b:c1:8a:86:71:69:4a:66:66:6c:8f:
7e:3c:70:bf:ad:29:22:06:f3:e4:c0:e6:80:ae:e2:
4b:8f:b7:99:7e:94:03:9f:d3:47:97:7c:99:48:23:
53:e8:38:ae:4f:0a:6f:83:2e:d1:49:57:8c:80:74:
b6:da:2f:d0:38:8d:7b:03:70:21:1b:75:f2:30:3c:
fa:8f:ae:dd:da:63:ab:eb:16:4f:c2:8e:11:4b:7e:
cf:0b:e8:ff:b5:77:2e:f4:b2:7b:4a:e0:4c:12:25:
0c:70:8d:03:29:a0:e1:53:24:ec:13:d9:ee:19:bf:
10:b3:4a:8c:3f:89:a3:61:51:de:ac:87:07:94:f4:
63:71:ec:2e:e2:6f:5b:98:81:e1:89:5c:34:79:6c:
76:ef:3b:90:62:79:e6:db:a4:9a:2f:26:c5:d0:10:
e1:0e:de:d9:10:8e:16:fb:b7:f7:a8:f7:c7:e5:02:
07:98:8f:36:08:95:e7:e2:37:96:0d:36:75:9e:fb:
0e:72:b1:1d:9b:bc:03:f9:49:05:d8:81:dd:05:b4:
2a:d6:41:e9:ac:01:76:95:0a:0f:d8:df:d5:bd:12:
1f:35:2f:28:17:6c:d2:98:c1:a8:09:64:77:6e:47:
37:ba:ce:ac:59:5e:68:9d:7f:72:d6:89:c5:06:41:
29:3e:59:3e:dd:26:f5:24:c9:11:a7:5a:a3:4c:40:
1f:46:a1:99:b5:a7:3a:51:6e:86:3b:9e:7d:72:a7:
12:05:78:59:ed:3e:51:78:15:0b:03:8f:8d:d0:2f:
05:b2:3e:7b:4a:1c:4b:73:05:12:fc:c6:ea:e0:50:
13:7c:43:93:74:b3:ca:74:e7:8e:1f:01:08:d0:30:
d4:5b:71:36:b4:07:ba:c1:30:30:5c:48:b7:82:3b:
98:a6:7d:60:8a:a2:a3:29:82:cc:ba:bd:83:04:1b:
a2:83:03:41:a1:d6:05:f1:1b:c2:b6:f0:a8:7c:86:
3b:46:a8:48:2a:88:dc:76:9a:76:bf:1f:6a:a5:3d:
19:8f:eb:38:f3:64:de:c8:2b:0d:0a:28:ff:f7:db:
e2:15:42:d4:22:d0:27:5d:e1:79:fe:18:e7:70:88:
ad:4e:e6:d9:8b:3a:c6:dd:27:51:6e:ff:bc:64:f5:
33:43:4f
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E
Signature Algorithm: sha256WithRSAEncryption
55:1f:58:a9:bc:b2:a8:50:d0:0c:b1:d8:1a:69:20:27:29:08:
ac:61:75:5c:8a:6e:f8:82:e5:69:2f:d5:f6:56:4b:b9:b8:73:
10:59:d3:21:97:7e:e7:4c:71:fb:b2:d2:60:ad:39:a8:0b:ea:
17:21:56:85:f1:50:0e:59:eb:ce:e0:59:e9:ba:c9:15:ef:86:
9d:8f:84:80:f6:e4:e9:91:90:dc:17:9b:62:1b:45:f0:66:95:
d2:7c:6f:c2:ea:3b:ef:1f:cf:cb:d6:ae:27:f1:a9:b0:c8:ae:
fd:7d:7e:9a:fa:22:04:eb:ff:d9:7f:ea:91:2b:22:b1:17:0e:
8f:f2:8a:34:5b:58:d8:fc:01:c9:54:b9:b8:26:cc:8a:88:33:
89:4c:2d:84:3c:82:df:ee:96:57:05:ba:2c:bb:f7:c4:b7:c7:
4e:3b:82:be:31:c8:22:73:73:92:d1:c2:80:a4:39:39:10:33:
23:82:4c:3c:9f:86:b2:55:98:1d:be:29:86:8c:22:9b:9e:e2:
6b:3b:57:3a:82:70:4d:dc:09:c7:89:cb:0a:07:4d:6c:e8:5d:
8e:c9:ef:ce:ab:c7:bb:b5:2b:4e:45:d6:4a:d0:26:cc:e5:72:
ca:08:6a:a5:95:e3:15:a1:f7:a4:ed:c9:2c:5f:a5:fb:ff:ac:
28:02:2e:be:d7:7b:bb:e3:71:7b:90:16:d3:07:5e:46:53:7c:
37:07:42:8c:d3:c4:96:9c:d5:99:b5:2a:e0:95:1a:80:48:ae:
4c:39:07:ce:cc:47:a4:52:95:2b:ba:b8:fb:ad:d2:33:53:7d:
e5:1d:4d:6d:d5:a1:b1:c7:42:6f:e6:40:27:35:5c:a3:28:b7:
07:8d:e7:8d:33:90:e7:23:9f:fb:50:9c:79:6c:46:d5:b4:15:
b3:96:6e:7e:9b:0c:96:3a:b8:52:2d:3f:d6:5b:e1:fb:08:c2:
84:fe:24:a8:a3:89:da:ac:6a:e1:18:2a:b1:a8:43:61:5b:d3:
1f:dc:3b:8d:76:f2:2d:e8:8d:75:df:17:33:6c:3d:53:fb:7b:
cb:41:5f:ff:dc:a2:d0:61:38:e1:96:b8:ac:5d:8b:37:d7:75:
d5:33:c0:99:11:ae:9d:41:c1:72:75:84:be:02:41:42:5f:67:
24:48:94:d1:9b:27:be:07:3f:b9:b8:4f:81:74:51:e1:7a:b7:
ed:9d:23:e2:be:e0:d5:28:04:13:3c:31:03:9e:dd:7a:6c:8f:
c6:07:18:c6:7f:de:47:8e:3f:28:9e:04:06:cf:a5:54:34:77:
bd:ec:89:9b:e9:17:43:df:5b:db:5f:fe:8e:1e:57:a2:cd:40:
9d:7e:62:22:da:de:18:27

But is DST Root CA X1! And now?

How about the other file?

[morta@5erver certs]$ openssl x509 -in /etc/ssl/certs/ISRG_Root_X1.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Validity
Not Before: Jan 20 19:14:03 2021 GMT
Not After : Sep 30 18:14:03 2024 GMT
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (4096 bit)
Modulus:
00:ad:e8:24:73:f4:14:37:f3:9b:9e:2b:57:28:1c:
87:be:dc:b7:df:38:90:8c:6e:3c:e6:57:a0:78:f7:
75:c2:a2:fe:f5:6a:6e:f6:00:4f:28:db:de:68:86:
6c:44:93:b6:b1:63:fd:14:12:6b:bf:1f:d2:ea:31:
9b:21:7e:d1:33:3c:ba:48:f5:dd:79:df:b3:b8:ff:
12:f1:21:9a:4b:c1:8a:86:71:69:4a:66:66:6c:8f:
7e:3c:70:bf:ad:29:22:06:f3:e4:c0:e6:80:ae:e2:
4b:8f:b7:99:7e:94:03:9f:d3:47:97:7c:99:48:23:
53:e8:38:ae:4f:0a:6f:83:2e:d1:49:57:8c:80:74:
b6:da:2f:d0:38:8d:7b:03:70:21:1b:75:f2:30:3c:
fa:8f:ae:dd:da:63:ab:eb:16:4f:c2:8e:11:4b:7e:
cf:0b:e8:ff:b5:77:2e:f4:b2:7b:4a:e0:4c:12:25:
0c:70:8d:03:29:a0:e1:53:24:ec:13:d9:ee:19:bf:
10:b3:4a:8c:3f:89:a3:61:51:de:ac:87:07:94:f4:
63:71:ec:2e:e2:6f:5b:98:81:e1:89:5c:34:79:6c:
76:ef:3b:90:62:79:e6:db:a4:9a:2f:26:c5:d0:10:
e1:0e:de:d9:10:8e:16:fb:b7:f7:a8:f7:c7:e5:02:
07:98:8f:36:08:95:e7:e2:37:96:0d:36:75:9e:fb:
0e:72:b1:1d:9b:bc:03:f9:49:05:d8:81:dd:05:b4:
2a:d6:41:e9:ac:01:76:95:0a:0f:d8:df:d5:bd:12:
1f:35:2f:28:17:6c:d2:98:c1:a8:09:64:77:6e:47:
37:ba:ce:ac:59:5e:68:9d:7f:72:d6:89:c5:06:41:
29:3e:59:3e:dd:26:f5:24:c9:11:a7:5a:a3:4c:40:
1f:46:a1:99:b5:a7:3a:51:6e:86:3b:9e:7d:72:a7:
12:05:78:59:ed:3e:51:78:15:0b:03:8f:8d:d0:2f:
05:b2:3e:7b:4a:1c:4b:73:05:12:fc:c6:ea:e0:50:
13:7c:43:93:74:b3:ca:74:e7:8e:1f:01:08:d0:30:
d4:5b:71:36:b4:07:ba:c1:30:30:5c:48:b7:82:3b:
98:a6:7d:60:8a:a2:a3:29:82:cc:ba:bd:83:04:1b:
a2:83:03:41:a1:d6:05:f1:1b:c2:b6:f0:a8:7c:86:
3b:46:a8:48:2a:88:dc:76:9a:76:bf:1f:6a:a5:3d:
19:8f:eb:38:f3:64:de:c8:2b:0d:0a:28:ff:f7:db:
e2:15:42:d4:22:d0:27:5d:e1:79:fe:18:e7:70:88:
ad:4e:e6:d9:8b:3a:c6:dd:27:51:6e:ff:bc:64:f5:
33:43:4f
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
Authority Information Access:
CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

        X509v3 Authority Key Identifier: 
            keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

        X509v3 Certificate Policies: 
            Policy: 2.23.140.1.2.1
            Policy: 1.3.6.1.4.1.44947.1.1.1
              CPS: http://cps.root-x1.letsencrypt.org

        X509v3 CRL Distribution Points: 

            Full Name:
              URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl

        X509v3 Subject Key Identifier: 
            79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E
Signature Algorithm: sha256WithRSAEncryption
     0a:73:00:6c:96:6e:ff:0e:52:d0:ae:dd:8c:e7:5a:06:ad:2f:
     a8:e3:8f:bf:c9:0a:03:15:50:c2:e5:6c:42:bb:6f:9b:f4:b4:
     4f:c2:44:88:08:75:cc:eb:07:9b:14:62:6e:78:de:ec:27:ba:
     39:5c:f5:a2:a1:6e:56:94:70:10:53:b1:bb:e4:af:d0:a2:c3:
     2b:01:d4:96:f4:c5:20:35:33:f9:d8:61:36:e0:71:8d:b4:b8:
     b5:aa:82:45:95:c0:f2:a9:23:28:e7:d6:a1:cb:67:08:da:a0:
     43:2c:aa:1b:93:1f:c9:de:f5:ab:69:5d:13:f5:5b:86:58:22:
     ca:4d:55:e4:70:67:6d:c2:57:c5:46:39:41:cf:8a:58:83:58:
     6d:99:fe:57:e8:36:0e:f0:0e:23:aa:fd:88:97:d0:e3:5c:0e:
     94:49:b5:b5:17:35:d2:2e:bf:4e:85:ef:18:e0:85:92:eb:06:
     3b:6c:29:23:09:60:dc:45:02:4c:12:18:3b:e9:fb:0e:de:dc:
     44:f8:58:98:ae:ea:bd:45:45:a1:88:5d:66:ca:fe:10:e9:6f:
     82:c8:11:42:0d:fb:e9:ec:e3:86:00:de:9d:10:e3:38:fa:a4:
     7d:b1:d8:e8:49:82:84:06:9b:2b:e8:6b:4f:01:0c:38:77:2e:
     f9:dd:e7:39

Is X3... So i can delete it and afterwards should work again?

I'd suggest that's the one you can remove yes, maybe just move it to a temporary location rather than delete it.

[morta@5erver ssl]$ openssl s_client -connect acme-v02.api.letsencrypt.org:443 -servername acme-v02.api.letsencrypt.org
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
notAfter=Sep 30 18:14:03 2024 GMT
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
notAfter=Sep 15 16:00:00 2025 GMT
verify return:1
depth=0 CN = acme-v02.api.letsencrypt.org
notAfter=Dec 29 00:27:01 2021 GMT
verify return:1

Certificate chain
0 s:CN = acme-v02.api.letsencrypt.org
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

Server certificate
-----BEGIN CERTIFICATE-----
MIIF4DCCBMigAwIBAgISA6tP1a8gFB06aVE/9do8f/PlMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MzAwMDI3MDJaFw0yMTEyMjkwMDI3MDFaMCcxJTAjBgNVBAMT
HGFjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDgcKVfd8NTebHu70mVIevv7Zt1u3oGDBRCwerNc4Cy6QsF
TAvt85j+pXHPJ6wkjKYp/NEY44Fe0Pk4esuPv52gnEfSmzCB9NaHLOFV4F7beW7k
YSpkwRmXOPl9r3FvIml5pj4axuiAyZmSNHuyBy+oPTTOPdFR1vLzo2S6Qa5Ds0b7
cAcZLHB4Hyl3XqQU/G4hQjXS7cnaV3PzMb9k6MEIg7loGLYoc1qW6KQU3FugFuzL
OamYGjMb3aVm1zn+ozvRcHGdiR+n87COyQ3PVfJngliGB/zS8T9GWq+Aymr1re8h
I242Ozq/M5ZObFZbsAwc3/QHJogvArPS7+blznH3AgMBAAGjggL5MIIC9TAOBgNV
HQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1Ud
EwEB/wQCMAAwHQYDVR0OBBYEFE/38SskswyIt6Dw91v9JKaCQNddMB8GA1UdIwQY
MBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEF
BQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8v
cjMuaS5sZW5jci5vcmcvMIHJBgNVHREEgcEwgb6CHmFjbWUtdjAyLTEuYXBpLmxl
dHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItMi5hcGkubGV0c2VuY3J5cHQub3Jngh5h
Y21lLXYwMi0zLmFwaS5sZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAyLTQuYXBpLmxl
dHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItNS5hcGkubGV0c2VuY3J5cHQub3Jnghxh
Y21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnMEwGA1UdIARFMEMwCAYGZ4EMAQIB
MDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0c2Vu
Y3J5cHQub3JnMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHUAXNxDkv7mq0VEsV6a
1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF8NE/hOAAABAMARjBEAiAvbzKpb3kODfAw
Dg2xzPB4U2yE2Mjf18j7y3I4oi4d8wIgGRGsKyAQIs83XSyiLqxD+S4BTRhorM2B
QK5gdpKpfxIAdgD2XJQv0XcwIhRUGAgwlFaO400TGTO/3wwvIAvMTvFk4wAAAXw0
T+E6AAAEAwBHMEUCICbc9CyWAsnSiQFKwD49gRBprGXg62CvwHTcBiYZtXlmAiEA
y6CjR7+SxiT33eCTOyLe+Ya1PW0Fleen8uVkp02VLFswDQYJKoZIhvcNAQELBQAD
ggEBAERlY4E0zuMJNUE2TePcpRNyM2Vu1AD0yweVjXgRNVOIwb7p/mdoELGkNY3a
cFPj/PP4o9HBs8edvqgL2l/nUSAKr2fSQZSmkkjVKy093ESItgxJcsjTbidQrEAr
+HmkwBZ7KzRq+9adHoHEX6/VYLRKxxj/uHpOUpSxdxYhEORGW4mWeUk69y6celmA
r40BZtFjGsdsVJY0Fu4SBsXSHleLqYPPV9cRLfZVVDDQNoTuhAtJlR9SCoMTKHVo
dd7BuI7EuF+N8/+ZZsEMyfRNBBy3EVcoQ47+RBiGTU5vyZeO9y2pH6SrXT8Q2pSG
Czfn2Dz9djDToKGY7pp6t1wcm4E=
-----END CERTIFICATE-----
subject=CN = acme-v02.api.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = R3


No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits

SSL handshake has read 3299 bytes and written 423 bytes
Verification error: certificate has expired

New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 1FF864883C7F94E8AF8DCC82697C36D1654171607F0668A4929CBD227DD8D59F
Session-ID-ctx:
Master-Key: AAF4EE189F2032C7A931625A3B75D6B8F2CD77E80C6D36C74C517CFBC366C01C463B9363DC43F5D3B7F2DF3C57CDF1AE
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1633585124
Timeout : 7200 (sec)
Verify return code: 10 (certificate has expired)
Extended master secret: yes

Still same error... I moved the cert to /tmp/. Have i to reboot it?
I have a Cert with Name too DST_Root_CA_X3.pem . Is this one?

Hmm, you could but how old is your /etc/ssl/certs/ca-certificates.crt file? It seems like something is still presenting the ISRG Root X1 intermediate instead of using ISRG Root X1 (self-signed)

Does Arch linux have a command for updating ca certificates? Like update-ca-certificates?

Arch Linux doesn't known update-ca-certificates but sudo update-ca-trust

Should be the same, or not? But still no effect

Ok, I think you need to research exactly how to update your ca-certificates bundle for your OS, that will help a lot especially for curl, openssl etc.

Google suggests it could be trust extract-compat How-to-Add-trusted-root-certificates/README.md at master · Busindre/How-to-Add-trusted-root-certificates · GitHub but update-ca-trust also appears to commit changes you've made to the /etc/ssl/certs

This issue is more about getting your OS to behave than anything Let's Encrypt specific.

Arch Linux has this two command to update the certs

  • trust extract-compat or
  • update-ca-trust

I ran the commands but no effect. So i have to download the cert of the letsencrypt site and add to the trusted certs?

Or what do you want to say exactly?

[morta@5erver certs]$ sudo update-ca-trust
[morta@5erver certs]$ openssl x509 -in /etc/ssl/certs/ISRG_Root_X1.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    00:ad:e8:24:73:f4:14:37:f3:9b:9e:2b:57:28:1c:
                    87:be:dc:b7:df:38:90:8c:6e:3c:e6:57:a0:78:f7:
                    75:c2:a2:fe:f5:6a:6e:f6:00:4f:28:db:de:68:86:
                    6c:44:93:b6:b1:63:fd:14:12:6b:bf:1f:d2:ea:31:
                    9b:21:7e:d1:33:3c:ba:48:f5:dd:79:df:b3:b8:ff:
                    12:f1:21:9a:4b:c1:8a:86:71:69:4a:66:66:6c:8f:
                    7e:3c:70:bf:ad:29:22:06:f3:e4:c0:e6:80:ae:e2:
                    4b:8f:b7:99:7e:94:03:9f:d3:47:97:7c:99:48:23:
                    53:e8:38:ae:4f:0a:6f:83:2e:d1:49:57:8c:80:74:
                    b6:da:2f:d0:38:8d:7b:03:70:21:1b:75:f2:30:3c:
                    fa:8f:ae:dd:da:63:ab:eb:16:4f:c2:8e:11:4b:7e:
                    cf:0b:e8:ff:b5:77:2e:f4:b2:7b:4a:e0:4c:12:25:
                    0c:70:8d:03:29:a0:e1:53:24:ec:13:d9:ee:19:bf:
                    10:b3:4a:8c:3f:89:a3:61:51:de:ac:87:07:94:f4:
                    63:71:ec:2e:e2:6f:5b:98:81:e1:89:5c:34:79:6c:
                    76:ef:3b:90:62:79:e6:db:a4:9a:2f:26:c5:d0:10:
                    e1:0e:de:d9:10:8e:16:fb:b7:f7:a8:f7:c7:e5:02:
                    07:98:8f:36:08:95:e7:e2:37:96:0d:36:75:9e:fb:
                    0e:72:b1:1d:9b:bc:03:f9:49:05:d8:81:dd:05:b4:
                    2a:d6:41:e9:ac:01:76:95:0a:0f:d8:df:d5:bd:12:
                    1f:35:2f:28:17:6c:d2:98:c1:a8:09:64:77:6e:47:
                    37:ba:ce:ac:59:5e:68:9d:7f:72:d6:89:c5:06:41:
                    29:3e:59:3e:dd:26:f5:24:c9:11:a7:5a:a3:4c:40:
                    1f:46:a1:99:b5:a7:3a:51:6e:86:3b:9e:7d:72:a7:
                    12:05:78:59:ed:3e:51:78:15:0b:03:8f:8d:d0:2f:
                    05:b2:3e:7b:4a:1c:4b:73:05:12:fc:c6:ea:e0:50:
                    13:7c:43:93:74:b3:ca:74:e7:8e:1f:01:08:d0:30:
                    d4:5b:71:36:b4:07:ba:c1:30:30:5c:48:b7:82:3b:
                    98:a6:7d:60:8a:a2:a3:29:82:cc:ba:bd:83:04:1b:
                    a2:83:03:41:a1:d6:05:f1:1b:c2:b6:f0:a8:7c:86:
                    3b:46:a8:48:2a:88:dc:76:9a:76:bf:1f:6a:a5:3d:
                    19:8f:eb:38:f3:64:de:c8:2b:0d:0a:28:ff:f7:db:
                    e2:15:42:d4:22:d0:27:5d:e1:79:fe:18:e7:70:88:
                    ad:4e:e6:d9:8b:3a:c6:dd:27:51:6e:ff:bc:64:f5:
                    33:43:4f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            Authority Information Access: 
                CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

            X509v3 Authority Key Identifier: 
                keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.root-x1.letsencrypt.org

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl

            X509v3 Subject Key Identifier: 
                79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E
    Signature Algorithm: sha256WithRSAEncryption
         0a:73:00:6c:96:6e:ff:0e:52:d0:ae:dd:8c:e7:5a:06:ad:2f:
         a8:e3:8f:bf:c9:0a:03:15:50:c2:e5:6c:42:bb:6f:9b:f4:b4:
         4f:c2:44:88:08:75:cc:eb:07:9b:14:62:6e:78:de:ec:27:ba:
         39:5c:f5:a2:a1:6e:56:94:70:10:53:b1:bb:e4:af:d0:a2:c3:
         2b:01:d4:96:f4:c5:20:35:33:f9:d8:61:36:e0:71:8d:b4:b8:
         b5:aa:82:45:95:c0:f2:a9:23:28:e7:d6:a1:cb:67:08:da:a0:
         43:2c:aa:1b:93:1f:c9:de:f5:ab:69:5d:13:f5:5b:86:58:22:
         ca:4d:55:e4:70:67:6d:c2:57:c5:46:39:41:cf:8a:58:83:58:
         6d:99:fe:57:e8:36:0e:f0:0e:23:aa:fd:88:97:d0:e3:5c:0e:
         94:49:b5:b5:17:35:d2:2e:bf:4e:85:ef:18:e0:85:92:eb:06:
         3b:6c:29:23:09:60:dc:45:02:4c:12:18:3b:e9:fb:0e:de:dc:
         44:f8:58:98:ae:ea:bd:45:45:a1:88:5d:66:ca:fe:10:e9:6f:
         82:c8:11:42:0d:fb:e9:ec:e3:86:00:de:9d:10:e3:38:fa:a4:
         7d:b1:d8:e8:49:82:84:06:9b:2b:e8:6b:4f:01:0c:38:77:2e:
         f9:dd:e7:39


[morta@5erver certs]$ sudo certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1129)')))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
[morta@5erver certs]$ 

So the cert is right?! And vaild till 2024? Should be right now?
But still the same error....