hmm...
So it doesn't look like an IPv6 thing after all - dang! (there goes my perfect score - LOL)
hmm...
I'd update certbot to the latest version [1.20.0] (or use another ACME client).
But first, we should have a look at the LE log file.
case_le.log.txt (45.2 KB)
Do you know of hand if an in-place upgrade of Certbot is doable?
Effectively "in-place" is the only thing that is ever done.
Which means: You can uninstall it and when you install the next version it picks up where this one left off.
Technically NOT an upgrade at all - more like a rip and replace, but the net effect is that you continue to have all the certs and setting from before - they are not removed during the uninstall, nor replaced during the subsequent (newer version) installation.
As for the problem, your logs show:
2021-10-06 03:50:04,559:WARNING:certbot._internal.main:Python 3.5 support will be dropped in the next release of Certbot - please upgrade your Python version.
2021-10-06 03:50:04,577:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2021-10-06 03:50:04,666:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/home/certbot/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/certbot/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/home/certbot/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
conn.connect()
File "/home/certbot/.local/lib/python3.5/site-packages/urllib3/connection.py", line 344, in connect
ssl_context=context)
File "/home/certbot/.local/lib/python3.5/site-packages/urllib3/util/ssl_.py", line 342, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
_context=self)
File "/usr/lib/python3.5/ssl.py", line 752, in __init__
self.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 988, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 633, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
So I'm leaning towards something updated URLLIB3 and now python3.5 is unhappy.
If so, you could try updating URLLIB3 and maybe this problem goes away...
OR
Uninstall certbot
v1.7.0 and switch to the latest snap
version of certbot
which brings all the necessary files with it (and any newer features - like: --preferred-chain
[added in v1.12]).
Let me guess...
Ubuntu 16? 18?
If so, then:
apt install python3-urllib3
I get:
python3-urllib3 is already the newest version (1.13.1-2ubuntu0.16.04.4).
python3-urllib3 is already the newest version (1.22-1ubuntu0.18.04.2).
So you've updated your ca-certificates now?
I have not. Does the certificate authority need to be updated?
I looked at some of the lines in those python files that were called out, and it seemed to be different SSL modules
Which O/S are you using (and version)?
old as dirt.
Ubuntu 16.04
This has to be related to the Root CA changing at the end of Sept. I have certbot certonly
being run for the 3 different DNS registrars on this certbot server, and all of them stopped working on the 1st of Oct.
I just don't know how to fix it.
Example output from LE errors:
certbot_dme_2021-09-27_cron_err.log
certbot_dme_2021-09-28_cron_err.log
certbot_dme_2021-09-29_cron_err.log
certbot_dme_2021-09-30_cron_err.log
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
certbot_dme_2021-10-01_cron_err.log
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
certbot_dme_2021-10-02_cron_err.log
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),)) ....
+--------------------------------------------------------------------------
certbot_fipa_2021-09-26_cron_err.log
certbot_fipa_2021-09-27_cron_err.log
certbot_fipa_2021-09-28_cron_err.log
certbot_fipa_2021-09-29_cron_err.log
certbot_fipa_2021-09-30_cron_err.log
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
certbot_fipa_2021-10-01_cron_err.log
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
certbot_fipa_2021-10-02_cron_err.log
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),)) ...
+------------------------------------------------------------------
certbot_pdns_2021-09-27_cron_err.log
certbot_pdns_2021-09-28_cron_err.log
certbot_pdns_2021-09-29_cron_err.log
certbot_pdns_2021-09-30_cron_err.log
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
certbot_pdns_2021-10-01_cron_err.log
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
certbot_pdns_2021-10-02_cron_err.log
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),)) ...
+------------------------------------------------------------------------------
I don't see that specific IdenTrust CA on my certbot server, and I do see the ISRG_Root_X1
one in my trust store.
$ ls -lsa /etc/ssl/certs | grep -i -E 'iden|isrg'
0 lrwxrwxrwx 1 root root 37 Oct 31 2019 1e08bfd1.0 -> IdenTrust_Public_Sector_Root_CA_1.pem
0 lrwxrwxrwx 1 root root 16 Oct 31 2019 4042bcee.0 -> ISRG_Root_X1.pem
0 lrwxrwxrwx 1 root root 37 Oct 31 2019 4be590e0.0 -> IdenTrust_Public_Sector_Root_CA_1.pem
0 lrwxrwxrwx 1 root root 16 Oct 31 2019 6187b673.0 -> ISRG_Root_X1.pem
0 lrwxrwxrwx 1 root root 34 Oct 31 2019 d18e9066.0 -> IdenTrust_Commercial_Root_CA_1.pem
0 lrwxrwxrwx 1 root root 34 Oct 31 2019 ef954a4e.0 -> IdenTrust_Commercial_Root_CA_1.pem
0 lrwxrwxrwx 1 root root 69 Oct 30 2019 IdenTrust_Commercial_Root_CA_1.pem -> /usr/share/ca-certificates/mozilla/IdenTrust_Commercial_Root_CA_1.crt
0 lrwxrwxrwx 1 root root 72 Oct 30 2019 IdenTrust_Public_Sector_Root_CA_1.pem -> /usr/share/ca-certificates/mozilla/IdenTrust_Public_Sector_Root_CA_1.crt
0 lrwxrwxrwx 1 root root 51 Oct 30 2019 ISRG_Root_X1.pem -> /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
$ openssl x509 -in ISRG_Root_X1.pem -text | grep After
Not After : Jun 4 11:04:38 2035 GMT
$ openssl x509 -in IdenTrust_Public_Sector_Root_CA_1.pem -text | grep After
Not After : Jan 16 17:53:32 2034 GMT
$ openssl x509 -in IdenTrust_Commercial_Root_CA_1.pem -text | grep After
Not After : Jan 16 18:12:23 2034 GMT
Again, please show the output of these:
sudo apt update
sudo apt-get update
sudo apt install python3-urllib3
Sorry, I meant to send this. Looks like that is already installed. I did apt update
/ apt-get update
$ apt list --installed | grep -i python3-u
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
python3-update-manager/now 1:16.04.16 all [installed,upgradable to: 1:16.04.17]
python3-urllib3/now 1.13.1-2ubuntu0.16.04.3 all [installed,upgradable to: 1.13.1-2ubuntu0.16.04.4]
And yet again...
please show the output of these:
sudo apt update
sudo apt-get update
Don't tell me you meant to send it that you have already done that.
SHOW ME THE COMPLETE OUTPUT.
If you were paying for this support (by the minute), I'm sure you wouldn't be wasting any time.
$ apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Ign:4 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial InRelease
Err:5 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial Release
404 Not Found
Get:6 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [1,648 kB]
Get:8 http://security.ubuntu.com/ubuntu xenial-security/main i386 Packages [1,159 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [2,049 kB]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [380 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [785 kB]
Get:12 http://security.ubuntu.com/ubuntu xenial-security/universe i386 Packages [665 kB]
Get:13 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [225 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [1,525 kB]
Get:15 http://us.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [482 kB]
Get:16 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [1,219 kB]
Get:17 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [1,086 kB]
Get:18 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [358 kB]
Get:19 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [11.3 kB]
Get:20 http://us.archive.ubuntu.com/ubuntu xenial-backports/universe i386 Packages [10.9 kB]
Reading package lists... Done
E: The repository 'https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
root@ply-certbot01:/etc/ssl/certs
$ apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Ign:5 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial InRelease
Ign:6 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial Release
Ign:7 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main amd64 Packages.diff/Index
Ign:8 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main i386 Packages.diff/Index
Ign:9 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main all Packages
Ign:10 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en_US
Ign:11 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en
Ign:12 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main amd64 Packages
Ign:13 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main i386 Packages
Ign:9 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main all Packages
Ign:10 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en_US
Ign:11 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en
Ign:12 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main amd64 Packages
Ign:13 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main i386 Packages
Ign:9 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main all Packages
Ign:10 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en_US
Ign:11 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en
Ign:12 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main amd64 Packages
Ign:13 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main i386 Packages
Ign:9 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main all Packages
Ign:10 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en_US
Ign:11 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en
Ign:12 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main amd64 Packages
Ign:13 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main i386 Packages
Ign:9 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main all Packages
Ign:10 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en_US
Ign:11 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en
Ign:12 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main amd64 Packages
Ign:13 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main i386 Packages
Ign:9 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main all Packages
Ign:10 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en_US
Ign:11 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main Translation-en
Err:12 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main amd64 Packages
404 Not Found
Ign:13 https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial/main i386 Packages
Reading package lists... Done
W: The repository 'https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest/dists/xenial/main/binary-amd64/Packages 404 Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.
"Post back if not resolved" here I am again posting, Mike. I think you are on to something with SSL. The errors are SSL errors. My openssl is 1.0.2g. It looks like I can maybe upgrade to 1.0.2k, so not much of an upgrade.
Yeah.
The only thing I can think of is (if you haven't already done this):
I did remove the DST Root X3 on the certbot server