Previously working connection to LE now fails- related to recent root cert exp.?

Help
43

@erglazier Sorry, been away most of the day and have not yet studied your letsencrypt log. I hope to have some time a bit later.

But, going back to my original post #2 to you I have something to try:
curl -I https://acme-v02.api.letsencrypt.org

Your openssl to that endpoint worked in your OP but I recently saw a case where curl and openssl verified the chain differently. With recent chain changes openssl is usually more trouble but perhaps in your case it is not. I think curl more closely mimics what Certbot itself does in its python scripts.

Your Certbot failure still seems like a missing ISRG Root X1. Can you explicitly check whether that appears in your root store?

I also do not think upping openssl from g to k will matter.

1 Like
44

@erglazier
What shows?:
sudo apt list --upgradable

@MikeMcQ
That was done earlier:

@erglazier
At this point, I still say:

The problem is within python/urllib
If it can't be updated, then just use another client.

45

No, the -k would skip cert verification. I want to try without that

Update: Just clarifying I was responding to quote from @rg305 - not @erglazier

1 Like
46

@MikeMcQ
Aaah yes!
Leave no stone unturned.

Although I still think it's python/urllib, that does need to be ruled out.

47

Thanks Mike.
$ curl -I https://acme-v02.api.letsencrypt.org
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 06 Oct 2021 20:56:19 GMT
Content-Type: text/html
Content-Length: 2174
Last-Modified: Wed, 18 Aug 2021 16:36:07 GMT
Connection: keep-alive
ETag: "611d36f7-87e"
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

48

@erglazier I have not yet looked at your log. But, I agree with Rudy and it is worth looking at python libs. IIRC you were at Certbot 1.7 which is fairly recent so unlikely a problem in it. But, another client that does not use python would be worth a try if no python updates helped.

I will still look at the logs to see if we missed something.

1 Like
49

update on certbot,
I was able to get it all running again, the fix was to update the python pip requests package used by certbot. This I believe updated the root cert bundle certbot used, as it was failing to validate the cert on lets encrypt itself when fetching new certs. So now it can fetch new certs and distribute them. This upgrade did not update the Python3-urllib3 version

1 Like
50

@erglazier Yes, I was just going to suggest that. I googled and found this:

Also thinking about it, it could be that there is a different cacert.pem in C:\Python27\lib\site-packages\certifi compared to C:\Python3\lib\site-packages\certifi - github.com/certifi – 
Danny Cullen

It was related to the DST Root CA X3 expiration. If this was the reason it was because your python cert store also had to be updated. Both the curl and openssl verified ok the new ISRG Root X1. But, the python cert store did not have that in it. Your update of certbot seemed to refresh that cert store - as you note.

Very glad you got it sorted. Was very puzzling. I had not seen anyone with this python issue before.

1 Like
51

Mike, you the man! Thank you for volunteering so much time. You made me hip to some other things in LE that I wasn't aware of.

1 Like
52

Many people read through these posts sometimes desperately looking for a solution to very tough problems.
Please include the O/S+version and the exact commands used to create the "fix".
[Think SEO]

1 Like
53

Certbot utilizes the requests library (module) in Python, for making https requests. HTTPs relies on the root CAs for the TLS/SSL chain-of-trust, to make secure requests. Normally these root CAs would be at a base level in an OS or part of a client browser, but in order to do it within Python via the requests library, the library needs the bundle of SSL Root Certificate Authorities (CAs). This older version of Python that this older version Certbot uses, did not have the new Root CA (ISRG Root X1) that Certbot now uses. Therefore, an update to the Python requests package was necessary, to get an updated Root CA bundle

$ certbot --versioncertbot 1.7.0

$ pip --versionpip 20.0.2 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5)

$ python --versionPython 2.7.12

$ head -3 /etc/os-releaseNAME="Ubuntu"VERSION="16.04.6 LTS (Xenial Xerus)"ID=ubuntu

$ pip list --outdated | grep reqrequests (2.9.1) - Latest: 2.26.0

$ pip3 install --upgrade requests.....

$ pip list | grep reqrequests            2.25.1

This assumes that the OS running certbot already is rid of the old Root CA and has the new one.

LE has a url for testing your chain of auth called helloworld.letsencrypt.org . Use this command from your certbot server to see that the chain of auth uses the correct Root CA.

openssl s_client -servername helloworld.letsencrypt.org -showcerts -connect helloworld.letsencrypt.org:443

Ensure that the expired root CA ( DST Root CA X3 ) is out of the list, and the new Root CA used by LE ( ISRG Root X1 ) is in. (Google how to remove / add root CA for specific OS version)
ex.

$ ls /etc/ssl/certs/ | grep ISRG
ISRG_Root_X1.pem
4 Likes
54

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.