SSLCertVerificationError of acme-v02.api.letsencrypt.org causes certbot renewal failure

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
heisenbug.com

I ran this command:
certbot -v --work-dir $(pwd)/work --logs-dir $(pwd)/logs --config-dir $(pwd)/config renew --dns-godaddy-credentials credentials.ini

It produced this output:
Failed to renew certificate home.heisenbug.com with error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1002)')))

The operating system my web server runs on is (include version):
Fedora 37

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 2.6.0

With that out of the way, the issue doesn't appear to have anything to do with any of the above. Seems to be an issue with the certificate for acme-v02.api.letsencrypt.org wrt python requests:

$ python -c 'import requests; print(requests.get(url = 'https://acme-v02.api.letsencrypt.org/directory\').text)'
...snip...
requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1002)')))

This is the same error I'm getting from certbot, so I'm assuming this is the cause. certifi is as up to date as I can make it:

$ pip show certifi
Name: certifi
Version: 2023.5.7
Summary: Python package for providing Mozilla's CA Bundle.
Home-page: GitHub - certifi/python-certifi: (Python Distribution) A carefully curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts.
Author: Kenneth Reitz
Author-email: me@kennethreitz.com
License: MPL-2.0
Location: /home/mbooth/.local/lib/python3.11/site-packages
Requires:
Required-by: msrest, requests

Any ideas?

2

Can you connect to https://acme-v02.api.letsencrypt.org/directory using other software on that system, such as curl? How about https://valid-isrgrootx1.letsencrypt.org?

Or maybe even openssl s_client -connect acme-v02.api.letsencrypt.org:443 and see what certificate chain you see?

Is there any chance of some "helpful" firewall intercepting the connection and presenting a fake certificate?

3 Likes
3

curl works fine. It's just python requests.

There is no firewall.

4

Hi @mdbooth, and welcome to the LE community forum :slight_smile:

One:
Is there some sort HTTPS inspection device inline?

What shows?:
openssl s_client -connect acme-v02.api.letsencrypt.org:443

2 Likes
5

Curl works fine:

$ curl https://acme-v02.api.letsencrypt.org/directory
{
"QQvFOvPswYM": "Adding random entries to the directory",
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"renewalInfo": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-01/renewalInfo/",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}

1 Like
6

certbot 2.6.0 runs from snap, so I don't see how you can mess that up.
Unless...
What does your path look like?

2 Likes
7

certbot is installed with pip.

$ which certbot
/home/mbooth/.local/bin/certbot

8

Ok, then that may explain how/why things are how/where they are.

2 Likes
9

Snap is an Ubuntu thing, right?

10

It's mostly Ubuntu, but more than just that.

How [good] are your pip skills?
[mine are less than "beginner"]
But, I think you may have needed to use a separate pip environment for certbot.

2 Likes
11

See:
Certbot Instructions | Certbot (eff.org)

1 Like
12

Can your Pythons requests get other sites successfully? Preferably with the same certificate/cert chain as the ACME server.

3 Likes
13

Yes to the first part (can get other sites). As to the same cert chain, I have no idea. Presumably not.

14

Did you try https://valid-isrgrootx1.letsencrypt.org via python? That would be a site using the same chain.

Is this in its own virtual environment like the certbot instructions say? Can you try reinstalling in a new one?

3 Likes
15

I have tried installing a new one in a different venv, but I should probably open another issue about how that didn't work.

16

I have installed snap, but the distributed version doesn't have the godaddy credentials plugin I use installed. Is there any way to install a plugin in a snap, or do I need to resolve the issues in my pip-installed version?

17

Python cannot fetch https://valid-isrgrootx1.letsencrypt.org, but curl can. Python can fetch https://google.com.

18

The plugin author has to publish their project as a snap.

I have published GitHub - alexzorin/certbot-dns-multi: Certbot DNS plugin supporting multiple providers, using github.com/go-acme/lego as a snap, which includes support for GoDaddy.

5 Likes
19

Seems like it's not using the default/system certificate root store.

2 Likes
20

No, the python requests module does not use the default system certificate root store. It's very annoying. However, this is the case for all deployments. I was kinda hoping that by posting here somebody would have hit this before. For example, if the snap works then it must have a solution for this.