Certbot CERTIFICATE_VERIFY_FAILED

Hi there,

I have an Ubuntu 20.04 Server and I have installed the Certbot via snap as described in the official documentation: Certbot - Certbot Instructions

My Problem is that i'am not able to run the Certbot correctly. When I start it with "certbot" or "certbot --nginx" it asks for my Email Address. After that the bot crashes with the following error Message:

An unexpected error occured:
requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get locale issuer certificate (_ssl.c:1131)')))

I have to mention that my Server is behind a Proxy. But the Proxy is correctly defined via the environment variables http_proxy, https_proxy, HTTP_PROXY, HTTPS_PROXY.

If I try to call the above URL via CURL e.g.:

curl -v https://acme-v02.api.letsencrypt.org/directory

I get a correct Webserver Response with JSON content. My Certbot Version is 1.17.0.

Any Ideas?

Thanks! :slight_smile:

Did you have any other version of certbot installed before the snapd version?
If so, did you uninstall it (as requested in that doc)?

Nope there was no older Version installed. It was a clean Ubuntu install.

Ok, what version of OpenSSL do you have?

Better yet, show full output of:
curl --version

I just tried snapd certbot on Ubuntu 21.04 and that worked just fine.

Is this proxy a TLS intercepting proxy? HTTPS proxies that intercept the connection don't work unless you make the client using them explicitly trust the proxy's own CA.

1 Like

OpenSSL is: 1.1.1f 31 Mar 2020
Curl is: 7.68.0

Thanks that sounds promising. Is there any way to tell Certbot to trust my CA? The Proxy Certificate is already set up in the OS. Other Programs (like curl or apt) use it and it works. But for some reason the Certbot does not.

1 Like

I've installed certbot via the apt repositories. This Version works out of the box. The snap Version does not.

Huh, well, Certbot is using requests, which is using urllib3.

It looks like using the OS trusted certificate database has been the default behavior in Python for a while.

@_az, do you have any ideas why different Certbot installation methods would now be showing different behavior with regard to the certificate stores they use when connecting as a client to an ACME API endpoint?

1 Like

This might be a wrong explanation but here goes ...

What it looks like to me is that Debian patch some Python packages to read CA certificates from /etc/ssl/certs/ca-certificates.crt (emphasis mine):

Description-en: root certificates for validating SSL certs and verifying TLS hosts (python3)
Certifi is a carefully curated collection of Root Certificates for
validating the trustworthiness of SSL certificates while verifying
the identity of TLS hosts. It has been extracted from the Requests
project.
.
The version of certifi in this Debian package is patched to return
the location of Debian-provided CA certificates, instead of those
packaged by upstream.

If you install certifi from pip directly, you do not benefit from this patch, and the trust store bundled with certifi ends up being the one used. (I don't know really how this squares with what PEP 476 appears to suggest).

With the snap, Certbot was built by sourcing packages directly from pip rather than from the Debian-provided package. As such, it uses certifi's trust store:

# python3 -c "import requests; print(requests.certs.where())"
/snap/certbot/1280/lib/python3.8/site-packages/certifi/cacert.pem

Unlike what would happen with python3-certifi:

$ python3 -c "import requests; print(requests.certs.where())"
/etc/ssl/certs/ca-certificates.crt

So the upshot would seem to be: snapped Certbot (nor pipped, for that matter) is not aware of the Debian-specific /etc/ssl/certs/ca-certificates.crt trust store.

I would guess that what you would want to do in this case, is in addition to setting http_proxy and https_proxy, to also set REQUESTS_CA_BUNDLE to point to that file. The Certbot snap will respect it.

In the future (when Certbot moves to use requests 3.x), it should also be possible to just set SSL_CERT_FILE ... but that was never backported to requests 2.x, which is what Certbot uses today.

2 Likes