Renew doesn't work

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=consultancygrid.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: consultancygrid.com

I ran this command:

[root@consultancygrid ~]# /usr/bin/certbot renew

=================================================
It produced this output:

Processing /etc/letsencrypt/renewal/consultancygrid.com.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Attempting to renew cert (consultancygrid.com) from /etc/letsencrypt/renewal/consultancygrid.com.conf produced an unexpected error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/consultancygrid.com/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/consultancygrid.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

=================================================
My web server is (include version):

Server version: Apache/2.4.6 (CentOS)
Server built: Oct 19 2017 20:39:16

The operating system my web server runs on is (include version):

[root@consultancygrid ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[root@consultancygrid ~]# uname -r
3.10.0-514.10.2.el7.x86_64

=================================================
I can login to a root shell on my machine: yes

[root@consultancygrid ~]# rpm -aq | grep certbot
certbot-0.23.0-1.el7.noarch
python2-certbot-0.23.0-1.el7.noarch
python2-certbot-apache-0.23.0-1.el7.noarch
[root@consultancygrid ~]# rpm -aq | grep urllib
python-urllib3-1.10.2-3.el7.noarch

=================================================

letsencrypt.log

2018-05-08 20:10:54,373:DEBUG:certbot.main:certbot version: 0.23.0
2018-05-08 20:10:54,373:DEBUG:certbot.main:Arguments: [’–test-cert’]
2018-05-08 20:10:54,373:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-05-08 20:10:54,390:DEBUG:certbot.log:Root logging level set at 20
2018-05-08 20:10:54,391:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-05-08 20:10:54,391:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2018-05-08 20:10:54,503:DEBUG:certbot_apache.configurator:Apache version is 2.4.6
2018-05-08 20:10:54,931:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_centos.CentOSConfigurator object at 0x205b810>
Prep: True
2018-05-08 20:10:54,931:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.override_centos.CentOSConfigurator object at 0x205b810> and installer <certbot_apache.override_centos.CentOSConfigurator object at 0x205b810>
2018-05-08 20:10:54,931:INFO:certbot.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2018-05-08 20:11:10,351:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
2018-05-08 20:11:10,354:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
2018-05-08 20:11:10,453:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/bin/certbot”, line 9, in
load_entry_point(‘certbot==0.23.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 1266, in main
return config.func(config, plugins)
File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 1023, in run
le_client = _init_le_client(config, authenticator, installer)
File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 635, in _init_le_client
acc, acme = _determine_account(config)
File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 514, in _determine_account
config, account_storage, tos_cb=_tos_cb)
File “/usr/lib/python2.7/site-packages/certbot/client.py”, line 164, in register
acme = acme_from_config_key(config, key)
File “/usr/lib/python2.7/site-packages/certbot/client.py”, line 46, in acme_from_config_key
return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
File “/usr/lib/python2.7/site-packages/acme/client.py”, line 718, in init
directory = messages.Directory.from_json(net.get(server).json())
File “/usr/lib/python2.7/site-packages/acme/client.py”, line 1041, in get
self._send_request(‘GET’, url, **kwargs), content_type=content_type)
File “/usr/lib/python2.7/site-packages/acme/client.py”, line 990, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/usr/lib/python2.7/site-packages/requests/sessions.py”, line 464, in request
resp = self.send(prep, **send_kwargs)
File “/usr/lib/python2.7/site-packages/requests/sessions.py”, line 576, in send
r = adapter.send(request, **kwargs)
File “/usr/lib/python2.7/site-packages/requests/adapters.py”, line 431, in send
raise SSLError(e, request=request)
SSLError: [Errno 2] No such file or directory
2018-05-08 20:11:10,455:ERROR:certbot.log:An unexpected error occurred:

2

Hi @consultancygrid,

It sounds like your server is unable to connect to the Let’s Encrypt service. Could you try running this command?

curl -v https://acme-v01.api.letsencrypt.org/directory

3

[root@consultancygrid tmp]# curl -v https://acme-v01.api.letsencrypt.org/directory

  • About to connect() to acme-v01.api.letsencrypt.org port 443 (#0)
  • Trying 104.66.73.97...
  • Connected to acme-v01.api.letsencrypt.org (104.66.73.97) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  • Server certificate:
  •   subject: CN=acme-v02.api.letsencrypt.org

  •   start date: Mar 16 00:14:19 2018 GMT

  •   expire date: Jun 14 00:14:19 2018 GMT

  •   common name: acme-v02.api.letsencrypt.org

  •   issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US

GET /directory HTTP/1.1
User-Agent: curl/7.29.0
Host: acme-v01.api.letsencrypt.org
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: application/json
< Content-Length: 658
< Replay-Nonce: zvcR8S1XcSW1uKSww1zi9H3cpkvijPIiqXFcxLqjBns
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
< Expires: Tue, 08 May 2018 19:07:22 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Tue, 08 May 2018 19:07:22 GMT
< Connection: keep-alive
<
{
"NlM-ayIfrWM": "Adding random entries to the directory",
"key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
"new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
"new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
"revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"

4

I suspect that Python library have a bug and cause that issue:
python-urllib3

unfortunately can’t be sure at the moment.

5

Logs are showing connections to staging-v02.

please show: /etc/letsencrypt/renewal/consultancygrid.com.conf

6

You could try (edited according to @mnordhoff’s correction below)

$ python
>>> import requests
>>> v1 = requests.get("https://acme-v01.api.letsencrypt.org/directory")
>>> v2 = requests.get("https://acme-v02.api.letsencrypt.org/directory")
>>> print(str(v1), str(v1.text), str(v2), str(v2.text))
7

python2.7, not python3.

8

Oh yeah, I misread python-urllib3 as python3-urllib3. Thanks for the correction, @mnordhoff!

9

[root@consultancygrid ~]# python
Python 2.7.5 (default, Aug 4 2017, 00:39:18)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import requests
v1 = requests.get("https://acme-v01.api.letsencrypt.org/directory")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/requests/api.py", line 68, in get
return request('get', url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
v2 = requests.get("https://acme-v02.api.letsencrypt.org/directory")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/requests/api.py", line 68, in get
return request('get', url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
print(str(v1), str(v1.text), str(v2), str(v2.text))
Traceback (most recent call last):
File "", line 1, in
NameError: name 'v1' is not defined

10

OK, that’s pretty weird in combination with your curl test succeeding. (If the curl test had failed, I would have said that you might have a firewall intercepting connections to the Let’s Encrypt server, but with the curl test’s success, it seems more likely that the problem is with Python here.) Do you know how Python and the urllib3 package were installed? Did you do anything that could have changed their certificate stores?

11

I use yum to work with rpm package system
all software is installed thru yum. is it worth to try build this library from source ?

[root@consultancygrid ~]# yum info python-urllib3-1.10.2-3.el7.noarch
Loaded plugins: fastestmirror, keys, langpacks, priorities, remove-with-leaves, rpm-warm-cache, upgrade-helper
Loading mirror speeds from cached hostfile

12

@joohoi @bmw, have you seen a problem like this before?

13

This issue looks like it’s caused by outdated, broken or outright wrong CA certificate bundle being used by a Python library called requests that Certbot uses for the communication with Let’s Encrypt.

Short intro: CA certificate bundle is a list of CA root certificates that should be trusted. Some versions of requests shipped with vendored certifi library, providing it with a static CA certificate bundle, but I’m not sure which version and approach is used in CentOS.

Have you done any manual changes to CA bundle files (usually called cacert.pem)?

I don’t have a CentOS 7 installation at hand, so going blind and relying on search results here, take the following with a grain of salt;

One simple thing to try that comes to mind to try would be:
CentOS has a distribution package that ships the CA certificate bundle, called ca-certificates.

yum reinstall ca-certificates

If requests packaged with CentOS 7 uses the distribution CA certificate bundle, this should fix the issue in case of it being corrupted in some way.

1 Like
14

I’ve reinstalled the package:

[root@consultancygrid ~]# yum reinstall ca-certificates
Loaded plugins: fastestmirror, keys, langpacks, priorities, remove-with-leaves, rpm-warm-cache, upgrade-helper
Loading mirror speeds from cached hostfile

Dependencies Resolved

============================================================================================================================================================================================
Package Arch Version Repository Size

Reinstalling:
ca-certificates noarch 2017.2.14-71.el7 base 472 k

Transaction Summary

Reinstall 1 Package

Total download size: 472 k
Installed size: 1.2 M
Is this ok [y/d/N]: y
Downloading packages:
ca-certificates-2017.2.14-71.el7.noarch.rpm | 472 kB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : ca-certificates-2017.2.14-71.el7.noarch 1/1
Verifying : ca-certificates-2017.2.14-71.el7.noarch 1/1

Installed:
ca-certificates.noarch 0:2017.2.14-71.el7

Complete!

unfortunately I receive the same error with certbot renew

15

@joohoi, do you know if there’s any way to get requests to verbosely explain its trust decisions or what certificate chain it tried to validate?

@consultancygrid, I’ve really not seen this particular problem before!

If you need a certificate quickly, you could try to use a different Let’s Encrypt client, like acme.sh, which has a different set of dependencies and which doesn’t use the Python requests package. (I’m still interested in figuring out why this is broken, though.)

16

Looked it up, @consultancygrid could you run the following to find out which CA certificate bundle is used by requests in your system:

python -c 'import requests;print(requests.certs.where())'

this will print the path of the bundle.

2 Likes
17

[root@consultancygrid ~]# python -c ‘import requests;print(requests.certs.where())’
/usr/lib/python2.7/site-packages/certifi/cacert.pem

so I can replace this file with more up to date file and resolve that issue ?

18

I did it and now
certbot renew
works fine

thank you

2 Likes
19

Correct way to handle this would be to probably update the installed certifi package with pip:

pip install certifi -U

Glad to hear you got it working again!

2 Likes
20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.