Python requests library can't make HTTPS connections (was: Apache certbot error)

Help
1

Please fill out the fields below so we can help you better.

My domain is:

I ran this command: certbot --apache certonly

It produced this output:
An unexpected error occurred:
Error: [(‘system library’, ‘fopen’, ‘No such file or directory’), (‘BIO routines’, ‘BIO_new_file’, ‘no such file’), (‘x509 certificate routines’, ‘X509_load_cert_crl_file’, ‘system lib’)]
Please see the logfiles in /var/log/letsencrypt for more details.

My operating system is (include version): Debian GNU/Linux 9 (stretch)

My web server is (include version): Server version: Apache/2.4.25 (Debian)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

2

Could you post a copy of the log file from /var/log/letsencrypt?

3

hi Schoen,

Yes, please:

# cat /var/log/letsencrypt/letsencrypt.log
2017-04-06 07:32:40,732:DEBUG:certbot.main:Root logging level set at 20
2017-04-06 07:32:40,733:INFO:certbot.main:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-04-06 07:32:40,734:DEBUG:certbot.main:certbot version: 0.11.1
2017-04-06 07:32:40,734:DEBUG:certbot.main:Arguments: ['--apache']
2017-04-06 07:32:40,735:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#webroot,PluginEntryPoint#null,PluginEntryPoint#manual,PluginEntryPoint#standalone)
2017-04-06 07:32:40,735:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2017-04-06 07:32:42,423:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.configurator:ApacheConfigurator
Initialized: <certbot_apache.configurator.ApacheConfigurator object at 0x7f7412459910>
Prep: True
2017-04-06 07:32:42,425:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.configurator:ApacheConfigurator
Initialized: <certbot_apache.configurator.ApacheConfigurator object at 0x7f7412459910>
Prep: True
2017-04-06 07:32:42,425:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.configurator.ApacheConfigurator object at 0x7f7412459910> and installer <certbot_apache.configurator.ApacheConfigurator object at 0x7f7412459910>
2017-04-06 07:32:50,181:DEBUG:root:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2017-04-06 07:32:50,187:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2017-04-06 07:32:51,339:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in
load_entry_point('certbot==0.11.1', 'console_scripts', 'certbot')()
File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 882, in main
return config.func(config, plugins)
File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 654, in obtain_cert
le_client = _init_le_client(config, auth, installer)
File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 400, in _init_le_client
acc, acme = _determine_account(config)
File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 385, in _determine_account
config, account_storage, tos_cb=_tos_cb)
File "/usr/lib/python2.7/dist-packages/certbot/client.py", line 128, in register
acme = acme_from_config_key(config, key)
File "/usr/lib/python2.7/dist-packages/certbot/client.py", line 42, in acme_from_config_key
return acme_client.Client(config.server, key=key, net=net)
File "/usr/lib/python2.7/dist-packages/acme/client.py", line 69, in init
self.net.get(directory).json())
File "/usr/lib/python2.7/dist-packages/acme/client.py", line 657, in get
self._send_request('GET', url, **kwargs), content_type=content_type)
File "/usr/lib/python2.7/dist-packages/acme/client.py", line 630, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 423, in send
timeout=timeout
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
chunked=chunked)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 350, in _make_request
self._validate_conn(conn)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 837, in validate_conn
conn.connect()
File "/usr/lib/python2.7/dist-packages/urllib3/connection.py", line 323, in connect
ssl_context=context)
File "/usr/lib/python2.7/dist-packages/urllib3/util/ssl.py", line 308, in ssl_wrap_socket
context.load_verify_locations(ca_certs, ca_cert_dir)
File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 392, in load_verify_locations
self._ctx.load_verify_locations(cafile, capath)
File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 525, in load_verify_locations
_raise_current_error()
File "/usr/lib/python2.7/dist-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
raise exception_type(errors)
Error: [('system library', 'fopen', 'No such file or directory'), ('BIO routines', 'BIO_new_file', 'no such file'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]

4

This looks like Python was unable to verify the connection to the Let’s Encrypt API URL because of missing trusted root CA files on your system. Do you have a ca-certificates package installed on your OS? Does /etc/ssl/certs exist and have files in it?

5

I checked - package is installed and files exists:

root@xxx:/etc/ssl/certs# dpkg -l | grep ca-certificates
ii ca-certificates 20161130 all Common CA certificates

root@xxx:/etc/ssl/certs# ls -l | wc -l
530

6

That’s really weird. Can you use urllib3 to connect to other HTTPS web sites from Python?

Like

import requests
requests.request("GET", "https://www.google.com/")
7

root@xxx:/home/user6# python test.py
Traceback (most recent call last):
File “test.py”, line 2, in
requests.request(“GET”, “https://www.google.com/”)
File “/usr/lib/python2.7/dist-packages/requests/api.py”, line 56, in request
return session.request(method=method, url=url, **kwargs)
File “/usr/lib/python2.7/dist-packages/requests/sessions.py”, line 488, in request
resp = self.send(prep, **send_kwargs)
File “/usr/lib/python2.7/dist-packages/requests/sessions.py”, line 609, in send
r = adapter.send(request, **kwargs)
File “/usr/lib/python2.7/dist-packages/requests/adapters.py”, line 423, in send
timeout=timeout
File “/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py”, line 594, in urlopen
chunked=chunked)
File “/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py”, line 350, in _make_request
self._validate_conn(conn)
File “/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py”, line 837, in validate_conn
conn.connect()
File “/usr/lib/python2.7/dist-packages/urllib3/connection.py”, line 323, in connect
ssl_context=context)
File "/usr/lib/python2.7/dist-packages/urllib3/util/ssl.py", line 308, in ssl_wrap_socket
context.load_verify_locations(ca_certs, ca_cert_dir)
File “/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py”, line 392, in load_verify_locations
self._ctx.load_verify_locations(cafile, capath)
File “/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py”, line 525, in load_verify_locations
_raise_current_error()
File “/usr/lib/python2.7/dist-packages/OpenSSL/_util.py”, line 48, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [(‘system library’, ‘fopen’, ‘No such file or directory’), (‘BIO routines’, ‘BIO_new_file’, ‘no such file’), (‘x509 certificate routines’, ‘X509_load_cert_crl_file’, ‘system lib’)]
root@xxx:/home/user#

8

So, there’s something broken about your Python installation which prevents it from making successful HTTPS requests (as a client) at all. That’s the reason that Certbot isn’t working here, because it has to make HTTPS requests to the Let’s Encrypt ACME API via HTTPS. Unfortunately, I haven’t encountered this particular problem before and don’t know the solution to it.

Can you think of anything unusual about how you installed your system or about the version of Python you have?

9

I also renamed this thread in the hope of attracting interest from other people who might be better able to help.

10

hi @ITM

Review this article: Certbot - Certificate renewal on Ubuntu 14.04.5

Can you please do the following

python

This should tell you the version you are running (can you paste it here)

pip freeze

this should tell you the packages you have installed

openssl version

it should look like something below. Yours will be different as I am on a windows system

image.png966×651 17.1 KB

Andrei

11

Hi Andrei,
Thank You for your mail. Please find command below:

root@xxx:/home/user# python
Python 2.7.13 (default, Jan 19 2017, 14:48:08)
[GCC 6.3.0 20170118] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
root@xxx:/homeuser# pip freeze
bash: pip: command not found
root@xxx:/home/user# openssl version
OpenSSL 1.1.0e 16 Feb 2017
root@xxx:/home/user#

12

*Update

After installing python-pip, this is the output of the command pip freeze:

root@xxx:/home/user# pip freeze
acme==0.11.1
certbot==0.11.1
certbot-apache==0.11.1
chardet==2.3.0
ConfigArgParse==0.11.0
configobj==5.0.6
cryptography==1.7.1
dnspython==1.15.0
enum34==1.1.6
funcsigs==1.0.2
httplib2==0.9.2
idna==2.2
ipaddress==1.0.17
keyring==10.1
keyrings.alt==1.3
mock==2.0.0
parsedatetime==2.1
pbr==1.10.0
pyasn1==0.1.9
pycrypto==2.6.1
pycurl==7.43.0
pygobject==3.22.0
PyICU==1.9.5
pyOpenSSL==16.2.0
pyRFC3339==1.0
python-apt==0.9.3.12
python-augeas==0.5.0
python-debian==0.1.30
python-debianbts==2.6.1
pytz==2016.7
pyxdg==0.25
reportbug==6.6.3
requests==2.12.4
SecretStorage==2.3.1
six==1.10.0
urllib3==1.19.1
virtualenv==15.1.0
zope.component==4.3.0
zope.event==4.2.0
zope.hookable==4.0.4
zope.interface==4.3.2

13

I tried to follow your suggestion from the second article, but it doesn’t work:

(venv) root@xxx:/home/user/venv# pip install --upgrade setuptools
Requirement already up-to-date: setuptools in ./lib/python2.7/site-packages
(venv) root@xxx:/home/user/venv# pip install certbot==0.11.1
Collecting certbot==0.11.1
Could not find a version that satisfies the requirement certbot==0.11.1 (from versions: )
No matching distribution found for certbot==0.11.1
(venv) root@xxx:/home/user/venv# pip install certbot==0.12.0
Collecting certbot==0.12.0
Could not find a version that satisfies the requirement certbot==0.12.0 (from versions: )
No matching distribution found for certbot==0.12.0
(venv) root@xxx:/home/user/venv# pip install certbot==0.13.0
Collecting certbot==0.13.0
Could not find a version that satisfies the requirement certbot==0.13.0 (from versions: )
No matching distribution found for certbot==0.13.0

(venv) root@xxx:/home/user/venv# pip install -vvv certbot==0.11.1
Converted retries value: Retry(total=5, connect=None, read=None, redirect=None) -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=None, read=None, redirect=None)
Converted retries value: Retry(total=5, connect=None, read=None, redirect=None) -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=None, read=None, redirect=None)
Collecting certbot==0.11.1
1 location(s) to search for versions of certbot:

14

Solved by:

  1. Removed file: /etc/ssl/certs/ca-certificates.crt
  2. Running:
    $ sudo update-ca-certificates in virtual environment.

Thank You Everyone for Help.

1 Like
15

Thanks for sharing your solution with the community, @ITM.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.