Certificate Verify Failed


#1

I found this topic which is pretty much the same issue:

However removing and re-installing the ‘certbot’ package did not resolve the issue. For now, I’m adding
no-verify-ssl = true to the cli.ini file to work around this, but would like to see a more secure solution.

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: rflm.net

I ran this command: certbot certonly --register-unsafely-without-email --config-dir /tmp/lets_encrypt -d box.rflm.net,rflm.net,www.rflm.net

It produced this output:
root@box:~# certbot certonly --register-unsafely-without-email --config-dir /tmp/lets_encrypt -d box.rflm.net,rflm.net,www.rflm.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Plugins selected: Authenticator webroot, Installer None
From cffi callback <function _verify_callback at 0x7f5f8333ebf8>:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 309, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
An unexpected error occurred:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 438, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1907, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 846, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 445, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    raise SSLError(e)
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.26.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1238, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 648, in _init_le_client
    return client.Client(config, acc, authenticator, installer, acme=acme)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 247, in __init__
    acme = acme_from_config_key(config, self.account.key, self.account.regr)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 50, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 744, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1078, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1027, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)
Please see the logfiles in /var/log/letsencrypt for more details.

My web server is (include version): nginx 1.4.6-1ubuntu3.8

The operating system my web server runs on is (include version): Ubuntu 14.04.5 LTS

My hosting provider, if applicable, is: Digital Ocean

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No.


#2

Hi,

Can you please share us the log file @ /var/log/letsencrypt ?

by the way, the command to issue mutiple certificate should be
certbot certonly --register-unsafely-without-email --config-dir /tmp/lets_encrypt -d box.rflm.net -d rflm.net -d www.rflm.net

Thank you


#3

Hi @phred,

This suggests that either you’re behind a firewall that’s interfering with your connection to Let’s Encrypt, or the CDN that hosts the Let’s Encrypt API has some other problem handling your connection properly, or the local trusted CA store on your system is missing, out of date, or corrupted.

What happens if you run something like this?

curl -v https://acme-v01.api.letsencrypt.org/directory


#4

This happens, which seems to be ok. The SSL verification appears to be working fine via Curl.

# curl -v https://acme-v01.api.letsencrypt.org/directory
* Hostname was NOT found in DNS cache
*   Trying 23.38.185.196...
* Connected to acme-v01.api.letsencrypt.org (23.38.185.196) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
* 	 subject: CN=acme-v02.api.letsencrypt.org
* 	 start date: 2018-08-03 01:36:30 GMT
* 	 expire date: 2018-11-01 01:36:30 GMT
* 	 subjectAltName: acme-v01.api.letsencrypt.org matched
* 	 issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* 	 SSL certificate verify ok.
> GET /directory HTTP/1.1
> User-Agent: curl/7.35.0
> Host: acme-v01.api.letsencrypt.org
> Accept: */*
> 
< HTTP/1.1 200 OK
* Server nginx is not blacklisted
< Server: nginx
< Content-Type: application/json
< Content-Length: 658
< Replay-Nonce: WaQy_S43LjF5ZVpuYIaRHzHlUVIDS50Vy3Q7YDwzS7U
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
< Expires: Wed, 15 Aug 2018 02:13:42 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Wed, 15 Aug 2018 02:13:42 GMT
< Connection: keep-alive
< 
{
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "q7n5H-NeTPE": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host acme-v01.api.letsencrypt.org left intact
}

#5

Interesting!

How about adding the option -6 to the curl command line?

Also, how about

$ python3.4
>>> import requests
>>> requests.get("https://acme-v01.api.letsencrypt.org/directory")

#6

I copied the syntax used by ‘mailinabox’ (https://mailinabox.email/) as it appears to call things from within it’s scripts. Running from the command line generated the same errors as the ‘mailinabox’ scripts did. My intent, once I figure this out for the standalone task, is to provide feedback for the ‘mailinabox’ folk as an issue.

Heres my log from the sample command:

# cat letsencrypt.log
2018-08-14 20:13:15,566:DEBUG:certbot.main:certbot version: 0.26.1
2018-08-14 20:13:15,567:DEBUG:certbot.main:Arguments: ['--register-unsafely-without-email', '--config-dir', '/tmp/lets_encrypt', '-d', 'box.rflm.net', '-d', 'rflm.net', '-d', 'www.rflm.net']
2018-08-14 20:13:15,567:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-08-14 20:13:15,576:DEBUG:certbot.log:Root logging level set at 20
2018-08-14 20:13:15,577:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-08-14 20:13:15,578:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2018-08-14 20:13:15,635:DEBUG:certbot.plugins.selection:Multiple candidate plugins: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7f1396a525c0>
Prep: True

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7f139a0d0748>
Prep: True
2018-08-14 20:13:20,606:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7f139a0d0748> and installer None
2018-08-14 20:13:20,606:INFO:certbot.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2018-08-14 20:13:20,612:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, key=None, only_return_existing=None, agreement=None, terms_of_service_agreed=None, contact=()), new_authzr_uri=None, terms_of_service=None, uri='https://acme-v02.api.letsencrypt.org/acme/acct/40188428'), 7328dbb449058693496375abfba6f12c, Meta(creation_dt=datetime.datetime(2018, 8, 14, 0, 8, 25, tzinfo=<UTC>), creation_host='box.rflm.net'))>
2018-08-14 20:13:20,613:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2018-08-14 20:13:20,616:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2018-08-14 20:13:20,636:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 438, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1907, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 846, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 445, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    raise SSLError(e)
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.26.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1238, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 648, in _init_le_client
    return client.Client(config, acc, authenticator, installer, acme=acme)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 247, in __init__
    acme = acme_from_config_key(config, self.account.key, self.account.regr)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 50, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 744, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1078, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1027, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)
2018-08-14 20:13:20,647:ERROR:certbot.log:An unexpected error occurred:

#7

Curl, with the -6 (ipv6) option fails to connect. Which I believe is expected since I didn’t enable ipv6 networking for this ‘droplet’ on DigitalOcean.

The Python test fails much in the same way as ‘certbot’ does. Which, as you suspected, points to a broken python3.4 in Ubuntu 14.04.5. Other threads I found suggest trying to upgrade the “cryptography” package in Python but that fails, because ‘cryptography’ is owned by the OS (os delivered package).
‘Mail-in-a-box’ only installs on 14.04.5, so using a newer distribution for my case is not really an option.

# python3.4
Python 3.4.3 (default, Nov 28 2017, 16:41:13) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("https://acme-v01.api.letsencrypt.org/directory")
From cffi callback <function _verify_callback at 0x7f0e00a0d0d0>:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 309, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 438, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1907, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 846, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 445, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    raise SSLError(e)
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 
'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)


Ray Frush


#8

Well, you could try the certbot-auto script (which installs its own dependencies) or a different client.

If neither of these seems like a good option for you, I can try to ask a colleague with more expertise about the Ubuntu packaging.


#9

Except for submitting a pull request against the mailinabox project on GitHub, I don’t have any reasonable way to change how the tool calls certbot. If you could talk to your Ubuntu expert, that would be great. They can probably quickly come up with “the answer” or confirm that this is a problem with no solution for the plain certbot script.

Thanks!


Ray Frush


#10

Have you done anything unusual with your Python packages on this system such as installing anything yourself via pip or from source?


#11

Also, do you know why these packages are in /usr/local/lib rather than /usr/lib?


#12

Schoen-
Good catch there on the /usr/lib vs /usr/local/lib. It appears that ‘mail-in-a-box’ is delivering a separate instance of python3. I still haven’t figured out the systems’s python (/usr/bin/python3) called by the certbot script is pulling up /usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py instead of /usr/lib/python3/dist-packages/OpenSSL/SSL.py

For now, I’m guessing it’s some artifact of the ‘mail-in-a-box’ setup. I’ll hassle them.

Thanks again for catching that detail. That’s very helpful.


#13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.