Letsencrypt docker image not starting with cryptical python error

aschrijver · March 13, 2018, 6:59am

My domain is: innercircles.community

I ran this command:

note: email removed

sudo docker run -it \
    --name letsencrypt \
    --volumes-from nginx-tmp \
    -v /home/alpine/logs:/var/log/letsencrypt \
    quay.io/letsencrypt/letsencrypt \
    certonly \
    --agree-tos \
    --webroot \
    --webroot-path /usr/share/nginx/html \
    -m <my_email> \
    -d innercircles.community

It produced this output:

Warning: This Docker image will soon be switching to Alpine Linux.
You can switch now using the certbot/certbot repo on Docker Hub.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
An unexpected error occurred:
SystemError: error return without exception set
Please see the logfiles in /var/log/letsencrypt for more details.

The letsencrypt.log contains the following:

2018-03-13 06:32:01,316:DEBUG:certbot.main:certbot version: 0.23.0.dev0
2018-03-13 06:32:01,316:DEBUG:certbot.main:Arguments: ['--agree-tos', '--webroot', '--webroot-path', '/usr/share/nginx/html', '-m', '<my_email>', '-d', 'innercircles.community']
2018-03-13 06:32:01,316:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-03-13 06:32:01,331:DEBUG:certbot.log:Root logging level set at 20
2018-03-13 06:32:01,331:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-03-13 06:32:01,332:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2018-03-13 06:32:01,338:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x6826f815a910>
Prep: True
2018-03-13 06:32:01,338:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x6826f815a910> and installer None
2018-03-13 06:32:01,338:INFO:certbot.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2018-03-13 06:32:01,445:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2018-03-13 06:32:01,447:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2018-03-13 06:32:01,451:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/opt/certbot/venv/bin/certbot", line 11, in <module>
    load_entry_point('certbot', 'console_scripts', 'certbot')()
  File "/opt/certbot/src/certbot/main.py", line 1266, in main
    return config.func(config, plugins)
  File "/opt/certbot/src/certbot/main.py", line 1141, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/opt/certbot/src/certbot/main.py", line 635, in _init_le_client
    acc, acme = _determine_account(config)
  File "/opt/certbot/src/certbot/main.py", line 514, in _determine_account
    config, account_storage, tos_cb=_tos_cb)
  File "/opt/certbot/src/certbot/client.py", line 164, in register
    acme = acme_from_config_key(config, key)
  File "/opt/certbot/src/certbot/client.py", line 46, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/opt/certbot/src/acme/acme/client.py", line 717, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/opt/certbot/src/acme/acme/client.py", line 1040, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/opt/certbot/src/acme/acme/client.py", line 989, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/urllib3/connection.py", line 314, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 274, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 400, in verify_mode
    _verify_callback
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1061, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/opt/certbot/venv/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 337, in __init__
    "int (*)(int, X509_STORE_CTX *)", wrapper)
SystemError: error return without exception set
2018-03-13 06:32:01,453:ERROR:certbot.log:An unexpected error occurred:

My web server is (include version): nginx 1.13.9 (latest) run in a temporary docker container (its just to get the certificates)

The operating system my web server runs on is (include version): Alpine-3.7-r2-Hardened-EC2

My hosting provider, if applicable, is: AWS EC2, a t2.micro instance

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no, bash shell via SSH

When I did this same operation previously, but on a RancherOS instance, everything worked like a charm.

PS. Running docker with sudo is not the issue. I also tried with doing sudo -i first.

aschrijver · March 13, 2018, 7:49am

I can confirm that the procedure I followed should work, as I’ve created a t2.micro running RancherOS again and successfully created the certificate. So the problem seems to be specific to running this on alpine!

schoen · March 13, 2018, 5:08pm

So I think there was a problem before with some security modules complaining about self-modifying code and the interface that Python uses to load OpenSSL (which effectively uses self-modifying code, akin to a JIT). I haven’t been able to find the earlier forum threads about this and I don’t remember the details. Do you know which particular Linux security tools the Alpine “hardened” image incorporates?

aschrijver · March 13, 2018, 9:38pm

Thanks for your response @schoen! I configured the alpine os to use openssl in the initial setup-alpine script, but I just now tried to find where the AMI is actually coming from.
The stupid thing on AWS is that there is no source information at all mentioned in the AMI list (other than Alpine-3.7-r2-Hardened-EC2 and ami-a96ff8c6), like you have for instance on Docker hub. I thought it was ‘official’, but it seems to be created from this github repo + file:

github.com

mcrute/alpine-ec2-ami/blob/master/make_ami.sh

#!/bin/sh
# vim:set ts=4:

set -eu

: ${ALPINE_RELEASE:="3.7"} # not tested against edge
: ${APK_TOOLS_URI:="https://github.com/alpinelinux/apk-tools/releases/download/v2.8.0/apk-tools-2.8.0-x86_64-linux.tar.gz"}
: ${APK_TOOLS_SHA256:="da21cefd2121e3a6cd4e8742b38118b2a1132aad7f707646ee946a6b32ee6df9"}
: ${ALPINE_KEYS:="http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/alpine-keys-2.1-r1.apk"}
: ${ALPINE_KEYS_SHA256:="7b2d1e9a00324c8eee49785dc22355be02534201e77473ba9762027e1a475cc7"}

die() {
	printf '\033[1;31mERROR:\033[0m %s\n' "$@" >&2  # bold red
	exit 1
}

einfo() {
	printf '\n\033[1;36m> %s\033[0m\n' "$@" >&2  # bold cyan
}

This file has been truncated. show original

aschrijver · March 16, 2018, 10:17am

I am receiving a somewhat similar error on a different instance running the same Alpine OS AMI, but instead trying to run nginx-proxy (actually using 2 separate containers docker-gen and docker-letsencrypt-nginx-proxy-companion):

2018/03/16 10:05:02 Received event start for container 7b720d6df977
2018/03/16 10:05:02 Received event start for container 3f7bb00ca186
2018/03/16 10:05:17 Debounce minTimer fired
2018/03/16 10:05:17 Generated '/app/letsencrypt_service_data' from 5 containers
2018/03/16 10:05:17 Running '/app/update_certs'
/etc/nginx/certs/my.example.com /app
Reloading nginx docker-gen (using separate container nginx-gen)...
Reloading nginx (using separate container nginx)...
Creating/renewal my.example.com certificates... (my.example.com)
2018-03-16 10:05:18,000:INFO:simp_le:1538: Retrieving Let's Encrypt latest Terms of Service.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1558, in main
    return main_with_exceptions(cli_args)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 1539, in main_with_exceptions
    args.tos_sha256 = get_le_tos_hash(LE_PRODUCTION_URI)
  File "/usr/lib/python2.7/site-packages/simp_le.py", line 229, in get_le_tos_hash
    le_directory = requests.get(le_uri).json()
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/lib/python2.7/site-packages/urllib3/connection.py", line 314, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/usr/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 274, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/usr/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 400, in verify_mode
    _verify_callback
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1061, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 337, in __init__
    "int (*)(int, X509_STORE_CTX *)", wrapper)
SystemError: error return without exception set

Unhandled error has happened, traceback is above

I will now check if the new way of creating certs using new certbot/certbot docker image yields the same results. I’ve also notified the author of the Alpine AMI who is prepared to help, if the error is somehow related to the configuration of the Alpine hardened kernel: Issue generating letsencrypt certificates from hardened kernel.

aschrijver · March 16, 2018, 10:46am

The new certbot/cerbot docker image has the same issue:

Starting as described in Running with docker with:

sudo docker run -it --rm --name certbot \
            -v "/etc/letsencrypt:/etc/letsencrypt" \
            -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
            certbot/certbot certonly

Yields:

alpine-srv:~/log$ cat letsencrypt.log
2018-03-16 10:40:28,451:DEBUG:certbot.main:certbot version: 0.22.0
2018-03-16 10:40:28,451:DEBUG:certbot.main:Arguments: []
2018-03-16 10:40:28,451:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-03-16 10:40:28,465:DEBUG:certbot.log:Root logging level set at 20
2018-03-16 10:40:28,466:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-03-16 10:40:28,467:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2018-03-16 10:40:28,554:DEBUG:certbot.plugins.selection:Multiple candidate plugins: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7422884a97d0>
Prep: True

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7422884cdb10>
Prep: True
2018-03-16 10:40:34,069:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7422884cdb10> and installer None
2018-03-16 10:40:34,070:INFO:certbot.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2018-03-16 10:40:42,385:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2018-03-16 10:40:42,387:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2018-03-16 10:40:42,396:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 11, in <module>
    load_entry_point('certbot', 'console_scripts', 'certbot')()
  File "/opt/certbot/src/certbot/main.py", line 1266, in main
    return config.func(config, plugins)
  File "/opt/certbot/src/certbot/main.py", line 1141, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/opt/certbot/src/certbot/main.py", line 635, in _init_le_client
    acc, acme = _determine_account(config)
  File "/opt/certbot/src/certbot/main.py", line 514, in _determine_account
    config, account_storage, tos_cb=_tos_cb)
  File "/opt/certbot/src/certbot/client.py", line 164, in register
    acme = acme_from_config_key(config, key)
  File "/opt/certbot/src/certbot/client.py", line 46, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/opt/certbot/src/acme/acme/client.py", line 718, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/opt/certbot/src/acme/acme/client.py", line 1041, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/opt/certbot/src/acme/acme/client.py", line 990, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python2.7/site-packages/urllib3/connection.py", line 314, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/usr/local/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 274, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/usr/local/lib/python2.7/site-packages/urllib3/contrib/pyopenssl.py", line 400, in verify_mode
    _verify_callback
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1061, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/usr/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 337, in __init__
    "int (*)(int, X509_STORE_CTX *)", wrapper)
SystemError: error return without exception set
2018-03-16 10:40:42,399:ERROR:certbot.log:An unexpected error occurred:
alpine-srv:~/log$

Patches · March 16, 2018, 3:20pm

Does the host’s kernel log anything when certbot crashes?

sudo dmesg | tail

aschrijver · March 16, 2018, 3:40pm

Thx @Patches! No, unfortunately nothing showing there…

Patches · March 16, 2018, 5:34pm

It’s weird that it doesn’t log anything but this is probably PaX.

I built a tchollingsworth/certbot-pax container on top of the official certbot one that marks the python executable with the appropriate xattr to exempt it from PaX. Can you try it and see if it works for you? If so, I’ll submit the change upstream.

(You should examine the Dockerfile first to make sure I’m not stealing your private keys. ) EDIT: I forgot the Dockerfile doesn’t show up unless you link it to GitHub or whatever. If you don’t trust me, the Dockerfile is just:

FROM certbot/certbot

RUN apk add --no-cache --virtual .pax-deps attr \
    && setfattr -n user.pax.flags -v "emr" /usr/local/bin/python2.7 \
    && apk del .pax-deps

aschrijver · March 16, 2018, 6:05pm

Ha ha, thanks again @Patches
I am afraid the results are exactly the same

I may have to go for a different OS temporarily, need my build server (but can still spin up a small t2.micro to test alpine stuff)

Patches · March 16, 2018, 6:18pm

The official Alpine Linux images use PaX, but I couldn’t reproduce this issue with them. I guess if you really wanted to get to the bottom of this you would have to contact the author of this image and figure out what changes they are making that could cause this.

One other thing you could try is the using the certbot apk package directly on the host instead of using a Docker container. It’s available in the community repository. It may be just as broken as the Docker container with your unofficial image though.

aschrijver · March 16, 2018, 6:21pm

I already contacted the author and he is willing to help if the issue persists. The AMI he is creating may well become the official one, once it has matured a bit. I will notify him in this github issue:
https://github.com/mcrute/alpine-ec2-ami/issues/4#issuecomment-373239985

system · April 15, 2018, 6:22pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.