Letsencrypt cli hangs on certificate request


#1

Please fill out the fields below so we can help you better.

My domain is:tab.heretical-web.xyz

I ran this command:letsencrypt --webroot certonly -w /var/www/letsencrypt/ -d tab.heretical-web.xyz

It produced this output:None, it just hung with no output

My operating system is (include version):Ubuntu 16.04

My web server is (include version):nginx 1.10.0

My hosting provider, if applicable, is:Linode

I can login to a root shell on my machine (yes or no, or I don’t know):
yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):no

If i run either:

$ sudo letsencrypt --agree-tos --email kiwiheretic@myself.com --staging --webroot certonly -w /var/www/letsencrypt/ -d tab.heretical-web.xyz

or the command shown in questions from a root prompt the session just hangs until ctrl-c is pressed.

from /var/log/letsencrypt/letsencrypt.log

2017-02-06 01:46:03,726:DEBUG:letsencrypt.cli:Root logging level set at 30
2017-02-06 01:46:03,728:INFO:letsencrypt.cli:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-02-06 01:46:03,728:DEBUG:letsencrypt.cli:letsencrypt version: 0.4.1
2017-02-06 01:46:03,728:DEBUG:letsencrypt.cli:Arguments: ['--webroot', '-w', '/var/www/letsencrypt/', '-d', 'tab.heretical-web.xyz']
2017-02-06 01:46:03,729:DEBUG:letsencrypt.cli:Discovered plugins: PluginsRegistry(PluginEntryPoint#webroot,PluginEntryPoint#null,PluginEntryPoint#manual,PluginEntryPoint#standalone)
2017-02-06 01:46:03,734:DEBUG:letsencrypt.cli:Requested authenticator webroot and installer None
2017-02-06 01:46:03,735:DEBUG:letsencrypt.plugins.webroot:Creating root challenges validation dir at /var/www/letsencrypt/.well-known/acme-challenge
2017-02-06 01:46:03,735:DEBUG:letsencrypt.display.ops:Single candidate plugin: * webroot
Description: Webroot Authenticator
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = letsencrypt.plugins.webroot:Authenticator
Initialized: <letsencrypt.plugins.webroot.Authenticator object at 0x7f99d5b7abd0>
Prep: True
2017-02-06 01:46:03,736:DEBUG:letsencrypt.cli:Selected authenticator <letsencrypt.plugins.webroot.Authenticator object at 0x7f99d5b7abd0> and installer None

My nginx configuration is:

# configuration of the server
server {
    # the port your site will be served on
    listen      80;
    # the domain name it will serve for
    server_name tab.heretical-web.xyz; # substitute your machine's IP address or FQDN

        include acme_challenge.conf;
    charset     utf-8;
    sendfile on;
    # max upload size
    client_max_body_size 75M;   # adjust to taste
    keepalive_timeout 0;


    # Finally, send all non-media requests to the Django server.
    location / {
        proxy_pass http://localhost:8006;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
    }
}

The contents of acme_challenge.conf is:

#############################################################################
# Configuration file for Let's Encrypt ACME Challenge location
# This file is already included in listen_xxx.conf files.
# Do NOT include it separately!
#############################################################################
#
# This config enables to access /.well-known/acme-challenge/xxxxxxxxxxx
# on all our sites (HTTP), including all subdomains.
# This is required by ACME Challenge (webroot authentication).
# You can check that this location is working by placing ping.txt here:
# /var/www/letsencrypt/.well-known/acme-challenge/ping.txt
# And pointing your browser to:
# http://xxx.domain.tld/.well-known/acme-challenge/ping.txt
#
# Sources:
# https://community.letsencrypt.org/t/howto-easy-cert-generation-and-renewal-with-nginx/3491
#
#############################################################################

# Rule for legitimate ACME Challenge requests (like /.well-known/acme-challenge/xxxxxxxxx)
# We use ^~ here, so that we don't check other regexes (for speed-up). We actually MUST cancel
# other regex checks, because in our other config files have regex rule that denies access to files with dotted names.
location ^~ /.well-known/acme-challenge/ {

    # Set correct content type. According to this:
    # https://community.letsencrypt.org/t/using-the-webroot-domain-verification-method/1445/29
    # Current specification requires "text/plain" or no content header at all.
    # It seems that "text/plain" is a safe option.
    default_type "text/plain";

    # This directory must be the same as in /etc/letsencrypt/cli.ini
    # as "webroot-path" parameter. Also don't forget to set "authenticator" parameter
    # there to "webroot".
    # Do NOT use alias, use root! Target directory is located here:
    # /var/www/common/letsencrypt/.well-known/acme-challenge/
    root         /var/www/letsencrypt;
}

# Hide /acme-challenge subdirectory and return 404 on all requests.
# It is somewhat more secure than letting Nginx return 403.
# Ending slash is important!
location = /.well-known/acme-challenge/ {
    return 404;
}

##################################################

Any ideas why this is hanging?

Thanks in advance.


#2

interesting

try running with more verbose mode and paste the output

-v, --verbose This flag can be used multiple times to incrementally
increase the verbosity of output, e.g. -vvv. (default:
-2)


#3

Ok I tried.

$ sudo letsencrypt -vvv --email kiwiheretic@myself.com --agree-tos --webroot certonly -w /var/www/letsencrypt/ -d tab.heretical-web.xyz

It gives this output in a dialog box, hence the bad formatting…

‘–agree-tos’, ‘–webroot’, ‘-w’, ‘/var/www/letsencrypt/’, ‘-d’, x
x ‘tab.heretical-web.xyz’] x
x Discovered plugins: x
x PluginsRegistry(PluginEntryPoint#webroot,PluginEntryPoint#null,Plugi x
x nEntryPoint#manual,PluginEntryPoint#standalone) x
x Requested authenticator webroot and installer None x
x Creating root challenges validation dir at x
x /var/www/letsencrypt/.well-known/acme-challenge x
x Single candidate plugin: * webroot x
x Description: Webroot Authenticator x
x Interfaces: IAuthenticator, IPlugin x
x Entry point: webroot = letsencrypt.plugins.webroot:Authenticator x
x Initialized: <letsencrypt.plugins.webroot.Authenticator object at x
x 0x7f8f58d0df50> x
x Prep: True x
x Selected authenticator <letsencrypt.plugins.webroot.Authenticator x
x object at 0x7f8f58d0df50> and installer None x
x Sending GET request to x
x https://acme-v01.api.letsencrypt.org/directory. args: (), kwargs: {} x
x Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org

Thats where it stalls


#4

did you manage to verify HTTPS connectivity?


#5

try running wget https://acme-v01.api.letsencrypt.org/directory

if nothing happens

ping acme-v01.api.letsencrypt.org

dig acme-v01.api.letsencrypt.org

looking like a core connectivity issue - not sure if certbot times out if it can’t reach home base (https://acme-v01.api.letsencrypt.org/directory)


#6

That works from curl

splat@ubuntu:~$ curl https://acme-v01.api.letsencrypt.org/directory
{
“key-change”: “https://acme-v01.api.letsencrypt.org/acme/key-change”,
“new-authz”: “https://acme-v01.api.letsencrypt.org/acme/new-authz”,
“new-cert”: “https://acme-v01.api.letsencrypt.org/acme/new-cert”,
“new-reg”: “https://acme-v01.api.letsencrypt.org/acme/new-reg”,
“revoke-cert”: “https://acme-v01.api.letsencrypt.org/acme/revoke-cert
}splat@ubuntu:~$

not sure why the letsencrypt client fails tho


#7

So have we concluded that letsencrypt doesn’t work on Linode?


#8

No. Certbot can run on Ubuntu 16.04 installed on Linode. A VPS provider with a standard supported OS would probably have to go out of their way to break Certbot or any other Let’s Encrypt client.

Edit:

Connectivity issues between one or more of Linode’s data centers and Akamai (Let’s Encrypt’s CDN) are possible, though unusual. The Internet is never perfectly reliable, but good ISPs try. I’m sure they would be eager to fix any issues.


#9

Does letsencrypt actually have a timeout? I usually have to control-c out of it. Just wondering if there is a way to see if timeouts are actually happening by reducing it to a more reasonable value.


#10

I don’t suppose “curl https://acme-v01.api.letsencrypt.org/directory” took several seconds to run?


#11

No, it was almost instantaneous. Tried it again just now.


#12

Yeah, i just checked too, I don’t see any obvious connectivity issues between the Linode data center your VPS is apparently in and Let’s Encrypt. :confounded:


#13

Hmmm, so are we to conclude something else is causing it to hang? Can you tell from your end where it failed to proceed to the next step in the network conversation?


#14

I don’t think Certbot sets a timeout on its HTTPS requests, which it probably should do. Thanks for the suggestion!

I’m a little stumped about what could be the problem. It does look like things are hanging at the connect step rather than the request step.

You can try to reproduce a little more closely what Let’s Encrypt is doing with:

python -c 'import requests; print requests.get("https://acme-v01.api.letsencrypt.org/directory").text'

Also, I notice that your command is letsencrypt, not certbot. Around springtime of last year, the client was renamed to certbot, which means you are probably running a pretty old version. I can’t think of any reason an old version would have trouble connecting, but it’s probably a good idea to upgrade anyhow.


#15

When I typed in that python statement of yours its hanging also. Will look at certbot upgrade when I get a chance.


#16

Wild theory: Maybe IPv6 is misconfigured and broken; due to the vagaries of source address selection, curl prefers to use IPv64, but requests prefers to use IPv6.

Using curl -v can (dis)prove it.

Edit 45 minutes later: “IPv4”. Not “IPv64”. :crying_cat_face:


#17

Here is the output to curl -v

splat@ubuntu:~/devel/tab$ curl -v https://acme-v01.api.letsencrypt.org/directory

  • Trying 2600:1402:a:29f::3d5…
  • Trying 23.78.168.127…
  • Connected to acme-v01.api.letsencrypt.org (23.78.168.127) port 443 (#0)
  • found 173 certificates in /etc/ssl/certs/ca-certificates.crt
  • found 692 certificates in /etc/ssl/certs
  • ALPN, offering http/1.1
  • SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
  •    server certificate verification OK
    
  •    server certificate status verification SKIPPED
    
  •    common name: *.api.letsencrypt.org (matched)
    
  •    server certificate expiration date OK
    
  •    server certificate activation date OK
    
  •    certificate public key: RSA
    
  •    certificate version: #3
    
  •    subject: CN=*.api.letsencrypt.org,O=INTERNET SECURITY RESEARCH GROUP,L=Mountain View,ST=California,C=US
    
  •    start date: Fri, 26 Jun 2015 17:05:45 GMT
    
  •    expire date: Mon, 25 Jun 2018 17:05:45 GMT
    
  •    issuer: C=US,O=IdenTrust,OU=TrustID Server,CN=TrustID Server CA A52
    
  •    compression: NULL
    
  • ALPN, server accepted to use http/1.1

GET /directory HTTP/1.1
Host: acme-v01.api.letsencrypt.org
User-Agent: curl/7.47.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: application/json
< Content-Length: 352
< Boulder-Request-Id: EtahFL3yaJxeQWFOsbQABN6KJ1w-fwLl2g4X1XXEyG4
< Replay-Nonce: KHG_Z8Kay6d-8nNLnxz6PyKgJz3QTJuzcHdrPiIjz5Q
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
< Expires: Sat, 11 Feb 2017 00:27:00 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Sat, 11 Feb 2017 00:27:00 GMT
< Connection: keep-alive
<
{
“key-change”: “https://acme-v01.api.letsencrypt.org/acme/key-change”,
“new-authz”: “https://acme-v01.api.letsencrypt.org/acme/new-authz”,
“new-cert”: “https://acme-v01.api.letsencrypt.org/acme/new-cert”,
“new-reg”: “https://acme-v01.api.letsencrypt.org/acme/new-reg”,
“revoke-cert”: “https://acme-v01.api.letsencrypt.org/acme/revoke-cert


#18

Great, can you also try the same command with the -6 flag? If that hangs, you know the problem is with IPv6 routing somewhere.


#19

Ok, here was the result

$ curl -6 -v https://acme-v01.api.letsencrypt.org/directory

  • Trying 2600:1402:a:29f::3d5…
  • connect to 2600:1402:a:29f::3d5 port 443 failed: Connection timed out
  • Trying 2600:1402:a:2a2::3d5…
  • After 85027ms connect time, move on!
  • connect to 2600:1402:a:2a2::3d5 port 443 failed: Connection timed out
  • Failed to connect to acme-v01.api.letsencrypt.org port 443: Connection timed out
  • Closing connection 0
    curl: (7) Failed to connect to acme-v01.api.letsencrypt.org port 443: Connection timed out

For some reason that timeout information isn’t getting to the letsencrypt cli client maybe?


#20

I think the letsencrypt cli client just doesn’t have a timeout, or has a much longer timeout than curl.

Probably the simplest fix is to disable IPv6 on your host. Alternately, you can file a support ticket with your hosting provider saying you are having an IPv6 routing problem.