Certbot hangs on 'Generating key'

ABAJosh · October 21, 2019, 9:24pm

I’m running HAProxy for load balancing and SSL termination. We host ~400 domains, and are in the process of migrating to single site per certificate LetsEncrypt certs from 4 SAN certs, each hosting 100 or so domains. At this time, both the old certs and the new are hosted on the same machine, which has 4 public IP addresses. I’m using an ACL in HAProxy to direct requests to /.well-known/acme-challenge/ to an Apache server on the HAproxy box, listening on port 8080. I have a script to get domain names it runs the command below.

The basic certbot syntax I’m using is:
certbot certonly -n --deploy-hook "/etc/letsencrypt/scripts/cptohaproxy.sh $x" --agree-tos --http-01-port 8080 -m support@xxxx.org --webroot -w /var/www/html/ -d $x -d www.$x
(where $x is a domain name and cptohaproxy.sh cats together the resulting cert in an HAProxy friendly format).

Thus far I’ve gotten certs for about 250 domains. A few examples are below.
murderbooks.com
mysterylovescompany.com
mystgalaxy.com
nantucketbookpartners.com
nebookfair.com
newtownbookshop.com
nicolasbooks.com
northshire.com
northtownbooks.com

novelmemphis.com
nowherebookshop.com
octaviabooks.com
onemorepagebooks.com
opendoor-bookstore.com
overthemoonbookstore.com
page158books.com

This morning, certbot began hanging when getting new certs. Forced renewals still work, as do dry runs.
The basic output is:
certbot certonly -n --deploy-hook “/etc/letsencrypt/scripts/cptohaproxy.sh $x” --agree-tos --http-01-port 8080 -m support@xxxxxx.org --webroot -w /var/www/html/ -d $x
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
(full debug output is below)

My web server is (include version): Apache/2.4.18 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 16.04

I can login in root. I’m a reasonably advanced user.

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.23.0.

Verbose Output. After this point it hangs indefinitely.
certbot certonly -vvv -n --deploy-hook “/etc/letsencrypt/scripts/cptohaproxy.sh store.wellesleybooks.com” --agree-tos --http-01-port 8080 -m support@xxxxxxxx.org --webroot -w /var/www/html/ -d store.wellesleybooks.com
Root logging level set at -10
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requested authenticator webroot and installer None
Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7f99d473cf90>
Prep: True
Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7f99d473cf90> and installer None
Plugins selected: Authenticator webroot, Installer None
Picked account: <Account(RegistrationResource(body=Registration(status=u’valid’, terms_of_service_agreed=None, contact=(u’mailto:support@xxxxxxx.org’,), agreement=u’https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf’, key=JWKRSA(key=
<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7f99d9891a90>)>)), uri=u’https://acme-v01.api.letsencrypt.org/acme/reg/69068921’, new_authzr_uri=u’https://acme-v01.api.letsencrypt.org/acme/new-authz’, terms_of_service=u’https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf’), a1aad3ff76bd944078fec8553a2dbcf1, Meta(creation_host=u’localhost’, creation_dt=datetime.datetime(2019, 10, 11, 19, 48, 26, tzinfo=)))>
Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
“GET /directory HTTP/1.1” 200 658
Received response:
HTTP 200
Content-Length: 658
Strict-Transport-Security: max-age=604800
Server: nginx
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Date: Mon, 21 Oct 2019 21:19:21 GMT
X-Frame-Options: DENY
Content-Type: application/json
Replay-Nonce: 0002i-EJf9IQFgGf3brhlgqCVP7jkRUQg9NZVBHEoRtroIQ

{
  "hMVTQkTl2mA": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
}
Obtaining a new certificate
Generating key (2048 bits): /etc/letsencrypt/keys/0428_key-certbot.pem

_az · October 21, 2019, 9:28pm

Could you try these and let us know whether they complete successfully?

openssl prime -generate -bits 2048
openssl genrsa 2048

ABAJosh · October 21, 2019, 9:30pm

Both commands complete successfully. I apologize I forgot to mention in the original post that I'd tried generating a self-signed cert, with the idea that something might be up with openssl.

-Josh

_az · October 21, 2019, 9:37pm

Tbh, reading through Certbot, it seems that “Generating key” message is emitted after the key is already generated and written to disk. So it might be a red herring.

Are you able to run Certbot with strace and see what syscalls it is getting stuck on when it hangs?

apt -y install strace
strace -ff certbot certonly -vvv -n \
--deploy-hook "/etc/letsencrypt/scripts/cptohaproxy.sh store.wellesleybooks.com" \
--http-01-port 8080 --webroot -w /var/www/html/ \
-d store.wellesleybooks.com

mnordhoff · October 21, 2019, 9:43pm

....But use regular ASCII " quotes, not fancy Unicode quotes.

_az · October 21, 2019, 9:44pm

I wish Discourse detected code blocks automatically.

ABAJosh · October 21, 2019, 10:09pm

Yeah, even the ‘preformatted text’ doesn’t work all that well. Still, I’m really grateful for everyone’s help here!

Here’s a link to the full strace output. Let me know if I screwed up the command. I just put 'strace -ff ’ in front of the previous command I’d used to dodge any funky unicode issues…

gist.github.com

https://gist.github.com/ABAJosh/c2ebec4b424de2c2220cbcbae3df8576

certbot.strace.txt

execve("/usr/bin/certbot", ["certbot", "certonly", "-vvv", "-n", "--deploy-hook", "/etc/letsencrypt/scripts/cptohap"..., "--agree-tos", "--http-01-port", "8080", "-m", "support@bookweb.org", "--webroot", "-w", "/var/www/html/", "-d", "store.wellesleybooks.com", ...], [/* 22 vars */]) = 0
brk(NULL)                               = 0x1b0c000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=30743, ...}) = 0
mmap(NULL, 30743, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd0d2044000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3

This file has been truncated. show original

_az · October 21, 2019, 10:17pm

if I read that right, it got to the last futex, you decided it was hung, and then you killed it?

What's that one log line it wrote to the log file after "Generating key"? It should start with:

2019-10-21 21:58:45,682:DEBUG:ce

Is it "Creating CSR:" ?

ABAJosh · October 21, 2019, 10:33pm

I confess that I did kill it. It doesn’t look like we got as far as a CSR… The last lines in the log are:
2019-10-21 21:58:45,389:INFO:certbot.main:Obtaining a new certificate
2019-10-21 21:58:45,682:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0431_key-certbot.pem

Just to be sure I checked disk space, and there’s plenty free, including in /tmp…

_az · October 21, 2019, 10:43pm

Man, I just noticed that your Certbot version is completely ancient. No wonder things haven’t been making much sense - it’s doing the ACME v1 worflow.

Sorry, this should have been the first step. Could you please upgrade Certbot from the Ubuntu PPA: https://certbot.eff.org/lets-encrypt/ubuntuxenial-apache

You might need to also use --server https://acme-v02.api.letsencrypt.org/directory , though I think Certbot should start using that on its own.

If the issue persists after all that, could you re-do the strace?

ABAJosh · October 21, 2019, 10:55pm

Oh for crying out loud! I accidentally installed Certbot without adding the PPA. I added the PPA, upgraded to Certbot 0.31.0, and everything went smoothly. I didn’t even need to specify the server.

That’s just downright embarrassing after all the other work I’ve put into this project. Ah well, I guess it’s a lesson for everyone else, and yet another reminder to me: just because a distro has a package in its repos doesn’t mean it’s a vaguely recent one.

Thanks for taking so much time to work through this with me @_az! You’ve been really generous with your knowledge.

Perhaps the most amazing thing is that I was able to get ~200 certs before the whole thing exploded…

system · November 20, 2019, 11:06pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Using certbot with --renew-hook hangs Server	1	2337	June 19, 2017
Help automatically renew certs with haproxy Server	2	2632	December 18, 2016
Certbot hangs. So does Snappy Help	3	1224	January 28, 2022
Server Changed and DNS mapped with New IP Help	6	600	December 10, 2020
Renewal Error - This was working 1 month ago with a dry run Help	2	1181	October 31, 2016

Certbot hangs on 'Generating key'

Related topics