Empty response from 'POST /acme/new-authz' with one particular cert


#1

It seems like maybe this problem is on the LetsEncrypt server side, because there is demonstrably connectivity (the initial GET and HEAD requests work), and no error message is returned in the HTTP response, which I assume is not how the protocol is supposed to work.

https://crt.sh/?q=www.greensfelder.com shows that a cert was issued a couple weeks ago when this started, but it never arrived (or if it did, certbot failed to store it).

This same client machine holds a number of certificates, and an identical command is used for each one. The problem only occurs - and consistently occurs during my attempts over the last two weeks - with this one domain. The workflow has been working fine for a long time, and the same workflow on the same machine has successfuly issued and renewed certificates for other domains both before and after this. I checked the web server logs and there was no attempt to do the normal http challenge. This is a Debian 8 server with all packages up to date, and certbot from jessie-backports. Has anyone come across a similar thing? Thanks for any pointers.

The log below is from an attempt to issue a new cert; I tried this after renewing also failed in the same way.

certbot certonly --non-interactive --agree-tos --email re@dacted --force-renewal --webroot -w /var/lib/certbot/greensfelder -d www.greensfelder.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Renewing an existing certificate
An unexpected error occurred:
ZeroReturnError
Please see the logfiles in /var/log/letsencrypt for more details.

2018-07-30 09:39:51,534:DEBUG:certbot.main:Root logging level set at 20
2018-07-30 09:39:51,534:INFO:certbot.main:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-07-30 09:39:51,534:DEBUG:certbot.main:certbot version: 0.10.2
2018-07-30 09:39:51,534:DEBUG:certbot.main:Arguments: [’–non-interactive’, ‘–agree-tos’, ‘–email’, ‘re@dacted’, ‘–force-renewal’,
‘–webroot’, ‘-w’, ‘/var/lib/certbot/greensfelder’, ‘-d’, ‘www.greensfelder.com’]
2018-07-30 09:39:51,535:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#webroot,PluginEntryPoint#null,PluginEntryPoint#manual,Plu
ginEntryPoint#standalone)
2018-07-30 09:39:51,535:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2018-07-30 09:39:51,537:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7f9eb0bdaf50>
Prep: True
2018-07-30 09:39:51,537:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7f9eb0bdaf50> and insta
ller None
2018-07-30 09:39:51,586:DEBUG:certbot.main:Picked account: <Account(8cd0f1c552599e66524ce0bf01701b70)>
2018-07-30 09:39:51,587:DEBUG:root:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2018-07-30 09:39:51,592:INFO:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2018-07-30 09:39:51,707:DEBUG:urllib3.connectionpool:“GET /directory HTTP/1.1” 200 658
2018-07-30 09:39:51,708:DEBUG:acme.client:Received response:
HTTP 200
content-length: 658
strict-transport-security: max-age=604800
expires: Mon, 30 Jul 2018 09:39:51 GMT
server: nginx
connection: keep-alive
pragma: no-cache
cache-control: max-age=0, no-cache, no-store
date: Mon, 30 Jul 2018 09:39:51 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: REDACTED

{
“J9R-gXjDQ-s”: “Adding random entries to the directory”,
“key-change”: “https://acme-v01.api.letsencrypt.org/acme/key-change”,
“meta”: {
“caaIdentities”: [
letsencrypt.org
],
“terms-of-service”: “https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf”,
“website”: “https://letsencrypt.org
},
“new-authz”: “https://acme-v01.api.letsencrypt.org/acme/new-authz”,
“new-cert”: “https://acme-v01.api.letsencrypt.org/acme/new-cert”,
“new-reg”: “https://acme-v01.api.letsencrypt.org/acme/new-reg”,
“revoke-cert”: “https://acme-v01.api.letsencrypt.org/acme/revoke-cert
}
2018-07-30 09:39:52,148:INFO:certbot.main:Obtaining a new certificate
2018-07-30 09:39:52,148:DEBUG:root:Requesting fresh nonce
2018-07-30 09:39:52,148:DEBUG:root:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-authz.
2018-07-30 09:39:52,178:DEBUG:urllib3.connectionpool:“HEAD /acme/new-authz HTTP/1.1” 405 0
2018-07-30 09:39:52,179:DEBUG:acme.client:Received response:
HTTP 405
content-length: 91
pragma: no-cache
expires: Mon, 30 Jul 2018 09:39:52 GMT
server: nginx
connection: keep-alive
allow: POST
cache-control: max-age=0, no-cache, no-store
date: Mon, 30 Jul 2018 09:39:52 GMT
content-type: application/problem+json
replay-nonce: REDACTED

2018-07-30 09:39:52,179:DEBUG:acme.client:Storing nonce: REDACTED
2018-07-30 09:39:52,180:DEBUG:acme.client:JWS payload:
{
“identifier”: {
“type”: “dns”,
“value”: “www.greensfelder.com
},
“resource”: “new-authz”
}
2018-07-30 09:39:52,182:DEBUG:root:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz:
{
“header”: {
“alg”: “RS256”,
“jwk”: {
“e”: “AQAB”,
“kty”: “RSA”,
“n”: “REDACTED”
}
},
“protected”: “REDACTED”,
“payload”: “REDACTED”,
“signature”: “cPOtnAePI8JDnWMg47RBfZtsmMpfHcfy3FAeUZ5nvhsZ_uP7akNFT5ATm3P_hGe72t1eE_iDD4LJho_JvKb-uvzk3tTcCE7ed_nLTulzocF3GbgAqUb7qXJgWKcg_9h3tA5OuCGOJ
ku5zqCOdXyixRisvUjLoRurFt2wKRzCmHju2ChDHW-n6A0WQf6cIarLAT_k3PEI7nUWhHTYkGrzMGCBuP8y11JU5reS-Q_t1t3O_pXAyr52b94lIAJHRASwFgJBC3p94EuMGmvvxwhIAP4YKvfTQ5kLYz
MTkU-e29FfGeq7gki7QuCNZWniU2e3W0hWyveq0euvjI8puVx2Fw”
}
2018-07-30 09:39:52,301:DEBUG:urllib3.connectionpool:“POST /acme/new-authz HTTP/1.1” 201 None
2018-07-30 09:39:52,348:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.10.2’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 849, in main
return config.func(config, plugins)
File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 626, in obtain_cert
action, _ = _auth_from_available(le_client, config, domains, certname, lineage)
File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 107, in _auth_from_available
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File “/usr/lib/python2.7/dist-packages/certbot/client.py”, line 291, in obtain_and_enroll_certificate
certr, chain, key, _ = self.obtain_certificate(domains)
File “/usr/lib/python2.7/dist-packages/certbot/client.py”, line 262, in obtain_certificate
self.config.allow_subset_of_names)
File “/usr/lib/python2.7/dist-packages/certbot/auth_handler.py”, line 67, in get_authorizations
domain, self.account.regr.new_authzr_uri)
File “/usr/lib/python2.7/dist-packages/acme/client.py”, line 216, in request_domain_challenges
typ=messages.IDENTIFIER_FQDN, value=domain), new_authzr_uri)
File “/usr/lib/python2.7/dist-packages/acme/client.py”, line 196, in request_challenges
new_authz)
File “/usr/lib/python2.7/dist-packages/acme/client.py”, line 671, in post
return self._post_once(*args, **kwargs)
File “/usr/lib/python2.7/dist-packages/acme/client.py”, line 682, in _post_once
response = self._send_request(‘POST’, url, data=data, **kwargs)
File “/usr/lib/python2.7/dist-packages/acme/client.py”, line 614, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/usr/lib/python2.7/dist-packages/requests/sessions.py”, line 457, in request
resp = self.send(prep, **send_kwargs)
File “/usr/lib/python2.7/dist-packages/requests/sessions.py”, line 606, in send
r.content
File “/usr/lib/python2.7/dist-packages/requests/models.py”, line 724, in content
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File “/usr/lib/python2.7/dist-packages/requests/models.py”, line 653, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File “/usr/lib/python2.7/dist-packages/urllib3/response.py”, line 256, in stream
data = self.read(amt=amt, decode_content=decode_content)
File “/usr/lib/python2.7/dist-packages/urllib3/response.py”, line 186, in read
data = self._fp.read(amt)
File “/usr/lib/python2.7/httplib.py”, line 602, in read
s = self.fp.read(amt)
File “/usr/lib/python2.7/socket.py”, line 380, in read
data = self._sock.recv(left)
File “/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py”, line 188, in recv
data = self.connection.recv(*args, **kwargs)
File “/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py”, line 1321, in recv
self._raise_ssl_error(self._ssl, result)
File “/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py”, line 1171, in _raise_ssl_error
raise ZeroReturnError()
ZeroReturnError


#2

Does this happen every time? If so, could you tell me if this helps?


#3

It did happen every time. I added the hosts entry, and the certificate was successfully issued. So hooray for that!

Since we were reliably able to get other certs, but reliably not able to get this one, the request size threshold range for triggering the problem must be quite tight.

If I understand the other post correctly, this isn’t a long-term solution because CDN resources may change. If there’s any other diagnostic information I can provide that will help to develop a long-term solution, I’m happy to help.


#4

If you can run the same diagnostics (make sure to undo the workaround) and post in the other thread, that might help when the staff eventually get around to looking at it.


#5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.