Error renewing vseprochleba.cz: The server experienced an internal error :: Error creating new authz

Help
1

Please fill out the fields below so we can help you better.

My domain is: vseprochleba.cz, www.vseprochleba.cz

I ran this command: certbot renew --pre-hook '/bin/run-parts /etc/letsencrypt/pre-hook.d/' --post-hook '/bin/run-parts /etc/letsencrypt/post-hook.d/' --renew-hook '/bin/run-parts /etc/letsencrypt/renew-hook.d/'

It produced this output:

Processing /etc/letsencrypt/renewal/vseprochleba.cz.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Running pre-hook command: /bin/run-parts /etc/letsencrypt/pre-hook.d/
Renewing an existing certificate
Attempting to renew cert from /etc/letsencrypt/renewal/vseprochleba.cz.conf produced an unexpected error: urn:acme:error:serverInternal :: The server experienced an internal error :: Error creating new authz. Skipping.

My operating system is (include version): Debian 8.0 (Jessie)

My web server is (include version): nginx 1.10.1-2~20160711125958.4+jessie (compiled with OpenSSL 1.0 for ALPN support)

My certbot version is (include version): 0.9.3-1~bpo8+1

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The logfile says:

2016-12-14 19:31:37,563:INFO:certbot.main:Renewing an existing certificate
2016-12-14 19:31:37,564:DEBUG:root:Requesting fresh nonce
2016-12-14 19:31:37,564:DEBUG:root:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-authz. args: (), kwargs: {}
2016-12-14 19:31:37,752:DEBUG:requests.packages.urllib3.connectionpool:"HEAD /acme/new-authz HTTP/1.1" 405 0
2016-12-14 19:31:37,753:DEBUG:root:Received <Response [405]>. Headers: {'Content-Length': '91', 'Pragma': 'no-cache', 'Boulder-Request-Id': 'ost4pddcIDeG3Z_if8wBXASyG9HtJpU0jg65Vb_Ffes', 'Expires': 'Wed, 14 Dec 2016 19:31:37 GMT', 'Server': 'nginx', 'Connection': 'keep-
alive', 'Allow': 'POST', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Date': 'Wed, 14 Dec 2016 19:31:37 GMT', 'Content-Type': 'application/problem+json', 'Replay-Nonce': 'uwTFhP4j84euIIHLwW2WPUqtdtAZM2o83Ohzl6sq5wQ'}. Content: ''
2016-12-14 19:31:37,753:DEBUG:acme.client:Storing nonce: '\xbb\x04\xc5\x84\xfe#\xf3\x87\xae \x81\xcb\xc1m\x96=J\xadv\xd0\x193j<\xdc\xe8s\x97\xab*\xe7\x04'
2016-12-14 19:31:37,753:DEBUG:acme.jose.json_util:Omitted empty fields: status=None, combinations=None, expires=None, challenges=None
2016-12-14 19:31:37,753:DEBUG:acme.client:Serialized JSON: {"identifier": {"type": "dns", "value": "vseprochleba.cz"}, "resource": "new-authz"}
2016-12-14 19:31:37,755:DEBUG:acme.jose.json_util:Omitted empty fields: x5c=(), crit=(), typ=None, jku=None, cty=None, x5t=None, alg=None, x5tS256=None, x5u=None, kid=None, jwk=None
2016-12-14 19:31:37,760:DEBUG:acme.jose.json_util:Omitted empty fields: x5c=(), crit=(), typ=None, jku=None, nonce=None, cty=None, x5t=None, kid=None, x5tS256=None, x5u=None
2016-12-14 19:31:37,761:DEBUG:root:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz. args: (), kwargs: {'data': '{"header": {"alg": "RS256", "jwk": {"e": "AQAB", "kty": "RSA", "n": "4nAB6Le6xdiGxuDk2UC684AdAxkpsgAfFCX31Wz6Vf1zF72C6trIRdH4VXiCh81yM-Df8QFcb2-G2BYnidNAu_9UXoCVu6jnaeq87PY9d6SFlICOFc_DdcqExbhTTCkb-J2wmAp0mce3FYXjalFElfHy7XTDOLTBwuFDt2Hi5G0kli3o1eiJJh311vhB_u6daueeU7M8kOnyDuWseFRIDeRH_5Hq5rgeo3TRpo593QyWZzQXMITgdQrd7dYNMctkS1ySPamjiuP2dnb0BjvOMWdJ32BF78GOuORJYKukLTfSRYcVzpQjPQ3-vBfeQzVI-rafb6_XfqCQlTz1Qa7pJw"}}, "protected": "eyJub25jZSI6ICJ1d1RGaFA0ajg0ZXVJSUhMd1cyV1BVcXRkdEFaTTJvODNPaHpsNnNxNXdRIn0", "payload": "eyJpZGVudGlmaWVyIjogeyJ0eXBlIjogImRucyIsICJ2YWx1ZSI6ICJ2c2Vwcm9jaGxlYmEuY3oifSwgInJlc291cmNlIjogIm5ldy1hdXRoeiJ9", "signature": "iPLldRLOhbyp0f6ckBxH3_Afg5i7otomay5OGbEhONR0QbjfTqxyUEbh55WcG-F5tJqR07zLfR-zGhkdi18RwyuMVGwBof97GWqsj3xDE-yXgx5bLTFf2RpFbH8rgKrBdzqxv6vF2UFFYuQtGsWachH3QqWvh6wVARCGHU1j7LUvm1ncXGxdM1H_wk-TSODMNRQ2Rx1jCrwFJVuMhaedhV99Sh2MCmwU3mZQ8pybgzH0AIsX5cjWoJOiXlXpcSbVaIFJ_vKmp5SYzrpprgvE4mYoqY7AtfqO89SXfW4yM8gSwesHWTh574P-PjK-vav5lsi1Vrjpa7pRS7wWhnH8tQ"}'}
2016-12-14 19:31:40,620:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/new-authz HTTP/1.1" 500 102
2016-12-14 19:31:40,624:DEBUG:root:Received <Response [500]>. Headers: {'Content-Length': '102', 'Boulder-Request-Id': 'Fut2G06DeOvsP-lx3tWR5yFQ-ZzdBVErI-ZTmOWBaQg', 'Expires': 'Wed, 14 Dec 2016 19:31:40 GMT', 'Server': 'nginx', 'Connection': 'close', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Pragma': 'no-cache', 'Boulder-Requester': '3478', 'Date': 'Wed, 14 Dec 2016 19:31:40 GMT', 'Content-Type': 'application/problem+json', 'Replay-Nonce': 'PRrfg7MDidbGHMMq3v5xyF-CMNI4v9v8DRAAjvovT9M'}. Content: '{\n  "type": "urn:acme:error:serverInternal",\n  "detail": "Error creating new authz",\n  "status": 500\n}'
2016-12-14 19:31:40,625:DEBUG:acme.client:Storing nonce: '=\x1a\xdf\x83\xb3\x03\x89\xd6\xc6\x1c\xc3*\xde\xfeq\xc8_\x820\xd28\xbf\xdb\xfc\r\x10\x00\x8e\xfa/O\xd3'
2016-12-14 19:31:40,625:DEBUG:acme.client:Received response <Response [500]> (headers: {'Content-Length': '102', 'Boulder-Request-Id': 'Fut2G06DeOvsP-lx3tWR5yFQ-ZzdBVErI-ZTmOWBaQg', 'Expires': 'Wed, 14 Dec 2016 19:31:40 GMT', 'Server': 'nginx', 'Connection': 'close', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Pragma': 'no-cache', 'Boulder-Requester': '3478', 'Date': 'Wed, 14 Dec 2016 19:31:40 GMT', 'Content-Type': 'application/problem+json', 'Replay-Nonce': 'PRrfg7MDidbGHMMq3v5xyF-CMNI4v9v8DRAAjvovT9M'}): '{\n  "type": "urn:acme:error:serverInternal",\n  "detail": "Error creating new authz",\n  "status": 500\n}'
2016-12-14 19:31:40,625:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/vseprochleba.cz.conf produced an unexpected error: urn:acme:error:serverInternal :: The server experienced an internal error :: Error creating new authz. Skipping.
2016-12-14 19:31:40,627:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/certbot/renewal.py", line 348, in renew_all_lineages
    main.obtain_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 563, in obtain_cert
    action, _ = _auth_from_domains(le_client, config, domains, lineage)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 96, in _auth_from_domains
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/dist-packages/certbot/renewal.py", line 238, in renew_cert
    new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
  File "/usr/lib/python2.7/dist-packages/certbot/client.py", line 253, in obtain_certificate
    self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 68, in get_authorizations
    domain, self.account.regr.new_authzr_uri)
  File "/usr/lib/python2.7/dist-packages/acme/client.py", line 210, in request_domain_challenges
    typ=messages.IDENTIFIER_FQDN, value=domain), new_authzr_uri)
  File "/usr/lib/python2.7/dist-packages/acme/client.py", line 190, in request_challenges
    new_authz)
  File "/usr/lib/python2.7/dist-packages/acme/client.py", line 649, in post
    return self._check_response(response, content_type=content_type)
  File "/usr/lib/python2.7/dist-packages/acme/client.py", line 565, in _check_response
    raise messages.Error.from_json(jobj)
2

Hi @oerdnj

I started looking into this for you. At this point it seems that it’s a problem on our end (perhaps related to the earlier instability we posted about on status.letsencrypt.org).

I’ll let you know when I know more. We’re fighting fires on a few fronts this week. Thanks for understanding!

3

Hi @oerdnj

Can you try the renewal again? It looks like the majority of the “Error creating new authz”'s we experienced were during our earlier production incident. I haven’t seen any since and this error is often indicative of database performance issues on our end.

4

Hi @cpu, nope, still getting the error. And the generated error from the log was produced just as I wrote this report.

5

Hi @oerdnj Ok, thanks for checking! Back to the drawing board. I see your most recent attempts in our logs producing the same error. Continuing to investigate from this end.

6

Hi @oerdnj,

Here’s a bit more detail: We’ve found that there are a large number of current, valid authorizations for this hostname. They appear to be leftover from a misconfiguration in March-May 2016 where your client was repeatedly validating and issuing for the same domain name. We’ve since added mitigations, but the backlog of authorizations means that queries of the authorizations on your account are moderately expensive. Normally they complete fast enough, but currently our database is experiencing high load and timing out these queries.

As a short-term workaround, you can create a new account with the same email address.

7

Hi @jsha, thanks for the information. Do you think the load will go away within ~20 days? I am not in particular hurry to renew the cert, it just needs to be renewed before it expires. So I would rather follow the normal procedure than making workarounds.

Or would it help you long term to generate a new account for all my domain names and let this mess I created cleanup itself in the old account?

8

I think there’s a good chance the load will subside within ~20 days and you’ll be able to issue once more on your regular account. I’d recommend setting certbot to attempt renewal once a day, and set an alert for yourself in 20 days to take additional steps if you haven’t successfully renewed.

The old authorizations will start expiring off your account in late January, though that will be after your upcoming expiration date.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.