Dry-run cert renewal works, live cert renewal fails on 400 error

gguillotte · June 10, 2019, 10:31pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
oznogon.com

I ran this command:
certbot-auto renew

Same results when run manually. Was configured to run twice daily has a cron job at 700 and 1900 server time—never 5 times per day, much less 5 times per hour.

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/oznogon.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Running pre-hook command: service lighttpd stop
Renewing an existing certificate
Attempting to renew cert (oznogon.com) from /etc/letsencrypt/renewal/oznogon.com.conf produced an unexpected error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/oznogon.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/oznogon.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: service lighttpd start
1 renew failure(s), 0 parse failure(s)

My web server is (include version):
lighttpd/1.4.35

The operating system my web server runs on is (include version):
Debian 8.11

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 0.35.0

–

My certs expired on 8 June. Prior to that certbot was logging that its webroot challenges were failing

Running with --dry-run works:

root@garrettatlantic:~/bin# ./certbot-auto renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/oznogon.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Running pre-hook command: service lighttpd stop
Renewing an existing certificate

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/oznogon.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/oznogon.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: service lighttpd start

gguillotte · June 10, 2019, 11:39pm

Timeout expired, new error message:

# ./certbot-auto renew
Cleaning up challenges
Attempting to renew cert (oznogon.com) from /etc/letsencrypt/renewal/oznogon.com.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/oznogon.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/oznogon.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: service lighttpd start
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: oznogon.com
   Type:   connection
   Detail: Fetching
   http://oznogon.com/.well-known/acme-challenge/3qwR4dohGpUjFuxh9q_e8DvrppzqVKg_INAlyANJAGI:
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

Renew run log claims a 400 error received from oznogon.com:

    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "Fetching http://oznogon.com/.well-known/acme-challenge/hAjEjJOApmIyrvV2aOsopYzRwixJRn5epbWy2Tp
lrjU: Connection refused",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/-A1zARYSZb7J1b7pzOjXBneuucvCgA5_xiPdCmphBCM/169
15423411",
      "token": "hAjEjJOApmIyrvV2aOsopYzRwixJRn5epbWy2TplrjU",
      "validationRecord": [
        {
          "url": "http://oznogon.com/.well-known/acme-challenge/hAjEjJOApmIyrvV2aOsopYzRwixJRn5epbWy2TplrjU",
          "hostname": "oznogon.com",
          "port": "80",
          "addressesResolved": [
            "104.245.33.236"
          ],
          "addressUsed": "104.245.33.236"
        }
      ]
    }

I confirmed that the challenge file is present on the server during the run by running a watch -n 0.1 ls on the webroot .well-known/acme-challenge directory.

I confirmed that arbitrary files placed in the .well-known/acme-challenge directory are accessible without error.

Dry run still works without issue:

/certbot-auto renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/oznogon.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Running pre-hook command: service lighttpd stop
Renewing an existing certificate

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/oznogon.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/oznogon.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: service lighttpd start

Dry run log doesn’t flag any issues with accessing the webroot challenge on oznogon.com:

    {
      "type": "http-01",
      "status": "valid",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/a_l0Fj
xZGOoiWMb88vIMHRcA-Z_zdnWTJMi_bgizRtc/321806287",
      "token": "0Mw1i_r8YTLRqyR9Nsi8fRjwyCe3OEU3xVPmmZqPn3w",
      "validationRecord": [
        {
          "url": "http://oznogon.com/.well-known/acme-challenge/0Mw1i_r8YTLRqyR9
Nsi8fRjwyCe3OEU3xVPmmZqPn3w",
          "hostname": "oznogon.com",
          "port": "80",
          "addressesResolved": [
            "104.245.33.236"
          ],
          "addressUsed": "104.245.33.236"
        }
      ]
    }

gguillotte · June 10, 2019, 11:45pm

Access and error logs logged no attempts to access the challenge file in either dry runs or by a live renewal attempt.

gguillotte · June 11, 2019, 12:10am

Resolved this by abandoning certbot in favor of getssl, which had no problems validating the challenge file at the same webroot path.

JuergenAuer · June 11, 2019, 6:34am

Hi @gguillotte

if you use webroot, you need a running webserver.

So that can't work if you use a pre-hook which stops your webserver.

There are checks of your domain - https://check-your-website.server-daten.de/?q=oznogon.com

Your first certificate is from 2017-07-27, your last from 2019-03-10. Looks like you have used tls-sni-01 validation, that's not longer supported, support ended ~~ 2019-03-15.

There the pre-hook command is required. But not with webroot.

gguillotte · June 11, 2019, 7:47am

Thanks, Jurgen. Please request an update to these instructions by @bmw:

which states:

Do a full renewal dry run:
sudo certbot renew --dry-run
If the dry run succeeds, and your Certbot version is 0.28 or higher, you’re good to go! No further action should be required to deal with the end of TLS-SNI-01 support.

The dry run returned a success even with the webserver-stopping pre-hook enabled, but the actual renewal failed. Therefore, stating "If the dry run succeeds, and your Certbot version is 0.28 or higher, you’re good to go" is inaccurate.

Regardless of those instructions, why does the dry run succeed with the pre-hook enabled? The certbot documentation states:

For advanced certificate management tasks, it is possible to manually modify the certificate’s renewal configuration file, but this is discouraged since it can easily break Certbot’s ability to renew your certificates. If you choose to modify the renewal configuration file we advise you to test its validity with the certbot renew --dry-run command.

The --help-all output regarding --dry-run states:

It also calls --pre-hook and --post-hook commands if they are defined because they may be necessary to accurately simulate renewal.

These statements do not seem to reflect reality. If a pre-hook stopping the server is causing renewal to fail, I would expect consistent failures when running with and without --dry-run.

JuergenAuer · June 11, 2019, 7:52am

Thanks for your details.

Normally, --dry-run should be enough to check a configuration.

I have no idea why your --dry-run works, without that not.

gguillotte · June 11, 2019, 8:02am

Further muddying things: I have just successfully force-renewed the certificate via webroot challenge, with the lighttpd service-stopping pre-hook still enabled.

# ./certbot-auto renew --force-renewal
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/oznogon.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator webroot, Installer None
Running pre-hook command: service lighttpd stop
Renewing an existing certificate

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/oznogon.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/oznogon.com/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: service lighttpd restart

lighttpd’s error.log confirms that the server is being stopped and started at the same time as the pre-hook being executed and restarted when the post-hook is executed, so it seems unlikely that the hooks are preventing validation.

certbot’s inconsistent behavior here is nonsense, especially in light of the outdated getssl script working at the point in time when certbot was not.

mnordhoff · June 11, 2019, 8:22am

If Let’s Encrypt is successfully issuing certificates (production or staging) using HTTP validation while your web server is stopped, it could only be because it’s reusing a valid authorization from earlier and isn’t actually validating your domain again.

That’s likely why Certbot isn’t printing anything about having to do any validations. (I haven’t looked at “certbot renew”'s output recently, so I’m not 100% certain.)

system · July 11, 2019, 8:35am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Certbot Renew --dry-run fails with a 400 message Help	3	786	August 9, 2020
Certbot renew --dry-run fails Help	6	4997	February 16, 2020
Error renewing certificates while --dry-run succeeds Help	57	2921	November 11, 2020
Certificate renewal failure Help	3	2890	March 12, 2020
Certbot autorenew and renew fails Help	11	12581	March 5, 2019

Dry-run cert renewal works, live cert renewal fails on 400 error

Related topics