Unable to renew cert using nginx plugin, fresh cert creation succeeds

I was able to renew this certificate (with the nginx plugin) just now by using certbot -d www.friendsofvalledeoro.org but renewals keep failing recently. I’ve also experienced this with other sites on this same VPS and hosting setup, but I think one test case it probably the best place to start.

I tried earlier earlier renewals with and without dry-run and staging, and dry-run attempts after I made the new cert. All of them fail with similar errors, as it appears nginx is not being reloaded to apply the challenge info.

The server block (part of larger site config for www. and the root domain) as applied during an earlier --debug-challenges test:

    server{rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot
        listen       80;
        listen       443 ssl http2;
        server_name www.friendsofvalledeoro.org;
        return 301 $scheme://friendsofvalledeoro.org$request_uri;

        ssl_certificate /etc/letsencrypt/live/www.friendsofvalledeoro.org/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/www.friendsofvalledeoro.org/privkey.pem; # managed by Certbot
    location = /.well-known/acme-challenge/n1dYB0Zqon81WNhNEAoWIMTawF65HjNXkXuPR4gJK6Q{default_type text/plain;return 200 n1dYB0Zqon81WNhNEAoWIMTawF65HjNXkXuPR4gJK6Q.hy--gdyqV-OaRa9DxNUfXNuD5i6WFpFIGC7GYszOTog;} # managed by Certbot
    }

When I added that block to the config (after the test-run had been completed) and reloaded nginx, I was able to get a valid plaintext response. I have since removed that again for testing.

My domain is: www.friendsofvalledeoro.org
Let’s Debug lists no issues.

I ran this command as root: certbot renew --dry-run --cert-name www.friendsofvalledeoro.org

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.friendsofvalledeoro.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.friendsofvalledeoro.org
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (www.friendsofvalledeoro.org) from /etc/letsencrypt/renewal/www.friendsofvalledeoro.org.conf produced an unexpected error: Failed authorization procedure. www.friendsofvalledeoro.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://friendsofvalledeoro.org/.well-known/acme-challenge/FxncpvICAGchcCBQIqlotqyC1GMkg4Wcf-n_WDAtm_4 [138.197.233.49]: "<!DOCTYPE html>\n\n<html class=\"no-js\" lang=\"en-US\">\n\n<head>\n  \n<meta charset=\"UTF-8\">\n<meta name=\"viewport\" content=\"width=device". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/www.friendsofvalledeoro.org/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/www.friendsofvalledeoro.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.friendsofvalledeoro.org
   Type:   unauthorized
   Detail: Invalid response from
   https://friendsofvalledeoro.org/.well-known/acme-challenge/FxncpvICAGchcCBQIqlotqyC1GMkg4Wcf-n_WDAtm_4
   [138.197.233.49]: "<!DOCTYPE html>\n\n<html class=\"no-js\"
   lang=\"en-US\">\n\n<head>\n  \n<meta charset=\"UTF-8\">\n<meta
   name=\"viewport\" content=\"width=device"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version):

nginx version: nginx/1.16.1
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11)
built with OpenSSL 1.0.2g  1 Mar 2016
TLS SNI support enabled

The operating system my web server runs on is (include version): Ubuntu Xenial 16.04 - fully patched (per repositories)

My hosting provider, if applicable, is: DigitalOcean VPS, no overlaying load balancers or DNS/Let’s Encrypt integration.

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.31.0

Hi @beret

if you use the nginx Authenticator, Certbot adds a definition so your redirect http -> https shouldn’t be visible.

But Letsencrypt

sees your redirect, perhaps the wrong vHost is used.

Perhaps there are too much other (wrong) definitions.

What says

nginx -T

PS: Now the check is complete - your configuration is not typical - https://check-your-website.server-daten.de/?q=friendsofvalledeoro.org

Issuer not before not after Domain names LE-Duplicate next LE
Let’s Encrypt Authority X3 2019-07-17 2019-10-15 shop.friendsofvalledeoro.org - 1 entries
Let’s Encrypt Authority X3 2019-07-15 2019-10-13 friendsofvalledeoro.org - 1 entries
Let’s Encrypt Authority X3 2019-07-08 2019-10-06 www.friendsofvalledeoro.org - 1 entries

Different certificates non-www and www.

Your non-www:

CN=friendsofvalledeoro.org
	15.07.2019
	13.10.2019
expires in 18 days	friendsofvalledeoro.org - 1 entry

is old, your www

CN=www.friendsofvalledeoro.org
	25.09.2019
	24.12.2019
expires in 90 days	www.friendsofvalledeoro.org - 1 entry

is new. So you have two different vHosts

And your www redirects to your non-www. Looks like you have too much vHosts, so the wrong is used.

Yes, I am doing a number of split vhosts.
www. is just the redirect to the root, and shop. is a different web application.

Are you recommending consolidating the subdomains in nginx (at least www. and root), using consolidated Let’sEncrypt cert requests, or both?

I am wary of merging www and root vhosts, as that is expressly discouraged by the nginx documentation.

The -T output is here, but it’s massive:
https://pastebin.com/Favsx2PT

As an aside, I retried the same process after some restarts of nginx, and www. appears to complete a --dry-run renewal. However it does not print any details of completing an HTTP-01 challenge. Other dry-run attempts (on the root domain) fail as they did before.

# certbot renew --dry-run --cert-name www.friendsofvalledeoro.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.friendsofvalledeoro.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/www.friendsofvalledeoro.org/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/www.friendsofvalledeoro.org/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

# certbot renew --dry-run --cert-name friendsofvalledeoro.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/friendsofvalledeoro.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for friendsofvalledeoro.org
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (friendsofvalledeoro.org) from /etc/letsencrypt/renewal/friendsofvalledeoro.org.conf produced an unexpected error: Failed authorization procedure. friendsofvalledeoro.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://friendsofvalledeoro.org/.well-known/acme-challenge/NMAgUlBDlGM5Zc7yAkiqBuEw4V5psETYGuAcM3_H4rY [138.197.233.49]: "<!DOCTYPE html>\n\n<html class=\"no-js\" lang=\"en-US\">\n\n<head>\n  \n<meta charset=\"UTF-8\">\n<meta name=\"viewport\" content=\"width=device". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/friendsofvalledeoro.org/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/friendsofvalledeoro.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: friendsofvalledeoro.org
   Type:   unauthorized
   Detail: Invalid response from
   https://friendsofvalledeoro.org/.well-known/acme-challenge/NMAgUlBDlGM5Zc7yAkiqBuEw4V5psETYGuAcM3_H4rY
   [138.197.233.49]: "<!DOCTYPE html>\n\n<html class=\"no-js\"
   lang=\"en-US\">\n\n<head>\n  \n<meta charset=\"UTF-8\">\n<meta
   name=\"viewport\" content=\"width=device"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

You have

    listen       80;
    listen       443 ssl http2;
    server_name www.friendsofvalledeoro.org;

port 80 and port 443, http and https in one definition.

Split http and https, but merge both port 80 / both port 443 of the same main domain.

And your log shows the same problem - http is redirected to https. So the wrong vHost is used (if you use --nginx, not webroot).

To be clear, your suggestion is two vhosts (server blocks) with www. to root redirects (using if statements) in each?

HTTP 80: friendsofvalledeoro.org, www.friendsofvalledeoro.org
HTTPS 443: friendsofvalledeoro.org, www.friendsofvalledeoro.org

For testing, I removed (commented) all server blocks regarding www, and set the root domain vhost block to only listen on 80 (http). I then made a separate one for 443, with just listen 443 ssl and server_name directives. They both respond in my testing, and the configuration can be seen in the linked debug output: https://pastebin.com/j0p7QQJW

Even with this root domain dry-run, I see that the validation continues to fail. The challenge appears to be correctly formatted for the config file. (as printed in the debug output)

Manual hooks aren’t used for the plugin, and --challenge-debug does not wait when renewing. What’s the best way to verify config changes are being written and nginx is being properly reloaded?

For further certainty, I also disabled PHP from the main server block, to ensure Wordpress could not apply any https rewriting. I verified that curl -v http://friendsofvalledeoro.org retrieved a static page with no redirection headers or meta refresh redirects. The dry-run against the root still fails, but now the error message lists http:// on the challenge URL rather than https://

By the way, thank you for stepping through this with me!

# certbot renew --dry-run --cert-name friendsofvalledeoro.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/friendsofvalledeoro.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for friendsofvalledeoro.org
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (friendsofvalledeoro.org) from /etc/letsencrypt/renewal/friendsofvalledeoro.org.conf produced an unexpected error: Failed authorization procedure. friendsofvalledeoro.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://friendsofvalledeoro.org/.well-known/acme-challenge/9a-yWnyNmiegLF-T3UaJ8wiNIVpsZ0EjsovC10u3UTE [138.197.233.49]: "<?php\n/**\n * Front to the WordPress application. This file doesn't do anything, but loads\n * wp-blog-header.php which does and t". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/friendsofvalledeoro.org/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/friendsofvalledeoro.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: friendsofvalledeoro.org
   Type:   unauthorized
   Detail: Invalid response from
   http://friendsofvalledeoro.org/.well-known/acme-challenge/9a-yWnyNmiegLF-T3UaJ8wiNIVpsZ0EjsovC10u3UTE
   [138.197.233.49]: "<?php\n/**\n * Front to the WordPress
   application. This file doesn't do anything, but loads\n *
   wp-blog-header.php which does and t"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

curl http verification output:

$ curl -v http://friendsofvalledeoro.org
* Rebuilt URL to: http://friendsofvalledeoro.org/
*   Trying 138.197.233.49...
* TCP_NODELAY set
* Connected to friendsofvalledeoro.org (138.197.233.49) port 80 (#0)
> GET / HTTP/1.1
> Host: friendsofvalledeoro.org
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 25 Sep 2019 23:26:53 GMT
< Content-Type: application/octet-stream
< Content-Length: 420
< Last-Modified: Wed, 15 May 2019 22:02:29 GMT
< Connection: keep-alive
< ETag: "5cdc8c75-1a4"
< X-XSS-Protection: 1; mode=block
< Accept-Ranges:  none
<
<?php
/**
 * Front to the WordPress application. This file doesn't do anything, but loads
 * wp-blog-header.php which does and tells WordPress to load the theme.
 *
 * @package WordPress
 */

/**
 * Tells WordPress to load the WordPress theme and output it.
 *
 * @var bool
 */
define( 'WP_USE_THEMES', true );

/** Loads the WordPress Environment and Template */
require( dirname( __FILE__ ) . '/wp-blog-header.php' );
* Connection #0 to host friendsofvalledeoro.org left intact

Yes, that’s the configuration.

In your log you see the problem.

Certbot adds a location definition

location = /.well-known/acme-challenge/3nDFtxsXkac5th8PGfoj9WljTwk9ZJWlY3TjRR1qQfk{default_type text/plain;return 200 3nDFtxsXkac5th8PGfoj9WljTwk9ZJWlY3TjRR1qQfk._aXkCzYdljdO7X_7roI7VYRyLzZt0m0gOCyeHx0aQAo;} # managed by Certbot

but that vHost isn’t used. The error says again, that Letsencrypt checks the https version.

You must have another vHost which is used instead.

I feel confident this is related to failing reloads of nginx when attempting certbot renew. Nginx appears to not be reloaded correctly, so the updated challenge configuration is not used even though it is written correctly. I don’t know if it’s reproducible elsewhere, or an issue in our environment, but I believe that’s what is happening.

Per your post, this renew issue occurred even on the stripped down config in my post above (#6) with PHP disabled and no HTTPS redirection. The output indicated the validation server tested against HTTP, and had the same failure.

As a test case, I ran another renewal # certbot renew --debug-challenges --cert-name friendsofvalledeoro.org while executing systemctl nginx reload twice a second in another shell. This renewal processed without issue.

A subsequent test without the manual reloads failed with the same “Failed authorization” output provided earlier, and no challenge response returned: # certbot renew --debug-challenges --dry-run --cert-name friendsofvalledeoro.org

After completing the www.friendsofvalledeoro.org renewal with the workaround, I reviewed my certbot per-site configuration files. I removed some pre and post hooks for a few legacy certificates, and unified the few which were not using both nginx validation and installer plugins.

However the other renewal impacted certificates, including my test-case in this thread, were already inline with the modern plugins and had no special hooks.

I’ve been able to complete those outstanding renewals, but I had to rerun certbot renew repeatedly as it churned through the renewals, failing to complete some renewals with similar failures consistent with improperly reloaded server configs.

Ackis opened an issue related to this issue on github, and I will continue work there: