Updated certbot, now nginx serves 404s to LE and gets “invalid response”

I set up my macOS 10.11 server running nginx 1.8.0 four years ago. I installed using homebrew. Certbot has worked great all that time, but the recent ACMEv1 deprecation has motivated me to upgrade Certbot from 0.4-ish to the latest version, 1.5.0. I am unfortunately having problems trying to get my server to respond to Let’s Encrypt. I have access to a shell, I manage the computer myself.

What happens is pretty simple: Let’s Encrypt reaches out to the /.well-known/acme-challenge/ URL, gets served a friendly HTML 404 page by my server, an “invalid response”, and fails. Here’s how it goes:

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for lovelysunny.company
Using default address 80 for authentication.
Waiting for verification...
Challenge failed for domain lovelysunny.company
http-01 challenge for lovelysunny.company
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: lovelysunny.company
   Type:   unauthorized
   Detail: Invalid response from
   https://lovelysunny.company/.well-known/acme-challenge/[a key i snipped out in case it's sensitive]
   [208.52.170.87]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
   Not Found</h1></center>\r\n<hr><center>"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Here’s what one of my nginx server .conf files looks like:

## Dumb HTTP server just redirects people to the HTTPS server. That's all it does.
server {
	listen							17539;
	server_name						lovelysunny.company www.lovelysunny.company;
	port_in_redirect				off;
	return 301						https://$host$request_uri;
}
# Actual HTTPS server
server {
	listen 							17540 ssl;
	server_name						lovelysunny.company;
	access_log						/usr/local/var/log/nginx/access.log;
	error_log						/usr/local/var/log/nginx/error.log error;
	port_in_redirect				off;
	root 							/Users/lovelysunny/Sites/lovelysunny.company;

	## SSL Settings
	add_header						Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
	ssl								on;
	ssl_certificate					/etc/letsencrypt/live/www.lovelysunny.company/fullchain.pem;
	ssl_certificate_key				/etc/letsencrypt/live/www.lovelysunny.company/privkey.pem;
	ssl_session_timeout				5m;
	ssl_protocols					TLSv1 TLSv1.1 TLSv1.2;
	ssl_ciphers						'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
	ssl_prefer_server_ciphers		on;
	ssl_dhparam						/usr/local/etc/nginx/ssl/lovelysunny.dhparam;

	# Allow Let's Encrypt to verify I own this domain
	include letsencrypt.conf;

	location ~ \.(php|html)$ {
		try_files					$uri =404;
		fastcgi_pass 				127.0.0.1:9000;
		fastcgi_index				index.html;
		fastcgi_param				SCRIPT_FILENAME $document_root$fastcgi_script_name;
		fastcgi_intercept_errors	on;
		include						/usr/local/etc/nginx/fastcgi.conf;
	}
}

Every one of my servers has one of those include letsencrypt.conf; lines. letsencrypt.conf looks like this:

location ^~ /.well-known/acme-challenge {
    alias /var/lib/letsencrypt/.well-known/acme-challenge;
    default_type "text/plain";
    try_files $uri =404;
}

Is this configuration old and crappy and needs updating? I don’t actually know where I got all the SSL settings lines, and the letsencrypt.conf — did certbot put it there on first install? Did I find it in some tutorial? I don’t know, but it seemed to work well.

I searched around and couldn’t actually find definitive instructions on what my nginx config is supposed to look like. I did see some example config saying I should use root, not alias. Should I?

Ideally, I want the most future-proof config here that both works and will work with the latest validation method for years to come.

An aside on ports

Nginx serves HTTP on port 17539 and HTTPS on port 17540. pf redirects all incoming 80 and 443 traffic to these weird ports. I can’t remember why I did this, but it’s how the system works and I’m not going to mess with it.

After doing some reading on how certbot works, I thought that this might be some kind of issue. (Even if it wouldn’t make a lot of sense that it wasn’t an issue before I upgraded.) I figured I needed to tell certbot to spin up its nginx server (??) to listen to port 17540. So I tried sudo certbot --http-01-port 17540. After selecting “nginx” and the number for the lovelysunny.company domain, these were the results:

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for lovelysunny.company
Using default address 17540 for authentication.
nginx: [warn] conflicting server name "lovelysunny.company" on 0.0.0.0:17540, ignored
Waiting for verification...
Challenge failed for domain lovelysunny.company
http-01 challenge for lovelysunny.company
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: lovelysunny.company
   Type:   connection
   Detail: Fetching
   https://lovelysunny.company/.well-known/acme-challenge/[removed key that might have sensitive info again]:
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

And that’s where I’m at. Any help in getting this to work — and in helping me get my configuration up to the latest standards — would be greatly appreciated. Sorry my server’s weird.

1 Like

Does this work?

certbot renew --cert-name lovelysunny.company \
-a webroot -w /var/lib/letsencrypt --dry-run

Not that I am advocating for this as a solution - I just want to confirm which Certbot authenticator produces what error.

1 Like

I ran it…

Maishul:~ lovelysunny$ sudo certbot renew --cert-name lovelysunny.company -a webroot -w /var/lib/letsencrypt --dry-run
Password:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
No certificate found with name lovelysunny.company (expected /etc/letsencrypt/renewal/lovelysunny.company.conf).

I took a look in /etc/letsencrypt/renewal/ and found a bunch of .conf files for my domains in there, but with www. prefixes. e.g. www.lovelysunny.company.conf. So…

Maishul:~ lovelysunny$ sudo certbot renew --cert-name www.lovelysunny.company -a webroot -w /var/lib/letsencrypt --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.lovelysunny.company.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for lovelysunny.company
http-01 challenge for www.lovelysunny.company
Using the webroot path /var/lib/letsencrypt for all unmatched domains.
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.lovelysunny.company/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/www.lovelysunny.company/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Is it good that this works?

1 Like

It’s good in that it confirms the problem is related to the nginx authenticator.

I notice that you opted for --http-01-port 17540, but I think that you should have used 17539. That would probably explain the “Error getting validation data” - it’s caused by a protocol mismatch.

Try renewing again (your way), but with that one change to the --http-01-port.

1 Like

It worked!

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for lovelysunny.company
Waiting for verification...
Cleaning up challenges
Deploying Certificate to VirtualHost /usr/local/etc/nginx/servers/lovelysunny.company.conf
No matching insecure server blocks listening on port 80 found.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled https://lovelysunny.company

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=lovelysunny.company
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/lovelysunny.company/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/lovelysunny.company/privkey.pem
   Your cert will expire on 2020-09-28. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

My guess is that my next step is to run sudo certbot --http-01-port 17539 manually, and to change the scheduled task that currently just reads certbot renew to certbot --http-01-port 17539 renew, but I haven’t done that yet. Is that right?

Also, my question about my potentially outdated nginx certbot config still stands… is this the right way to configure nginx? I can see in the announcements some mention of a future deprecation of “unauthenticated GET requests” — am I ready for this change? Is everything good?

1 Like

When you renewed it just now with the updated port, it updated the .conf file in /etc/letsencrypt/renewal with the new port. As such, you don’t need to change anything further.

:man_shrugging:. The way you’ve done it is fine, I think stuff like this largely comes down to sysadmin tastes.

The nginx authenticator in Certbot actually (temporarily) adds all of the /.well-known/acme-challenge/ nginx stanzas on its own during authentication, so the one you have configured on your own is redundant. It would be useful if you used the webroot authenticator (as in a couple of posts back). But it’s not a big deal.

At version 1.5.0, definitely, yes.

1 Like

Thank you so much! I suppose I will leave my nginx setup as it is, then.

Unfortunately, even though it looked like you just gave me a solution, I wasn’t able to get this to work for the rest of my domains. I don’t understand it.

If I run certbot --http-01-port 17539, pick “nginx”, and don’t enter a number (so it attempts to activate HTTPS for 21 of my domains) they all fail the challenge. I ran it a second time and selected another domain of mine with a near-identical simple, small nginx .conf, bcb.cat, and got this:

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for bcb.cat
Waiting for verification...
Challenge failed for domain bcb.cat
http-01 challenge for bcb.cat
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: bcb.cat
   Type:   unauthorized
   Detail: Invalid response from
   https://bcb.cat/.well-known/acme-challenge/[key removed]
   [208.52.170.87]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
   Not Found</h1></center>\r\n<hr><center>"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

The nginx .conf file for bcb.cat is very similar to lovelysunny.company:

server {
	listen							17539;
	server_name						bcb.cat www.bcb.cat;
	access_log						/usr/local/var/log/nginx/access.log;
	error_log						/usr/local/var/log/nginx/error.log error;
	root 							/Users/lovelysunny/Sites/bcb.cat;
	port_in_redirect				off;

	return 301						https://$host$request_uri;
}
server {
	listen 							17540 ssl;
	server_name						bcb.cat www.bcb.cat;
	access_log						/usr/local/var/log/nginx/access.log;
	error_log						/usr/local/var/log/nginx/error.log error;
	root 							/Users/lovelysunny/Sites/bcb.cat;
	port_in_redirect				off;

	## SSL Settings
	add_header						Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
	ssl								on;
	ssl_certificate					/etc/letsencrypt/live/www.bcb.cat/fullchain.pem;
	ssl_certificate_key				/etc/letsencrypt/live/www.bcb.cat/privkey.pem;
	ssl_session_timeout				5m;
	ssl_protocols					TLSv1 TLSv1.1 TLSv1.2;
	ssl_ciphers						'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
	ssl_prefer_server_ciphers		on;
	ssl_dhparam						/usr/local/etc/nginx/ssl/lovelysunny.dhparam;

	# Allow Let's Encrypt to verify I own this domain
	include letsencrypt.conf;


	# Redirect all other requests to full-URL site
	location ~ / { rewrite ^/(.*) https://www.bittersweetcandybowl.com/$1 redirect; }
}

I’m stuck once more. The first command you sent seems to work:

Maishul:certbot lovelysunny$ sudo certbot renew --cert-name www.bcb.cat -a webroot -w /var/lib/letsencrypt --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.bcb.cat.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for bcb.cat
http-01 challenge for www.bcb.cat
Using the webroot path /var/lib/letsencrypt for all unmatched domains.
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.bcb.cat/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/www.bcb.cat/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

But after doing this, running certbot --http-01-port 17539 and picking either the “www.bcb.cat” or “bcb.cat” options leads to the same “Challenge failed” error.

1 Like

I think it should work, I don’t see anything unusual.

The only thing that comes to mind is the possibility that you are running into this bug, which can be summed up as: when there are a lot of nginx virtualhosts, nginx doesn’t reload quickly enough when Certbot is responding to the challenges. I didn’t think that only 21 virtual hosts could trigger this, but who knows.

To confirm whether the above is the issue, you could try renew with --dry-run --debug-challenges, which will cause Certbot to pause right after it modifies your nginx configuration with the challenge response. You can then open up the config to see how the config has been changed and whether you can access that challenge resource in the browser.

If you don’t really care to investigate, you could just adopt the -a webroot method permanently. At least, it avoids excessive reloading of your nginx server, and if you have that letsencrypt.conf snippet included everywhere, it should work for all your domains.

1 Like

Unfortunately, for some reason it does not pause when I run certbot renew --dry-run --debug-challenges. It just ploughs right through with no waiting for user interaction. It’s kind of slow so I tried keeping an eye on my .conf files to grab one mid-update, but I can’t tell which ones are being updated so I don’t know what to look at. It’s not editing my actual /usr/local/etc/nginx/servers/* conf files, is it? If it is, it preserves the modification dates.

Here’s how it looks when I ran the command:

Maishul:certbot lovelysunny$ sudo certbot renew --dry-run --debug-challenges
Password:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/img.soaringrabbit.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for img.soaringrabbit.com
http-01 challenge for soaringrabbit.com
http-01 challenge for www.soaringrabbit.com
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/img.soaringrabbit.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/lovelysunny.company.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for lovelysunny.company
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenge failed for domain lovelysunny.company
http-01 challenge for lovelysunny.company
Cleaning up challenges
Attempting to renew cert (lovelysunny.company) from /etc/letsencrypt/renewal/lovelysunny.company.conf produced an unexpected error: Some challenges have failed.. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.bcb.cat.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for bcb.cat
http-01 challenge for www.bcb.cat
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.bcb.cat/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.bittersweetcandybowl.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for bittersweetcandybowl.com
http-01 challenge for cdn.bittersweetcandybowl.com
http-01 challenge for cdn2.bittersweetcandybowl.com
http-01 challenge for cdn3.bittersweetcandybowl.com
http-01 challenge for www.bittersweetcandybowl.com
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.bittersweetcandybowl.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.horseface.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for horseface.org
http-01 challenge for www.horseface.org
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.horseface.org/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.itca.se.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for itca.se
http-01 challenge for www.itca.se
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.itca.se/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.lovelysunny.company.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for lovelysunny.company
http-01 challenge for www.lovelysunny.company
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.lovelysunny.company/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.notenoughrings.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for notenoughrings.com
http-01 challenge for www.notenoughrings.com
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.notenoughrings.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following certs could not be renewed:
  /etc/letsencrypt/live/lovelysunny.company/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

The following certs were successfully renewed:
  /etc/letsencrypt/live/img.soaringrabbit.com/fullchain.pem (success)
  /etc/letsencrypt/live/www.bcb.cat/fullchain.pem (success)
  /etc/letsencrypt/live/www.bittersweetcandybowl.com/fullchain.pem (success)
  /etc/letsencrypt/live/www.horseface.org/fullchain.pem (success)
  /etc/letsencrypt/live/www.itca.se/fullchain.pem (success)
  /etc/letsencrypt/live/www.lovelysunny.company/fullchain.pem (success)
  /etc/letsencrypt/live/www.notenoughrings.com/fullchain.pem (success)

The following certs could not be renewed:
  /etc/letsencrypt/live/lovelysunny.company/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: lovelysunny.company
   Type:   unauthorized
   Detail: Invalid response from
   https://lovelysunny.company/.well-known/acme-challenge/[snip]
   [208.52.170.87]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
   Not Found</h1></center>\r\n<hr><center>"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

I don’t really want to change to a seemingly more flaky method than the old certbot renew that used to work so reliably. I’d rather solve this problem and have it work like it used to. But I don’t understand how I “fixed” the original lovelysunny.company setup, and I don’t understand what state everything’s in now.

Clearly /private/etc/letsencrypt/renewal now contains an updated lovelysunny.company.conf with a version = 1.5.0 config file and the special 17539 port specified, but the other config files do not have that info and I don’t know how to get certbot to update them. And based on the results of the above, I don’t even know if lovelysunny.company.conf works! It seems to consider it the sole “renew failure”.

By the way, is the following message relevant? It happens if I just run certbot and select every domain by typing nothing. I don’t understand the message… is it bad to “expand”? It almost reads like it’s trying to merge all my domains together into a single certificate. Agreeing to expand just leads to every page responding with a 404 and getting the same invalid response error.

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/letsencrypt/renewal/www.bittersweetcandybowl.com.conf)

It contains these names: www.bittersweetcandybowl.com, bittersweetcandybowl.com,
cdn.bittersweetcandybowl.com, cdn2.bittersweetcandybowl.com,
cdn3.bittersweetcandybowl.com

You requested these names for the new certificate: bcb.cat, www.bcb.cat,
bittersweetcandybowl.com, cdn.bittersweetcandybowl.com,
cdn2.bittersweetcandybowl.com, cdn3.bittersweetcandybowl.com,
www.bittersweetcandybowl.com, blasto.enterprises, horseface.org,
www.horseface.org, itca.se, www.itca.se, lovelysunny.company,
www.lovelysunny.company, notenoughrings.com, www.notenoughrings.com, [censoring this one], soaringrabbit.com, img.soaringrabbit.com, www.soaringrabbit.com.

Do you want to expand and replace this existing certificate with the new
certificate?
1 Like

Certbot will only update the config if the renewal succeeds.

If you want to keep them separate, I suggest specifying the domains on the CLI like:

certbot --nginx -d example.com -d www.example.com

Have a look at the “Plugins selected” line for each virtual host.

It turns out that you have been using the webroot authenticator for all of the other domains, and only lovelysunny.company (which you opened this thread with) was configured to use the nginx authenticator. That difference probably accounts for some of this confusion.

I am hoping that everything will succeed for all domains with both of these options, and then it’d be up to you to commit to one or the other:

Option A:

certbot renew -a weboot -w /var/lib/letsencrypt --dry-run

Option B:

certbot renew -a nginx --http-01-port 17539 --dry-run
1 Like

I wonder how on earth that happened. Unless I inadvertently changed it while creating and responding to this thread, the only explanation is that I just so happened to set it up differently years ago.

Option B does not work — all the domains return a 404. Option A (with the typo corrected — webroot) does work, declaring “Congratulations, all renewals succeeded”. In both cases, lovelysunny.company does not behave differently from any other domain.

If I run Option A without the dry-run flag, it just says nothing is due for renewal yet. Based on the success of the dry run, can I safely assume that if I change my scheduled task from certbot renew to certbot renew -a webroot -w /var/lib/letsencrypt and everything will work and I won’t have to trouble you any further? Does lovelysunny.company need to be reconfigured to be consistent with the others?

And regarding the “merging into a single certificate”… I guess I don’t know the consequence of this, but it would seem to me that it’d be weird for someone to investigate one of my certificates and see a list of every other domain on this server. I’d like to keep it the same way it used to be, where certificates solely cover the domain and its subdomains (bittersweetcandybowl.com and cdn.bittersweetcandybowl.com).

1 Like