Creating certificates with --nginx works. Renew or certonly doesn't

Vidyut · 2019-02-12 23:59:26 UTC

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: multiple domains - I can add individual names, but they are irrelevant.

I ran this command:
certbot renew - did not work acme-challenge timed out - all settings normal, same as always. Been using letsencrypt from the start. with/out --dry-run
certbot certonly - did not work. Acme challenge timed out. with/out --dry-run
certbot --nginx -d domain1.com -domain2.com etc - worked.

It produced this output: varied between 404 and timed out

My web server is (include version): Nginx 1.15.8

The operating system my web server runs on is (include version): debian 9

My hosting provider, if applicable, is: digitalocean

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.28.0

Main thing is that same domains, same server, same everything (permissions, test files work, etc) other than command works. Even after deleting all certificates and making fresh ones using certbot --nginx (works) certbot renew --dry-run fails. Literally immediately after. certbot --nginx works first try.

rg305 · 2019-02-13 00:11:35 UTC

One explanation is that the new certs are being issued via TLS-SNI-01 authentication.
While the --dry-run are forced to use HTTP-01 authentication [which (for yet unknown reasons) fails]

[there may be other reasons it fails - I just though I would offer one to start the conversation…]

Please show:
cerbot certificates

and also their matching renewal.conf files.
[found in the folder /etc/letsencrypt/renewal/]
[one specific test conf should do; as they should all be suffering from the same problem]

orangepizza · 2019-02-13 00:48:07 UTC

IIRC TLS-SNI-01 for new cert already disabled in 2018 Jan when it first discovered.

rg305 · 2019-02-13 01:03:36 UTC

I believe the “new” described in the topic may not actual be as the “new” you express in your post.
[for instance: “new” and “renew” are often thought as interchangeable]
And, thus, perhaps both are “true and correct” - while seemingly contradicting each other.

I definitely need more “details” to better understand the actual problem.
[just not ready to make any assumptions (yet)]

Vidyut · 2019-02-13 07:26:04 UTC

~# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Found the following certs:
Certificate Name: vidyut.net-0001
Domains: vidyut.net aadhaar.fail aamjanata.com aamjanata.in digitalindia.watch fekle.in homeschoolingindia.in map.aamjanata.com nisarga.info vidyut.info wide-aware.com www.aadhaar.fail www.aamjanata.com www.aamjanata.in www.digitalindia.watch www.fekle.in www.homeschoolingindia.in www.nisarga.info www.vidyut.info www.vidyut.net www.wide-aware.com
Expiry Date: 2019-05-13 22:34:05+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/vidyut.net-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/vidyut.net-0001/privkey.pem

Vidyut · 2019-02-13 07:31:08 UTC

Renewing failed, using certbot to create slightly modified certs failed. Issuing new certificates failed because directory already exists.

Certs were due to explire. Emergency situation.

So I deleted all the certs, brought the site down to http and reissued one combined cert for anything.

Cert issued with no hassles, but new one also fails the certbot renew --dry-run

Vidyut · 2019-02-13 07:36:04 UTC

renew_before_expiry = 30 days

version = 0.28.0
archive_dir = /etc/letsencrypt/archive/vidyut.net-0001
cert = /etc/letsencrypt/live/vidyut.net-0001/cert.pem
privkey = /etc/letsencrypt/live/vidyut.net-0001/privkey.pem
chain = /etc/letsencrypt/live/vidyut.net-0001/chain.pem
fullchain = /etc/letsencrypt/live/vidyut.net-0001/fullchain.pem

Options used in the renewal process

[renewalparams]
authenticator = nginx
account = [not sure if ok to share]
installer = nginx
server = https://acme-v02.api.letsencrypt.org/directory

rg305 · 2019-02-13 15:58:38 UTC

The good news:

version = 0.28.0
reissued one combined cert for anything
Expiry Date: 2019-05-13 22:34:05+00:00 (VALID: 89 days)
all names resolve to same IPv4 address: 128.199.212.123
Connecting to 128.199.212.123|:80… connected.

The bad news:

new one also fails the certbot renew –dry-run

Based on the good information, --dry-run should be working.
We need to better understand why it fails…
Please show (additional detailed) output of:
certbot renew --dry-run -vvv

Vidyut · 2019-02-13 23:12:17 UTC

https://pastebin.com/PXvbvGQQ

rg305 · 2019-02-14 00:04:27 UTC

I’m not sure why…
But the nginx installer is creating server blocks and modifications that don’t… “work” with the rest of the config.

Please upload the whole nginx config form the output of:
nginx -T

[the reason/explanation/fix will be found somewhere in there]

In any event, we can always fallback to using --webroot as PLAN #B.
[we have 89 days to get this right - LOL]

Vidyut · 2019-02-14 03:51:11 UTC

Ah ok. That may explain it. I have a different config. One file with all the server names and their roots. Another with the ssl, locations (since it is shared), etc that gets included in each server block. So if the plugin is trying to physically alter the server block… it may not find the parts it expects?

Strange that I’m running into this issue only now. Been using this method from the start of using letsencrypt and they were renewed using a cron.

Vidyut · 2019-02-14 04:21:33 UTC

Can produce the nginx -T, but it will unnecessarily expose my security configurations unless I edit a lot.

Basically a gist of what is going on is:

Server blocks are all in one file in sites-enabled
Something like

server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name .one;
root /var/www/one;
include global/one-type.conf;
}
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name .two;
root /var/www/two;
include global/another-type.conf;
}
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name .one;
root /var/www/three;
include global/one-type.conf;
}
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name .four;
root /var/www/four;
include global/one-type.conf;
}

The included file will have one out of 2-3 variations of
ssl_certificate /etc/letsencrypt/live/vidyut.net-0001/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/vidyut.net-0001/privkey.pem;
ssl_more-stuff* __ciphers, dhparam,stapling,whatever;
[add_header stuff]
[cache setup stuff]
location / {
try_files $uri $uri/ /index.php?$args;
}
location ~ /.well-known {
allow all;
}

[more locations for php, caching, etc]

[includes fastcgi_params]

**This is very convenient for updating configurations for servers - best practices, updating ssl config for security, etc. Edit one or two files and done instead of updating each server block and waaaay less lines of code to wade through. But obviously it isn’t more convenient than certbot renew vs what I went through **

So what are the parts I must bring back to the main config file so that letsencrypt can find them? Or does it want me to have separate config files per server?

rg305 · 2019-02-14 14:49:47 UTC

I think we can modify your config to suit both needs.

First:

increase the path specifically to /.well-known/acme-challenge
[or add another location if you use other /.well-known paths]
Add line to same location: default_type text/plain;

Then we test (using site one as example):
[using root /var/www/one]

mkdir /var/www/one/.well-known
mkdir /var/www/one/.well-known/acme-challenge
echo "testing" > /var/www/one/.well-known/acme-challenge/1234

We should be able to view this new file via the Internet at:
http://site-ONE/.well-known/acme-challenge/1234

If that works, we are on our way.
If that fails, then we can still opt for PLAN #B [--webroot]

Vidyut · 2019-02-14 15:29:47 UTC

adding /.well-known/acme-challenge and default_type text/plain; made no difference.

The folders exist in all domains. Can create and access the test file from all domains.

Configuration seems to be an issue for --webroot. Just realized that creating the combined cert with --nginx worked, but it could not install the cert. I did that manually, but thought nothing of it, since I’ve always done letsencrypt certs like that (earlier there was no installer - when I started using)

===================================
#certbot renew --webroot --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/vidyut.net-0001.conf

Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for aadhaar.fail
http-01 challenge for aamjanata.com
http-01 challenge for aamjanata.in
http-01 challenge for digitalindia.watch
http-01 challenge for fekle.in
http-01 challenge for homeschoolingindia.in
http-01 challenge for map.aamjanata.com
http-01 challenge for nisarga.info
http-01 challenge for vidyut.info
http-01 challenge for vidyut.net
http-01 challenge for wide-aware.com
http-01 challenge for www.aadhaar.fail
http-01 challenge for www.aamjanata.com
http-01 challenge for www.aamjanata.in
http-01 challenge for www.digitalindia.watch
http-01 challenge for www.fekle.in
http-01 challenge for www.homeschoolingindia.in
http-01 challenge for www.nisarga.info
http-01 challenge for www.vidyut.info
http-01 challenge for www.vidyut.net
http-01 challenge for www.wide-aware.com
Cleaning up challenges
Attempting to renew cert (vidyut.net-0001) from /etc/letsencrypt/renewal/vidyut.net-0001.conf produced an unexpected error: Missing command line flag or config entry for this setting:
Input the webroot for aadhaar.fail:. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/vidyut.net-0001/fullchain.pem (failure)

** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/vidyut.net-0001/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)

rg305 · 2019-02-14 15:54:26 UTC

How can this be?
Certbot would remove anything it created.

This is good and can be useful for PLAN #B --webroot

This is not enough information to get --webroot to work.
You need to also include -w /path/to/your/site/root
[and that needs to be done once for each separate domain path - it can’t guess]

It may also require using (renew or certonly) and/or also using --installer null
Like:
certbot [renew | certonly] --dryrun [--installer null] --weboot \
-w /var/www/one -d site.one \
-w /var/www/two -d site.two \
-w /var/www/three -d site.three \
-w /var/www/four -d site.four \
…
-w /var/www/LAST -d site.LAST

Vidyut · 2019-02-14 16:22:02 UTC

I made them to create the test files.

I did certbot certonly --webroot (figuring that if it created new certs that would renew, it was worth it) and it asks for the domains and roots again, like it did for the nginx install. Everything seems to go fine till the end - where it says archive directory already exists. So now I will take your idea and add renew and see if that makes it overwrite the archive directory.

certbot renew --dry-run --webroot -w /var/www/vin -d vidyut.info -d www.vidyut.info -w /var/www/vn -d vidyut.net -d www.vidyut.net -w /var/www/aj -d aamjanata.com -d www.aamjanata.com -d wide-aware.com -d www.wide-aware.com -w /var/www/diwdru -d digitalindia.watch -d www.digitalindia.watch -w /var/www/af -d aadhaar.fail -d www.aadhaar.fail -w /var/www/fl -d fekle.in -d www.fekle.in -w /var/www/ajin -d aamjanata.in -d www.aamjanata.in -w /var/www/hs -d homeschoolingindia.in -d www.homeschoolingindia.in -w /var/www/ni -d nisarga.info -d www.nisarga.info -w /var/www/map -d india.aamjanata.com Saving debug log to /var/log/letsencrypt/letsencrypt.log Currently, the renew verb is capable of either renewing all installed certificates that are due to be renewed or renewing a single certificate specified by its name. If you would like to renew specific certificates by their domains, use the certonly command instead. The renew verb may provide other options for selecting certificates to renew in the future.

=============================
certbot certonly --dry-run --webroot -w /var/www/vin -d vidyut.info -d www.vidyut.info -w /var/www/vn -d vidyut.net -d www.vidyut.net -w /var/www/aj -d aamjanata.com -d www.aamjanata.com -d wide-aware.com -d www.wide-aware.com -w /var/www/diwdru -d digitalindia.watch -d www.digitalindia.watch -w /var/www/af -d aadhaar.fail -d www.aadhaar.fail -w /var/www/fl -d fekle.in -d www.fekle.in -w /var/www/ajin -d aamjanata.in -d www.aamjanata.in -w /var/www/hs -d homeschoolingindia.in -d www.homeschoolingindia.in -w /var/www/ni -d nisarga.info -d www.nisarga.info -w /var/www/map -d india.aamjanata.com

gives same original result - timeouts on acme-challenge

==============
Thought I’d generate a new cert if that is what it takes, so took out dry run.

`#certbot certonly --webroot -w /var/www/vin -d vidyut.info -d www.vidyut.info -w /var/www/vn -d vidyut.net -d www.vidyut.net -w /var/www/aj -d aamjanata.com -d www.aamjanata.com -d wide-aware.com -d www.wide-aware.com -w /var/www/diwdru -d digitalindia.watch -d www.digitalindia.watch -w /var/www/af -d aadhaar.fail -d www.aadhaar.fail -w /var/www/fl -d fekle.in -d www.fekle.in -w /var/www/ajin -d aamjanata.in -d www.aamjanata.in -w /var/www/hs -d homeschoolingindia.in -d www.homeschoolingindia.in -w /var/www/ni -d nisarga.info -d www.nisarga.info -w /var/www/map -d india.aamjanata.com
IMPORTANT NOTES:

Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/vidyut.info-0001/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/vidyut.info-0001/privkey.pem
Your cert will expire on 2019-05-15. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew all of your certificates, run
“certbot renew”
If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let’s Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le`
===================================
YAY!!!
====================================
certbot renew --dry-run
=======================
Timeout - neither the old nor new cert renewed. Now what.
======================
Now what.

Vidyut · 2019-02-14 16:22:48 UTC

I can’t seem to get the formatting right.

rg305 · 2019-02-14 20:16:59 UTC

So this worked!

Please show:
certbot certificates

and show:
ls -l /etc/letsencrypt/renewal/

Vidyut · 2019-02-15 03:53:18 UTC

certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/vidyut.info.conf produced an unexpected error: renewal config file {} is missing a required file reference. Skipping.

Found the following certs:
Certificate Name: vidyut.info-0001
Domains: vidyut.info aadhaar.fail aamjanata.com aamjanata.in digitalindia.watch fekle.in homeschoolingindia.in india.aamjanata.com nisarga.info vidyut.net wide-aware.com www.aadhaar.fail www.aamjanata.com www.aamjanata.in www.digitalindia.watch www.fekle.in www.homeschoolingindia.in www.nisarga.info www.vidyut.info www.vidyut.net www.wide-aware.com
Expiry Date: 2019-05-15 15:16:22+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/vidyut.info-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/vidyut.info-0001/privkey.pem
Certificate Name: vidyut.net-0001
Domains: vidyut.net aadhaar.fail aamjanata.com aamjanata.in digitalindia.watch fekle.in homeschoolingindia.in map.aamjanata.com nisarga.info vidyut.info wide-aware.com www.aadhaar.fail www.aamjanata.com www.aamjanata.in www.digitalindia.watch www.fekle.in www.homeschoolingindia.in www.nisarga.info www.vidyut.info www.vidyut.net www.wide-aware.com
Expiry Date: 2019-05-13 22:34:05+00:00 (VALID: 87 days)
Certificate Path: /etc/letsencrypt/live/vidyut.net-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/vidyut.net-0001/privkey.pem

The following renewal configurations were invalid:
/etc/letsencrypt/renewal/vidyut.info.conf

ls -l /etc/letsencrypt/renewal/
total 8
-rw-r–r-- 1 root root 1340 Feb 14 21:46 vidyut.info-0001.conf
-rw-r–r-- 1 root root 0 Feb 14 21:29 vidyut.info.conf
-rw-r–r-- 1 root root 532 Feb 13 05:04 vidyut.net-0001.conf

rg305 · 2019-02-15 04:03:55 UTC

This file is empty and can be deleted.