Creating certificates with --nginx works. Renew or certonly doesn't


#6

Renewing failed, using certbot to create slightly modified certs failed. Issuing new certificates failed because directory already exists.

Certs were due to explire. Emergency situation.

So I deleted all the certs, brought the site down to http and reissued one combined cert for anything.

Cert issued with no hassles, but new one also fails the certbot renew --dry-run


#7

renew_before_expiry = 30 days

version = 0.28.0
archive_dir = /etc/letsencrypt/archive/vidyut.net-0001
cert = /etc/letsencrypt/live/vidyut.net-0001/cert.pem
privkey = /etc/letsencrypt/live/vidyut.net-0001/privkey.pem
chain = /etc/letsencrypt/live/vidyut.net-0001/chain.pem
fullchain = /etc/letsencrypt/live/vidyut.net-0001/fullchain.pem

Options used in the renewal process

[renewalparams]
authenticator = nginx
account = [not sure if ok to share]
installer = nginx
server = https://acme-v02.api.letsencrypt.org/directory


#8

The good news:

  • version = 0.28.0
  • reissued one combined cert for anything
  • Expiry Date: 2019-05-13 22:34:05+00:00 (VALID: 89 days)
  • all names resolve to same IPv4 address: 128.199.212.123
  • Connecting to 128.199.212.123|:80… connected.

The bad news:

  • new one also fails the certbot renew –dry-run

Based on the good information, --dry-run should be working.
We need to better understand why it fails…
Please show (additional detailed) output of:
certbot renew --dry-run -vvv


#9

https://pastebin.com/PXvbvGQQ


#10

I’m not sure why…
But the nginx installer is creating server blocks and modifications that don’t… “work” with the rest of the config.

Please upload the whole nginx config form the output of:
nginx -T

[the reason/explanation/fix will be found somewhere in there]

In any event, we can always fallback to using --webroot as PLAN #B.
[we have 89 days to get this right - LOL]


#11

Ah ok. That may explain it. I have a different config. One file with all the server names and their roots. Another with the ssl, locations (since it is shared), etc that gets included in each server block. So if the plugin is trying to physically alter the server block… it may not find the parts it expects?

Strange that I’m running into this issue only now. Been using this method from the start of using letsencrypt and they were renewed using a cron.


#12

Can produce the nginx -T, but it will unnecessarily expose my security configurations unless I edit a lot.

Basically a gist of what is going on is:

Server blocks are all in one file in sites-enabled
Something like

server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name .one;
root /var/www/one;
include global/one-type.conf;
}
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name .two;
root /var/www/two;
include global/another-type.conf;
}
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name .one;
root /var/www/three;
include global/one-type.conf;
}
server {
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name .four;
root /var/www/four;
include global/one-type.conf;
}

The included file will have one out of 2-3 variations of
ssl_certificate /etc/letsencrypt/live/vidyut.net-0001/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/vidyut.net-0001/privkey.pem;
ssl_more-stuff* __ciphers, dhparam,stapling,whatever;
[add_header stuff]
[cache setup stuff]
location / {
try_files $uri $uri/ /index.php?$args;
}
location ~ /.well-known {
allow all;
}

[more locations for php, caching, etc]

[includes fastcgi_params]

**This is very convenient for updating configurations for servers - best practices, updating ssl config for security, etc. Edit one or two files and done instead of updating each server block and waaaay less lines of code to wade through. But obviously it isn’t more convenient than certbot renew vs what I went through :disappointed_relieved: **

So what are the parts I must bring back to the main config file so that letsencrypt can find them? Or does it want me to have separate config files per server?


#13

I think we can modify your config to suit both needs.

First:

  • increase the path specifically to /.well-known/acme-challenge
    [or add another location if you use other /.well-known paths]
  • Add line to same location: default_type text/plain;

Then we test (using site one as example):
[using root /var/www/one]

mkdir /var/www/one/.well-known
mkdir /var/www/one/.well-known/acme-challenge
echo "testing" > /var/www/one/.well-known/acme-challenge/1234

We should be able to view this new file via the Internet at:
http://site-ONE/.well-known/acme-challenge/1234

If that works, we are on our way.
If that fails, then we can still opt for PLAN #B [--webroot]


#14

adding /.well-known/acme-challenge and default_type text/plain; made no difference.

The folders exist in all domains. Can create and access the test file from all domains.

Configuration seems to be an issue for --webroot. Just realized that creating the combined cert with --nginx worked, but it could not install the cert. I did that manually, but thought nothing of it, since I’ve always done letsencrypt certs like that (earlier there was no installer - when I started using)

===================================
#certbot renew --webroot --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/vidyut.net-0001.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for aadhaar.fail
http-01 challenge for aamjanata.com
http-01 challenge for aamjanata.in
http-01 challenge for digitalindia.watch
http-01 challenge for fekle.in
http-01 challenge for homeschoolingindia.in
http-01 challenge for map.aamjanata.com
http-01 challenge for nisarga.info
http-01 challenge for vidyut.info
http-01 challenge for vidyut.net
http-01 challenge for wide-aware.com
http-01 challenge for www.aadhaar.fail
http-01 challenge for www.aamjanata.com
http-01 challenge for www.aamjanata.in
http-01 challenge for www.digitalindia.watch
http-01 challenge for www.fekle.in
http-01 challenge for www.homeschoolingindia.in
http-01 challenge for www.nisarga.info
http-01 challenge for www.vidyut.info
http-01 challenge for www.vidyut.net
http-01 challenge for www.wide-aware.com
Cleaning up challenges
Attempting to renew cert (vidyut.net-0001) from /etc/letsencrypt/renewal/vidyut.net-0001.conf produced an unexpected error: Missing command line flag or config entry for this setting:
Input the webroot for aadhaar.fail:. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/vidyut.net-0001/fullchain.pem (failure)


** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/vidyut.net-0001/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)


1 renew failure(s), 0 parse failure(s)


#15

How can this be?
Certbot would remove anything it created.

This is good and can be useful for PLAN #B --webroot

This is not enough information to get --webroot to work.
You need to also include -w /path/to/your/site/root
[and that needs to be done once for each separate domain path - it can’t guess]

It may also require using (renew or certonly) and/or also using --installer null
Like:
certbot [renew | certonly] --dryrun [--installer null] --weboot \
-w /var/www/one -d site.one \
-w /var/www/two -d site.two \
-w /var/www/three -d site.three \
-w /var/www/four -d site.four \

-w /var/www/LAST -d site.LAST


#16

I made them to create the test files.

I did certbot certonly --webroot (figuring that if it created new certs that would renew, it was worth it) and it asks for the domains and roots again, like it did for the nginx install. Everything seems to go fine till the end - where it says archive directory already exists. So now I will take your idea and add renew and see if that makes it overwrite the archive directory.

certbot renew --dry-run --webroot -w /var/www/vin -d vidyut.info -d www.vidyut.info -w /var/www/vn -d vidyut.net -d www.vidyut.net -w /var/www/aj -d aamjanata.com -d www.aamjanata.com -d wide-aware.com -d www.wide-aware.com -w /var/www/diwdru -d digitalindia.watch -d www.digitalindia.watch -w /var/www/af -d aadhaar.fail -d www.aadhaar.fail -w /var/www/fl -d fekle.in -d www.fekle.in -w /var/www/ajin -d aamjanata.in -d www.aamjanata.in -w /var/www/hs -d homeschoolingindia.in -d www.homeschoolingindia.in -w /var/www/ni -d nisarga.info -d www.nisarga.info -w /var/www/map -d india.aamjanata.com Saving debug log to /var/log/letsencrypt/letsencrypt.log Currently, the renew verb is capable of either renewing all installed certificates that are due to be renewed or renewing a single certificate specified by its name. If you would like to renew specific certificates by their domains, use the certonly command instead. The renew verb may provide other options for selecting certificates to renew in the future.

=============================
certbot certonly --dry-run --webroot -w /var/www/vin -d vidyut.info -d www.vidyut.info -w /var/www/vn -d vidyut.net -d www.vidyut.net -w /var/www/aj -d aamjanata.com -d www.aamjanata.com -d wide-aware.com -d www.wide-aware.com -w /var/www/diwdru -d digitalindia.watch -d www.digitalindia.watch -w /var/www/af -d aadhaar.fail -d www.aadhaar.fail -w /var/www/fl -d fekle.in -d www.fekle.in -w /var/www/ajin -d aamjanata.in -d www.aamjanata.in -w /var/www/hs -d homeschoolingindia.in -d www.homeschoolingindia.in -w /var/www/ni -d nisarga.info -d www.nisarga.info -w /var/www/map -d india.aamjanata.com

gives same original result - timeouts on acme-challenge

==============
Thought I’d generate a new cert if that is what it takes, so took out dry run.

`#certbot certonly --webroot -w /var/www/vin -d vidyut.info -d www.vidyut.info -w /var/www/vn -d vidyut.net -d www.vidyut.net -w /var/www/aj -d aamjanata.com -d www.aamjanata.com -d wide-aware.com -d www.wide-aware.com -w /var/www/diwdru -d digitalindia.watch -d www.digitalindia.watch -w /var/www/af -d aadhaar.fail -d www.aadhaar.fail -w /var/www/fl -d fekle.in -d www.fekle.in -w /var/www/ajin -d aamjanata.in -d www.aamjanata.in -w /var/www/hs -d homeschoolingindia.in -d www.homeschoolingindia.in -w /var/www/ni -d nisarga.info -d www.nisarga.info -w /var/www/map -d india.aamjanata.com
IMPORTANT NOTES:

  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/vidyut.info-0001/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/vidyut.info-0001/privkey.pem
    Your cert will expire on 2019-05-15. To obtain a new or tweaked
    version of this certificate in the future, simply run certbot
    again. To non-interactively renew all of your certificates, run
    “certbot renew”

  • If you like Certbot, please consider supporting our work by:

    Donating to ISRG / Let’s Encrypt: https://letsencrypt.org/donate
    Donating to EFF: https://eff.org/donate-le`
    ===================================
    YAY!!!
    ====================================
    certbot renew --dry-run
    =======================
    Timeout - neither the old nor new cert renewed. Now what.
    ======================
    Now what. :sob:


#17

I can’t seem to get the formatting right.


#18

So this worked!

Please show:
certbot certificates

and show:
ls -l /etc/letsencrypt/renewal/


#19

certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/vidyut.info.conf produced an unexpected error: renewal config file {} is missing a required file reference. Skipping.


Found the following certs:
Certificate Name: vidyut.info-0001
Domains: vidyut.info aadhaar.fail aamjanata.com aamjanata.in digitalindia.watch fekle.in homeschoolingindia.in india.aamjanata.com nisarga.info vidyut.net wide-aware.com www.aadhaar.fail www.aamjanata.com www.aamjanata.in www.digitalindia.watch www.fekle.in www.homeschoolingindia.in www.nisarga.info www.vidyut.info www.vidyut.net www.wide-aware.com
Expiry Date: 2019-05-15 15:16:22+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/vidyut.info-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/vidyut.info-0001/privkey.pem
Certificate Name: vidyut.net-0001
Domains: vidyut.net aadhaar.fail aamjanata.com aamjanata.in digitalindia.watch fekle.in homeschoolingindia.in map.aamjanata.com nisarga.info vidyut.info wide-aware.com www.aadhaar.fail www.aamjanata.com www.aamjanata.in www.digitalindia.watch www.fekle.in www.homeschoolingindia.in www.nisarga.info www.vidyut.info www.vidyut.net www.wide-aware.com
Expiry Date: 2019-05-13 22:34:05+00:00 (VALID: 87 days)
Certificate Path: /etc/letsencrypt/live/vidyut.net-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/vidyut.net-0001/privkey.pem

The following renewal configurations were invalid:
/etc/letsencrypt/renewal/vidyut.info.conf


ls -l /etc/letsencrypt/renewal/
total 8
-rw-r–r-- 1 root root 1340 Feb 14 21:46 vidyut.info-0001.conf
-rw-r–r-- 1 root root 0 Feb 14 21:29 vidyut.info.conf
-rw-r–r-- 1 root root 532 Feb 13 05:04 vidyut.net-0001.conf


#20

This file is empty and can be deleted.


#21

The two certificates are ALMOST identical - in that they cover all the same names except:

Certificate Name: vidyut.info-0001
Has domain: india.aamjanata.com

Certificate Name: vidyut.net-0001
Has domain: map.aamjanata.com

So you need to ensure those two sites are all using their correct cert.
For now, all other sites can use either cert.

At some point, you may want to either:

  • consolidate them all into one new cert (and delete these two certs)
  • separate the sites into grouped certs or even individual certs

[whichever makes most sense to you and your configuration and workflow]


#22

Before running the --dry-run test again…
We need to first ensure we know which certs are actually being used.
Please show:
grep -Eri 'ssl_cert|server_name' /etc/nginx


#23

I thought I had… and it wasn’t empty - it had the cert that expired which triggered this emergency. It seems to have got recreated and empty. I wonder why and if that has something to do with failed renewals…

The output of this is very convoluted - it goes through sites-available and lists out non-existent certificates from obsolete configs also.

All the servers are currently using /etc/letsencrypt/live/vidyut.info-0001/fullchain.pem; (and privkey).

The india/map is a test site thing. Not really using it for anything much so far. Ignore the difference.


#24

One thing worth investigating may be that certbot certonly --dry-run failed, but certbot certonly worked.

Certbot renew is obviously not working right now with new cert, but if it is issued using webroot, maybe certbot renew will work when cert is due for renewal, but fails for certbot renew --dry-run similar to certonly?

Something different about how --dry-run handles authentication?

If so, short of waiting for certs to be due to expire, is there any other way I can make sure that it will work?


#25

Then you can (and should) delete the other cert:
certbot delete --cert-name vidyut.net-0001

Then recheck certs with:
certbot certificates

and ensure this folder looks OK (please show):
ls -l /etc/letsencrypt/renewal/