It produced this output:
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Pre-hook command already run, skipping: service apache2 stop
Simulating renewal of an existing certificate for jns.enterprises
Performing the following challenges:
http-01 challenge for jns.enterprises
Waiting for verification...
Challenge failed for domain jns.enterprises
http-01 challenge for jns.enterprises
Cleaning up challenges
Failed to renew certificate jns.enterprises with error: Some challenges have failed.
Yes. I got thinking about what you said about the redirects, and remembered I added this to my Apache config file back when I was getting everything set up - could it be an issue?
#Used for LetsEncrypt validation
AccessFileName .htaccess
AliasMatch ^/.well-known/acme-challenge/(.*)$ /var/www/html/.well-known/acme-challenge/$1
Alias /.well-known/acme-challenge /var/www/html/.well-known/acme-challenge/
<Directory /var/www/html/.well-known/acme-challenge>
Order allow,deny
Allow from all
Options None
AllowOverride None
ForceType text/plain
RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"
</Directory>
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator standalone, Installer None
Running pre-hook command: service apache2 stop
Simulating renewal of an existing certificate for football.jns.enterprises
Performing the following challenges:
http-01 challenge for football.jns.enterprises
Waiting for verification...
Cleaning up challenges
Where did you get that list? There should actually be four domains on the certificate, not three. I probably don't need a separate cert, but the way the renewal is erroring out, in the terminal it doesn't look like the cert is being renewed for the root domain.
No, your football domain has only ever been on its own cert. You can see that at https://crt.sh just know that crt.sh system is often badly delayed recently but it is still good for long-term history.
I was using the CT log system at another system (censys.io) but that requires an account. There are other CT log display providers (search google).
Looks like you are correct - I didn't realize I had so many. Back when I was setting things up I knew even less than I do now (which isn't saying much), so looks like I duplicated some efforts. As far as cleaning this mess up...is there any benefit to separate certs vs all on one?
The best method is one that works well for you. I prefer to have one cert for each unique VirtualHost. For me that is easiest to manage.
Example, if example.com and www.example.com were in same VirtualHost (with one as a ServerAlias) they would be together on one cert.
I think your Certbot list is not complete. There should have been one with several domain names.
And, I think the methods you have used for some of them differ from the others. Once you have chosen how you want to proceed we can help straighten out the renewal methods.
UPDATE:
Oops, my bad. I just saw (properly) that one of those certs was for 3 domains. Apart from that the rest of my comment stands.
I like the VirtualHost suggestion. To go that route:
To me, it looks like football and webmail are fine (added full text of webmail below, noticed it got cut off in the previous copy and paste).
Looks like I would need to revoke and reissue the root and recipes certificates? I can look up the syntax for that fairly easily I think.
Should I be concerned that the root didn't renew, but the recipe certificate (with the root on it) did? Not sure why one would work but not the other. Could it be because of the multiple domains? If so, I could add a www to the root to just make it work.
Hadn't we established that these lines during the dry run were bad, specifically the "challenge failed" piece?
Waiting for verification...
Challenge failed for domain jns.enterprises
http-01 challenge for jns.enterprises
Cleaning up challenges
Failed to renew certificate jns.enterprises with error: Some challenges have fai led.
Yes, should be concerned. I don't think it is related to the domain name but instead the method used in the renewal conf file for the respective certs.
All of your assumptions are correct, and looks like you nailed it on authentication method - webroot vs standalone. If i just change the renewal params to match those in recipe, should that fix the issue? Assuming it does, how do I shrink recipes?
jns.enterprises
# renew_before_expiry = 30 days
version = 1.12.0
archive_dir = /etc/letsencrypt/archive/jns.enterprises
cert = /etc/letsencrypt/live/jns.enterprises/cert.pem
privkey = /etc/letsencrypt/live/jns.enterprises/privkey.pem
chain = /etc/letsencrypt/live/jns.enterprises/chain.pem
fullchain = /etc/letsencrypt/live/jns.enterprises/fullchain.pem
# Options used in the renewal process
[renewalparams]
authenticator = webroot
account = cca51956f8d5f563f19f06abfdfc55fc
server = https://acme-v02.api.letsencrypt.org/directory
post_hook = service apache2 start
pre_hook = service apache2 stop
pref_challs = http-01,
[[webroot_map]]
jns.enterprises = /var/www/html
recipes
version = 1.12.0
archive_dir = /etc/letsencrypt/archive/recipes.jns.enterprises
cert = /etc/letsencrypt/live/recipes.jns.enterprises/cert.pem
privkey = /etc/letsencrypt/live/recipes.jns.enterprises/privkey.pem
chain = /etc/letsencrypt/live/recipes.jns.enterprises/chain.pem
fullchain = /etc/letsencrypt/live/recipes.jns.enterprises/fullchain.pem
# Options used in the renewal process
[renewalparams]
authenticator = standalone
account = cca51956f8d5f563f19f06abfdfc55fc
post_hook = service apache2 start
pre_hook = service apache2 stop
server = https://acme-v02.api.letsencrypt.org/directory
pref_challs = http-01,
That's not the best approach. The webroot is much better than standalone when you have a working web server (which you do).
We normally caution against modifying the renewal conf files by hand but in this case you could make a backup of the jns.enterprises.conf file and then remove the 2 lines for the pre_hook and post_hook.
If the dry-run works just remove --dry-run to get the production cert. No other changes are needed and it should auto-renew going forward
Shrinking recipes is entirely different. I'll need a minute but post back any results from above. Update: After further study I would like to see results of above before working on recipes.