Problem with Certbot renew

That's not true:

server ~ # certbot certificates
(...)
  Certificate Name:   le-test-01.example.com
    Serial Number:    ...
    Domains:          le-test-01.example.com
    Expiry Date:      2021-01-04 09:35:25+00:00 (INVALID: TEST_CERT)
    Certificate Path: /etc/letsencrypt/live/le-test-01.example.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/le-test-01.example.com/privkey.pem
(...)
server ~ # certbot certonly -a apache --cert-name le-test-01.example.com -d le-test-01.example.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Unable to read ssl_module file; not disabling session tickets.
Plugins selected: Authenticator apache (1.8.0.dev0), Installer None (None)
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/le-test-01.example.com.conf)

What would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Keep the existing certificate for now
2: Renew & replace the cert (may be subject to CA rate limits)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 

The output is different when adding a domain, but it still asks you what to do:

server ~ # certbot certonly -a apache --cert-name le-test-01.example.com -d le-test-01.example.com -d le-test-02.example.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Unable to read ssl_module file; not disabling session tickets.
Plugins selected: Authenticator apache (1.8.0.dev0), Installer None (None)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
You are updating certificate le-test-01.example.com to include new
domain(s):
+ le-test-02.example.com

You are also removing previously included domain(s):
(None)

Did you intend to make this change?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(U)pdate cert/(C)ancel: 

So I don't really understand your fascination with --keep-until-expiring.. Or I'm really missing something.. (But I don't think I do, as the examples above clearly state.)

2 Likes

Interesting... I was fairly certain I had tested this via several cases here in Help. Thank you so much for taking the time to test this.

The certbot people really need to clarify the official guide based on what I've quoted and what you've experienced. We both already know that the official guide needs a good scrubbing though.

Why give innocent users (who are naive to the perils of rate limits) a rope (the option) to hang themselves (by hitting a rate limit)? :slightly_smiling_face:

It also helps with scripting and thus automation (non-interactive usage).

You'll notice that I also explicitly specify "run" even though it's the default (because IMNSHO it is bad to have a default).

1 Like

Normally, when someone handles a rope, he/she doesn't hang themselves: it only happens to the most clumsy people by accident (i.e.: someone who doesn't read or didn't diligently read the documentation). More than a million certificates are issued daily. Not all of them will be new certs of course, but LE is big. From all those users out there, only a few end up here because they didn't read the documentation or didn't understand what they were doing.

All in all, I think we shouldn't overreact and overly criticise the current documentation.

2 Likes

I do not know if this is necessarily the best way to do it, but because my gTLD (.dev) is on Google's mandatory HSTS hosts list; I have found that I have to install the certificate in an odd way. Let's Encrypt seems to use port 80 to assign a certficate for any given subdomain any time it is run for that subdomain for the first time - I found this out because my firewall was (purposely) blocking traffic on port 80. Anyway, if Apache is running when I run the certbot --apache command, then the assignment fails because Let's Encrypt won't connect to the website. I got around this issue by temporarily stopping the Apache2 systemd service and then running certbot certonly -d example.codedragon.dev. Finally after that, I restart my Apache2 service making sure the vHost is linked in /etc/apache2/sites-enabled and run certbot again so that my automatic renew command via Cron works: certbot --apache -d example.codedragon.dev and choose the option to reinstall the certificate (not replacing it). Otherwise my crontab command run by root 0 0 * 9 * certbot --apache -q renew won't work. Most of this information I found out the hard way, but I am persistent. Also I like simplicity, so I know it is simple, but it works. I am honestly not sure why the certificate fails when apache is running (it may be my webserver's own HSTS configuration that I set up for redundancy), but I figured it out. Maybe this can help you towards understanding more about your own situation with issuing certificates using Certbot.

2 Likes

@CodeDragon5793

What follows might seem like a critique. It's not, per se. I just want to make sure you have the right understanding and best processes for you. :slightly_smiling_face:


To my knowledge HSTS does not have any impact on the http-01 authentication process.


The HTTP-01 challenge can only be done on port 80. Allowing clients to specify arbitrary ports would make the challenge less secure, and so it is not allowed by the ACME standard.

Our implementation of the HTTP-01 challenge follows redirects, up to 10 redirects deep. It only accepts redirects to “http:” or “https:”, and only to ports 80 or 443. It does not accept redirects to IP addresses. When redirected to an HTTPS URL, it does not validate certificates (since this challenge is intended to bootstrap valid certificates, it may encounter self-signed or expired certificates along the way).


Did you accidentally select the standalone authenticator instead of the apache authenticator?

At this point, I'm assuming that you selected the standalone authenticator.


Perfect.

Fair enough. The certbot apache installer could have done this for you.


That command would have certbot modify your apache configuration to "install" the certificate, so you should not have needed to modify the configuration yourself.

Thank you, thank you, thank... you, for reinstalling your existing certificate though! :clap: So many people don't do that and it drives us crazy.


:thinking:

Pretty sure that your crontab should just be:
certbot renew -q

The Apache pieces should have been saved in your renewal configuration file.

It could have been a whole lot simpler. :grin:


I'm glad you got a process working, but in my eyes you have presented a good example of what not to do. I have a strong feeling that you were trying to use the standalone authenticator while Apache was running. You should be using either the apache authenticator or the webroot authenticator.


The moral of this story is...

1 Like

[my apologies, in advance, on the lengthy read]

If --standalone was used to obtain the cert, some "extra" step(s) may have been taken manually in order for that request to have completed correctly; Namely: Stopping and restarting the apache web server.

We should be able to look into the renewal config to see if that is the case, or do a test renewal (with --dry-run), to confirm.
If so, then the automated renewal process is missing those steps taken manually and will not complete, as expected, in the coming months.

In review: If apache failed you with a .dev gTLD, while certbot spun a temporary web server and was able to do so, then the actual problem has nothing to do with the name being requested - and has all to do with the difference in the configurations in use by the two web servers.
So, you would do well by reviewing your apache's config.
Here I re-re-re-re-re-repeat myself on this forum: Apache is notorious for running at all costs.
[I've said it dozens of times and will probably say it a dozen more]
Apache will do everything possible to run with a config that would otherwise be rejected by almost any other web server.
Errors will always creep into things over time and repeated manual modification - this is normal; as is with just about everything else in life.
Overlooking those errors is not; apache, although with the best of intentions, seems to look the other way a bit too much at times. And this may be one of those times.
Please start by checking your apache config with: apachectl -S
If enough errors can be corrected (to the point where certbot and apache can be used to get/renew your certs), this process can then be easily automated without having to stop and then restart apache.

READERS: Get involved and participate: If you read something you like, then click to like it :heart:

2 Likes

That's exactly right, HSTS only affects browsers, not Let's Encrypt's validation software.

3 Likes

Yeah, I was confused because I thought that HSTS for this gTLD only affected the browsers.

I do not try to run standalone certificates while using the --apache flag. That was a misinterpretation on your part. The --apache flag fails domain ownership verification, so I turn to installing the cert as a standalone on a temporary certbot webserver. Then, because I do need it for Apache (and not the standalone), I rerun certbot for Apache and reinstall the cert. All of this is done while port 80 is open; then I close port 80 back off. I was explaining too, that Port 80 needs to be open when you generate and install a new certificate - that I had discovered this the hard way so to speak; you weren't telling me any new information by your little moral of the story tidbit at the end there.

I guess I could make the crontab command a little simpler; but, in my opinion, explicit is generally better than implied. As far as keeping port 80 open goes, I find that when renewing certificates, having port 80 closed doesn't seem to affect anything since my apache config itself is designed to hopefully only answer requests on port 443. Thus I only open 80 during new certificate assignments.

1 Like

Did you read this?


Is there a reason you're not forwarding http requests to https? Even if you're using an HSTS TLD, this does not stop anyone (including Let's Encrypt) from making http requests.

2 Likes

Yeah, my point is that my server doesn't fail the HTTP-01 authentication though because I figured out why and just opened the port anyway. Also there is likely no way that someone will not be redirected to the HTTPS version of my site because of the redundant ways HSTS is set up.

1 Like

Do you automate the opening of port 80 during renewal then?

2 Likes

Why did it fail? What OS and apache version are you using? I'm curious from a bug perspective for the benefit of the certbot developers.

2 Likes

Yes.

If I remember correctly it failed the part where it verifies that I am the owner of the domain. Which is why I chalked it up to my HSTS config. To be honest, I am new to webserver administration. I am doing all of this on my personal website as much to learn and teach myself new skills and understand stuff that I didn't before as it is to have a production site on the open web. That being said, my Apache config could have wrong configuration (as opposed to a bug in certbot) since I do all of this more as a hobby than for a business; thus I have to do everything I talk about here it in my free time around work and school (college). As for the OS, I am using Debian 10 Buster and Debian's customized version of Apache 2.4.38-3 (cause Debian has to customize every piece of software in its repositories :roll_eyes:)

2 Likes

It's cool to be new. :slightly_smiling_face: We all are at some point. I'm just wondering why the apache authenticator failed for you. :thinking: If you had port 80 open then I wonder if your apache has different locations for things. You're not using Bitnami are you?

3 Likes

If you want, I can create a new topic and show you a vHost example from one of my site's subdomains. I also recycle vHosts so most of them are identical, except for the www.codedragon.dev and www.linuxdragon.dev vHosts oh and one other where I was following a tutorial to set up something but realized I was doing the wrong thing and have to restart. Also if you go check out my sites there really isn't much to see there yet. I am redesigning the first one altogether so I have moved everything to my laptop as I redo it from the ground up. I don't have to do that, but the design was embarrassing me xD. I don't know what Bitnami is, so probably not.

2 Likes

Sometimes the Apache configuration doesn't play very well with the apache authenticator plugin unfortunately. However, I wouldn't recommend stopping the webserver, using standalone and then starting the webserver again: too much downtime/hassle. I would recommend the webroot plugin as authenticator plugin if the apache plugin doesn't work properly and it's too hard to debug why.

3 Likes

So here is the issue cause I went back and looked at everything. I have apache configured to point to the path that my certificates are located at using

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/www.linuxdragon.dev/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.linuxdragon.dev/privkey.pem

Oof it is has been awhile since I looked at this. The issue is that when I am trying to issue certificates for the first time, those do not exist, and Apache doesn't like that so it just decides to kill itself with an error. So to get around that I had been installing the certificate as a standalone since Apache must be running for the --apache flag to work. That created the paths that I need, so now I can restart apache. But I found that this process made a renew command fail so I realized I can just attempt to reinstall the existing certificate. Now here's the part where I came to the HSTS conclusion. Someone was helping me set this up originally. This is in large part based on his own server's configuration. His website doesn't have HSTS forced for the gTLD. So I made a correlation=causation conclusion when I was dumber. Now that I am looking at it, I think it may actually be because I was probably (and still probably am) running a different version of Apache since his server is powered by arch linux.

2 Likes

I agree wholeheartedly with @Osiris's webroot recommendation (followed by doing a graceful reload of apache using --deploy-hook). Honestly, it's often better to use than the apache authenticator and plugin, IMO.

2 Likes

My webroot path being /var/www/example-subdomain-dir?

1 Like

Most often, the webroot-path is the same as the DocumentRoot directive in the VirtualHost.

4 Likes