Using port 443 for renewal after TLS-SNI is disabled

Maybe I used the wrong word. What I meant was that these attacks aren't going on 100% of the time. Such attacks are bad but last only hours, days at worst, because the attacker has to pay a bot herder per hour.

In our case, the attacks do not happen on command; they aren't from a botnet. They are from individual machines infected with a virus that has no command-and-control server and which the author does not control. Nobody controls them. Which means that the attacks won't disappear completely until all such machines are disinfected (far chance) and that they become more or less intensive unpredictably (probably as new machines are infected or the old ones are cleaned).

Occasionally there is no problem (as, apparently, right now). Occasionally, our site is not accessible via HTTP for days and weeks.

Yes, but currently I have configured a crontab job to try to renew the certificate monthly. If on that date the port-80 DDoS is strong, the renewal will fail (has happened in the past at least once). Are you saying that I should set up the crontab job to try to renew the certificate daily?

I think you should delete 'em all. The trick is doing it safely...

Certbot also has an acme module, and maybe josepy.

Plus there could be other dependencies.

Yeah, I agree.

I didn't know pip got an uninstall command! :exploding_head:

In the case of the Certbot packages, the Certbot-installed cron job deactivates itself if systemd is in use. I'd leave it alone (though deleting it wouldn't do any harm either).

If you have a second cron job of your own, you can remove that, yes.

Unless the systemd timer is disabled.

It's recommended to run it twice a day. But, since you use the Certbot package, it should already be set up that way, so you don't need your own cron job.

1 Like

You can execute the cron job every hour. If the certificate is valide and longer then 30 days valide, nothing happens.

If the certificate expires in max. 30 days, then your server tries to renew it. Or start with the renew after 45 days.

It does, but I don't like what it displays and was afraid to approve it:

Uninstalling certbot-0.25.1:
  Would remove:
    /usr/bin/certbot
    /usr/local/bin/certbot
    /usr/local/lib/python2.7/dist-packages/certbot-0.25.1.dist-info/*
    /usr/local/lib/python2.7/dist-packages/certbot/*
Proceed (y/n)?

See? It would remove /usr/bin/certbot too. Maybe just uninstall everything (sudo -H pip uninstall certbot && sudo apt remove python-certbot-nginx) and try re-installing it again with apt?

Not sure I understand. I have created the cron job myself with sudo crontab -e:

@monthly /usr/bin/certbot renew && /usr/sbin/nginx -t && /usr/sbin/nginx -s reload

Should I remove that?

Anyway, the point is probably moot, since Certbot doesn't support TLS-ALPN-01; i.e., I should be looking to switch to a different ACME client... :frowning:

Yikes. If pip uninstall doesn't do anything very special, maybe just delete the stuff with rm?

(Or mv!)

There should already be a cron job in /etc/cron.d/certbot. (But it doesn't run.)

And the systemd timer.

You should delete your own cron job and configure a hook to handle reloading Nginx, or disable the other two. Having them all running is kind of problematic.

Yeah... If you switch to another ACME client at least you don't have to be careful about breaking Certbot...

Sigh… This thing is completely, utterly fucked up! :frowning:

OK, I did the following:

$ sudo -H pip uninstall certbot
Uninstalling certbot-0.25.1:
  Would remove:
    /usr/bin/certbot
    /usr/local/bin/certbot
    /usr/local/lib/python2.7/dist-packages/certbot-0.25.1.dist-info/*
    /usr/local/lib/python2.7/dist-packages/certbot/*
Proceed (y/n)? y
  Successfully uninstalled certbot-0.25.1
$ sudo apt remove python-certbot-nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  python-certbot-nginx
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 10.2 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 259408 files and directories currently installed.)
Removing python-certbot-nginx (0.28.0-1+ubuntu16.04.1+certbot+3) ...
$ sudo apt-get install python-certbot-nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python-certbot-nginx
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/2,736 B of archives.
After this operation, 10.2 kB of additional disk space will be used.
Selecting previously unselected package python-certbot-nginx.
(Reading database ... 259407 files and directories currently installed.)
Preparing to unpack .../python-certbot-nginx_0.28.0-1+ubuntu16.04.1+certbot+3_all.deb ...
Unpacking python-certbot-nginx (0.28.0-1+ubuntu16.04.1+certbot+3) ...
Setting up python-certbot-nginx (0.28.0-1+ubuntu16.04.1+certbot+3) ...

And now there is no certbot command anywhere?! Not in /usr/bin, nor anywhere else in the $PATH!

Are the installation instructions wrong? Did I miss something?

The command isn’t part of that package. Try reinstalling certbot.

$ dpkg -S /usr/bin/certbot
certbot: /usr/bin/certbot
$ dpkg -S /usr/bin/certbot
certbot: /usr/bin/certbot
$  certbot --version
certbot: command not found

What now?!

Reinstall certbot.

dpkg -S just shows ownership information, it doesn’t fix it.

How do I re-install it?! The instructions on the EFF site say to install python-certbot-nginx. You said that this doesn’t install the command? What installs the command?

certbot is a dependency of python-certbot-nginx. Certbot’s made up of half a dozen packages.

Try “sudo apt-get install --reinstall certbot”.

Edit: I typed apt instead of apt-get.

Ah, I see! OK, thanks, this at least re-installed a working certbot that is the latest version (0.28.0) and nothing else.

I’m using Ubuntu 16.04, so apt works just fine; apt-get is for the older versions.

OK, now at least I am back to where I started (minus the mess of different versions). Thanks for your help.

Now I have to go read up on Lego and see if I can get it to work on the Linux server…

1 Like

Yeah. That's why I accidentally typed it. :smile:

I think "sudo apt install --reinstall certbot" works too, but I would have to check the documentation.

If you are using python2.7
sudo -H python2 -m pip uninstall certbot
sudo -H python2 -m pip install certbot
sudo -H python2 -m pip uninstall certbot-apache

It’s really dangerous to use apt and pip to manage the same software. They’ll delete each other’s files, install different versions, and generally break things.

3 Likes

The default behavior is that certbot renew tries to renew any certificates that expire in less than 30 days, and no other certificates. Therefore, it's safe (and recommended) to run this command very frequently. We recommend running certbot renew twice per day, every day, and not scheduling it to run only on specific days. It will result in frequent renewal attempts during the 30-day expiry window if no renewal has succeeded, and once a renewal has succeeded, no renewal attempts on that certificate thereafter until the following 30-day expiry window.

You can change this interval with the renew_before_expiry option in the configuration file in /etc/letsencrypt/renewal associated with an individual certificate. For example, you could set renew_before_expiry = 60 days so that Certbot attempts to renew certificates when they're two months from expiry rather than one month from expiry. Anything up to the extreme of renew_before_expiry = 82 days is probably safe with respect to our rate limits as long as you only have a small number of different certificates. This would mean that Certbot would try to renew certificates very frequently, but would pause for about a week after each successful renewal, which is enough time to avoid running into our rate limits.

If you're certain that you don't want to have a port 80 listener the rest of the time, you could simply make your regular web server not listen on port 80, and then use -a standalone in Certbot (authenticator = standalone in the renewal configuration file). In that case, Certbot will create its own port 80 listener which will only exist during actual renewal attempts, typically for less than one minute every 60 days. This is only appropriate on systems where no other software is normally listening on port 80.

We're really sorry about this configuration change being necessary, but I think that there ought to be some workaround that will work for your environment. As you noted, reluctance to use port 80 for a reason other than an ISP firewall is relatively uncommon.

If you don't have an easy way to get a Certbot DNS plugin for your particular DNS provider but you want to use the DNS challenge instead of ever using port 80, you might want to try acme.sh (https://acme.sh/), which supports a lot of DNS providers out-of-the-box.

2 Likes

This thread reminds me that the letsencrypt community is amazingly supportive, patient and thorough. I created an account just to give a shout out to everyone helping on this thread, good job / well done!

2 Likes

A post was split to a new topic: Letsencrypt with Docker - replace Tls-sni-01 - validation

This sounds like a good reason to change DNS providers to me. Or keep your poor provider for most of your records and just CNAME the challenge records to another provider that is supported by your ACME client.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.