Using port 443 for renewal after TLS-SNI is disabled

Help
1

I am sorry, but I am totally confused by this whole mess… :frowning:

Yes, I have received the e-mail that TLS-SNI-01 support will disappear. The problem is, certificate renewal via TLS is a must for us.

Our port 80 is under sporadic DDoS attack, so certificate renewal via HTTP-01 tends to fail, and any changes to the DNS configuration can be done only manually using a web form, so automated renewal via DNS-01 is impossible.

Furthermore, we are using HSTS, so dropping HTTPS support is impossible, too. We even considered switching to a different (paid) certificate provider, but, sadly, that’s not a practical option for us at this time, either.

So, about the only solution that remains is certificate renewal using the TLS-ALPN-01 challenge. Will/does Certbot support that? If not, which ACME client would you suggest for Ubuntu 16.04+nginx?

Regarding the updating to the latest version of Certbot, I have a problem with this too. First of all, when I run

certbot --version

the output I get is

certbot 0.25.1

so I assume that I don’t have the latest version. I’ve followed the instructions for installing Certbot on Ubuntu 16.04 and it seems that version 0.28 is installed:

$  sudo apt-get install python-certbot-nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python-certbot-nginx
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 2,736 B of archives.
After this operation, 10.2 kB of additional disk space will be used.
Get:1 http://ppa.launchpad.net/certbot/certbot/ubuntu xenial/main amd64 python-certbot-nginx all 0.28.0-1+ubuntu16.04.1+certbot+3 [2,736 B]
Fetched 2,736 B in 0s (15.4 kB/s)
Selecting previously unselected package python-certbot-nginx.
(Reading database ... 259322 files and directories currently installed.)
Preparing to unpack .../python-certbot-nginx_0.28.0-1+ubuntu16.04.1+certbot+3_all.deb ...
Unpacking python-certbot-nginx (0.28.0-1+ubuntu16.04.1+certbot+3) ...
Setting up python-certbot-nginx (0.28.0-1+ubuntu16.04.1+certbot+3) ...

However, when I run certbot with the --version flag, I am still getting the same response, that the version is 0.25.1. Furthermore, running

sudo certbot renew --dry-run

results in the following error:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/pandora.nlcv.bas.bg.conf
-------------------------------------------------------------------------------
Attempting to parse the version 0.26.1 renewal configuration file found at /etc/letsencrypt/renewal/pandora.nlcv.bas.bg.conf with version 0.25.1 of Certbot. This might not work.
Cert not due for renewal, but simulating renewal for dry run
Could not choose appropriate plugin: The requested nginx plugin does not appear to be installed
Attempting to renew cert (pandora.nlcv.bas.bg) from /etc/letsencrypt/renewal/pandora.nlcv.bas.bg.conf produced an unexpected error: The requested nginx plugin does not appear to be installed. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/pandora.nlcv.bas.bg/fullchain.pem (failure)

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/pandora.nlcv.bas.bg/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------
1 renew failure(s), 0 parse failure(s)

What am I supposed to do now?! I still have about a month until the certificate expires but I really need some kind of solution before this deadline ends…

2

I split your post into a separate discussion.

Certbot doesn't support TLS-ALPN at all.

You might look at this thread: Which client support tls-alpn challenge?

Unfortunately, TLS-ALPN is operationally a lot more complicated than TLS-SNI was. Native webserver support is very sparse.

For example, if you want to use e.g. acme.sh's TLS-ALPN support, the suggested workflow is to stop nginx (in a pre-hook), run acme.sh (it will bind to port 443 and perform renewal), and then start nginx again (in a post-hook).

If you have nginx compiled with the SSL pre-read module, you can do it without stopping your server, but it causes some convoluted configuration.

lego also supports TLS-ALPN in a similar way (standalone mode).

Finally, you can also totally replace or front nginx with Caddy, which makes using TLS-ALPN very easy, but it's a pretty big change to your infrastructure.

The tl;dr; is, if you can keep 80 open or use the DNS challenge, it simplifies things a lot. Using TLS-ALPN is not very convenient at the moment, but not impossible.

Edit:

HSTS is a browser thing, not an HTTP protocol thing. It doesn't affect the ACME protocol, so you can keep using 80 in its presence.

2 Likes
3

What happens if you run “sudo apt-get upgrade”? At least part of Certbot is apparently version 0.25.1.

Edit: What does “sudo which certbot” show?

Edit: You were using version 0.26.1 before.

1 Like
4

Gosh, what a mess… I am already sorry that we opted to use Let’s Encrypt certificates and even that we switched to HTTPS at all… It is simply irresponsible to drop TLS challenge support without implementing a viable alternative first…

No, changing our web server is out-of-question.

We cannot use the DNS challenge. Port 80 is open, but renewing the certificates that way is unreliable, because it fails when that port is under DDoS attack.

I’ll try playing with Lego; I think we use that for our Windows+Apache servers, because Certbot doesn’t work on Windows (even after all these years; that’s another horrible annoyance). Hopefully I can get Lego running on Linux. Failing that, I’ll try to hack something with some script that stops the server, runs acme.sh, and starts the server again…

Regarding HSTS - you misunderstand. Yes, I know that it’s a “browser thing”. Because of that “thing”, every browser in the world that has ever visited our site, is remembering that it can be accessed via HTTPS only. Which means that we cannot drop HTTPS support for our site, because it will become inaccessible to all these browsers!

Gosh, what a mess…

5

I'm not sure what this is referring to.

The change affecting you, is that the certificate authority should be able to connect to your server over port 80. Nothing about your existing HTTPS setup is affected.

I am not even convinced that your DDoS reason is a valid complaint against using 80. Certbot would attempt renewal about 60 times across 30 days before a certificate expires. If your port 80 is inaccessible from the wider internet, that many times in a row, I would consider that your server is not actually on the internet.

I have to ask you genuinely - did you try to use HTTP validation in practice? Because it sounds like it would probably work fine, even if it was flakey due to attacks. Certbot does not rely on your server (or the CA) to be "online" at all times.

Yes, the extra complication is regrettable, sorry.

2 Likes
6

Why not?

Among other options, you could consider using acme-dns.

1 Like
7

It doesn't help. certbot --version still shows version 0.25.1.

It shows /usr/local/bin/certbot.

Yes, I know that this is what the config file says. Nevertheless, the fact remains that certbot --version shows version 0.25.1.

I also tried letsencrypt --version - that one shows version 0.28.0, which is supposedly the latest? Do I have two different Certbot versions installed? What should I do?

8

Yes, you have two different versions installed. The apt package – apparently version 0.28.0 – is /usr/bin/certbot and also provides an alias at /usr/bin/letsencrypt.

There’s some other thing in /usr/local/bin/certbot.

9

What is so hard to understand?! Our site is currently HTTPS-only. Because we have an HSTS policy, we cannot switch back to a HTTP site any more - because every browser that has ever visited the site will insist on accessing it only via HTTPS and the site will become inaccessible to them! Not to mention that the DDoS problem will start affecting our web site again.

Well, just trust me on this, OK? I am speaking from experience here. There is a silly virus out there that occasionally slams POST requests to the domain where our site is. It is widespread enough to effectively cause a not-very-high-volume but constant DDoS attack. It was one of the reasons why we switched to HTTPS in the first place; it allowed us to solve the DDoS problem (well, to get around it).

Well, that's what goddamn happens when your server is under DDoS, no? It's effectively not on the Internet for the duration of the attack - which in our case is indefinitely.

Yes, and our site became inaccessible when the certificates expired due to a failed renewal. That's when we switched to TLS-SNI-01.

10

Should I just delete that? I tried

sudo apt list --installed | grep -F certbot | grep -F 25

but did not find any package that has installed certbot version 0.25.

11

I'm not sure. :confused:

Moving or deleting it ought to be fine.

Or replacing it with a symlink to /usr/bin/certbot in case the path is hardcoded in anything.

What concerns me more is that it might have a zillion libraries installed in /usr/local/lib or somewhere that might interfere with the packaged Certbot as well.

12

I’ll try another approach. Your site currently is answering on port 80:

$ curl -X GET -I pandora.nlcv.bas.bg/.well-known/acme-challenge/xx
HTTP/1.1 307 Temporary Redirect
Server: nginx
Date: Sat, 19 Jan 2019 11:04:02 GMT
Content-Type: text/html
Content-Length: 180
Connection: keep-alive
Location: https://pandora.nlcv.bas.bg/.well-known/acme-challenge/xx

That is the sole requirement to use HTTP-01 validation. It doesn’t matter that you’re redirecting traffic to HTTPS, because Let’s Encrypt will follow the redirect to your secure port. Certbot’s nginx plugin will automatically configure nginx to respond to the challenge, on port 80.

Once you upgrade your Certbot and get rid of the duplicate certbot binary and libraries, HTTP validation should be completely seamless and automatic for you.

You do not seem like you need to do anything except fix your Certbot install.

3 Likes
13

How was renewal configured at the time? As @_az said, with the default setup, for a certificate to expire, renewal must have failed about 60 times over almost 30 days. Was the DDoS really that bad? Is it still that bad?

You might not want to bet your life on it, but an occasional failed renewal attempt shouldn't be an issue.

2 Likes
14

Hi @bontchev

you have really an open port 80:

Domainname Http-Status redirect Sec. G
• http://pandora.nlcv.bas.bg/
195.96.252.66 307 https://pandora.nlcv.bas.bg/ 0.083 A
• https://pandora.nlcv.bas.bg/
195.96.252.66 403 1.864 M
Forbidden
• http://pandora.nlcv.bas.bg/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
195.96.252.66 307 https://pandora.nlcv.bas.bg/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 0.083 A
• https://pandora.nlcv.bas.bg/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 404 1.403 A
Not Found

This is enough to use http - validation. The last http status 404 is wonderful.

And closing port 80 isn't good:

Why closing port 80 is bad for security

15

As far as I can see, /usr/local/lib contains only Python (2.7 and 3.5) packages. Do you want me to get a full list of the contents?

Also, I just checked, the crontab job runs /usr/bin/certbot explicitly. I ran it manually in renew --dry-run mode and it seemed to work just fine.

16

That only means that currently the DDoS attack is not strong enough to prevent it from doing so. There is no guarantee that it will remain so at the time of the certificate renewal. :frowning:

17

Look for files or directories with e.g. certbot or acme in their names in /usr/local/lib/python2.7/dist-packages, python2.7/site-packages, and the same for Python 3.

That's good.

(There should also be a systemd timer that hardcodes /usr/bin/certbot. You can see it with "systemctl cat certbot".)

Still, if you have any of your own scripts that run Certbot or something, they could be using it.

That's good. :slightly_smiling_face:

18

Sigh, you don’t understand.

Yes, the port is opened. My problem is not the problem most people complaining about this issue have - that their provider or whatever doesn’t have this port open.

My problem is that contacting our server over this port is unreliable. There are sporadic but constant DDoS attacks on our domain (so, changing IP addresses doesn’t help) that can prevent the site from answering on this port (80 only; 443 is fine). Then you won’t be getting redirects (307); you won’t be getting any response at all.

Just because it isn’t happening right now is no guarantee that it won’t happen when the certificate has to be renewed…

19

Dang, there are some. :frowning:

$ find /usr/local/lib -name 'certbot*' -print
/usr/local/lib/python2.7/dist-packages/certbot-0.25.1.dist-info
/usr/local/lib/python2.7/dist-packages/certbot

Should I do anything about them?

I think this means that the previous version of Certbot was installed with pip, instead of with apt. Should I do sudo -H pip uninstall certbot?

There is:

# /lib/systemd/system/certbot.service
[Unit]
Description=Certbot
Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
Documentation=https://letsencrypt.readthedocs.io/en/latest/
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true

I am not sufficiently familiar with Systemd. If there is a timer, does that mean that I still need the crontab job?

20

Sporadic attacks are not relevant. Every server has sporadic attacks.

There is a window of 30 days to renew a certificate.