Ubuntu 14.04 certbot fail after update

[MikeR]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mike-r.com

I ran this command: sudo --apache certbot renew --dry-run

It produced this output:

Processing /etc/letsencrypt/renewal/alpha.mike-r.com.conf

Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for alpha.mike-r.com
http-01 challenge for mike-r.com
Waiting for verification…
Cleaning up challenges
Attempting to renew cert (alpha.mike-r.com) from /etc/letsencrypt/renewal/alpha.mike-r.com.conf produced an unexpected error: Failed authorization procedure. mike-r.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mike-r.com/.well-known/acme-challenge/1Idyk2Cy2J38ealt6r6ZosCTGvWetm4Pqw0KrIkpAIQ: Timeout during connect (likely firewall problem), alpha.mike-r.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://alpha.mike-r.com/.well-known/acme-challenge/JupHS54zi8LBRqrXCRLGTT2JO7qd4BR4Yb4aEgiCyCM: Timeout during connect (likely firewall problem). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/alpha.mike-r.com/fullchain.pem (failure)

---

** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/alpha.mike-r.com/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

• The following errors were reported by the server:

  Domain: mike-r.com
  Type: connection
  Detail: Fetching
  http://mike-r.com/.well-known/acme-challenge/1Idyk2Cy2J38ealt6r6ZosCTGvWetm4Pqw0KrIkpAIQ:
  Timeout during connect (likely firewall problem)

  Domain: alpha.mike-r.com
  Type: connection
  Detail: Fetching
  http://alpha.mike-r.com/.well-known/acme-challenge/JupHS54zi8LBRqrXCRLGTT2JO7qd4BR4Yb4aEgiCyCM:
  Timeout during connect (likely firewall problem)

  To fix these errors, please make sure that your domain name was
  entered correctly and the DNS A/AAAA record(s) for that domain
  contain(s) the right IP address. Additionally, please check that
  your computer has a publicly routable IP address and that no
  firewalls are preventing the server from communicating with the
  client. If you’re using the webroot plugin, you should also verify
  that you are serving files from the webroot path you provided.

Note: My firewall is open on ports 80 and 443.
checking DNS:
dig alpha.mike-r.com returns:
…
;; ANSWER SECTION:
alpha.mike-r.com. 3599 IN A 31.154.173.227
…

dig mike-r.com returns:
…
;; ANSWER SECTION:
mike-r.com. 3599 IN A 31.154.173.227
…

My web server is (include version):
Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g

The operating system my web server runs on is (include version):
Ubuntu 16.04.4 LTS (32-bit)

My hosting provider, if applicable, is: Myself, on Linux as above

I can login to a root shell on my machine (yes or no, or I don’t know):yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

[MikeR]

@MikeR

sudo --apache certbot renew --dry-run

correction: command was sudo certbot --apache renew --dry-run

Sorry,
Mike

[Osiris]

Strangely, I’m getting a time out the one time, but the other time I can connect slowly, but “fine” in the end.

LetsDebug confirms the time out: https://letsdebug.net/mike-r.com/1787

The run before that was fine: https://letsdebug.net/mike-r.com/1786

So perhaps your network or router isn’t stable?

[MikeR]

So perhaps your network or router isn’t stable?

My web server has been on the air since Ubuntu 5.x, which makes it over a decade now.
I have not noticed any instability thus far.

I recently installed (automatically, with no user intervention) a new version of certbot. I have no idea whether this is connected.

[_az]

Seems to be totally down now. Not sure if you’re rebooting your router or whatever.

[MikeR]

What URL did you try?
I just tried https://alpha.mike-r.com from an external (different network, different location) and it seemed OK.

[_az]

You definitely have some hugely overzealous stateful firewall dropping connections. mod_security? fail2ban?

I’ve gone through 5 different internet connections.

First two requests succeed, and then all of a sudden, your server is permanently dropping all traffic from that IP. Like clockwork.

I bet a bunch of my IPs are currently in your firewall.

iptables -L -n

Edit: I also see this message in the markup on the final request that succeeds before I’m blocked:

<UL><LI> A system error was returned when seeking the document
<BR>OR<BR>
<LI> Access controls placed on that document by the HTTP server deny access.
<!-- REMOTE_HOST = "MY_IPS_PRT_RECORD" -->

[MikeR]

I have fail2ban set to catch addresses making five (or more) hits in 120 seconds. This is the first time i’ve heard (read) complaints.

I’v set all the addresses that accessed “/.well-known/acme-challenge/” as “ignoreip” in fail2ban, and just checked iptables – none of them are banned.

Let me know your IP and I’ll unban it; next time try more slowly.

[_az]

Your rules are not working properly.

Here’s a server where I got banned after a single request (I had never connected to your server from that host before):

https://asciinema.org/a/V8tKPxbIXJ32vzsVZgZrezuyM

In any case, the cause is now evident - it’s self inflicted with fail2ban.

[MikeR]

@_az

1. That request was refused due to attempting to access a non-existent directory.

2. As it seems you have no concrete advice to offer, would you please leave, with my thanks.

[cpu]

Hi @mikeR,

@MikeR

1. As it seems you have no concrete advice to offer, would you please leave, with my thanks.

I think the concrete advice here is to fix or remove your fail2ban rules. I agree with @_az's assessment of the problem: You're likely to run into continued difficulty using HTTP-01 challenges with this system in place as it is today.

I'm unfamiliar with fail2ban, is it possible to make an exception for the /.well-known/ path that the ACME challenges are placed under?

[MikeR]

OK I’m persuaded.
This approaching week-end I will zero all the iptables rules, stop fail2ban and try
sudo certbot --apache renew --dry-run
again,

I will report back with any results.
Here’s hoping!!

[MikeR]

There’s a saying “If two people tell you you’drunk – go to bed!!”

I got impatient, and tried removing all the iptables chains, and tried
sudo certbot --apache renew --dry-run again

I was wrong, you were right!! the certbot command ended successfully!

Unfortunately to restore everything I had to reboot. But now I have a workaround until I find which fail2ban rule (there are several hundred) is getting in the way,

Marking this item Solved, And thank you _az and cpu!

Mike

[cpu]

@MikeR

Marking this item Solved, And thank you _az and cpu!

Glad to hear you got things working!

[MikeR]

Found the problem.
After a spate of hacking attempts from AWS sites last year I blocked all Amazon addresses.
Well, it seems as if certbot renew uses some of those addresses.

c.f. http://paste.ubuntu.com/p/ZrwW98CnM4/