Ubuntu 14.04 certbot fail after update


#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mike-r.com

I ran this command: sudo --apache certbot renew --dry-run

It produced this output:

Processing /etc/letsencrypt/renewal/alpha.mike-r.com.conf

Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for alpha.mike-r.com
http-01 challenge for mike-r.com
Waiting for verification…
Cleaning up challenges
Attempting to renew cert (alpha.mike-r.com) from /etc/letsencrypt/renewal/alpha.mike-r.com.conf produced an unexpected error: Failed authorization procedure. mike-r.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mike-r.com/.well-known/acme-challenge/1Idyk2Cy2J38ealt6r6ZosCTGvWetm4Pqw0KrIkpAIQ: Timeout during connect (likely firewall problem), alpha.mike-r.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://alpha.mike-r.com/.well-known/acme-challenge/JupHS54zi8LBRqrXCRLGTT2JO7qd4BR4Yb4aEgiCyCM: Timeout during connect (likely firewall problem). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/alpha.mike-r.com/fullchain.pem (failure)


** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/alpha.mike-r.com/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

Note: My firewall is open on ports 80 and 443.
checking DNS:
dig alpha.mike-r.com returns:

;; ANSWER SECTION:
alpha.mike-r.com. 3599 IN A 31.154.173.227

dig mike-r.com returns:

;; ANSWER SECTION:
mike-r.com. 3599 IN A 31.154.173.227

My web server is (include version):
Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g

The operating system my web server runs on is (include version):
Ubuntu 16.04.4 LTS (32-bit)

My hosting provider, if applicable, is: Myself, on Linux as above

I can login to a root shell on my machine (yes or no, or I don’t know):yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no


#2

correction: command was sudo certbot --apache renew --dry-run

Sorry,
Mike


#3

Strangely, I’m getting a time out the one time, but the other time I can connect slowly, but “fine” in the end.

LetsDebug confirms the time out: https://letsdebug.net/mike-r.com/1787

The run before that was fine: https://letsdebug.net/mike-r.com/1786

So perhaps your network or router isn’t stable?


#4

So perhaps your network or router isn’t stable?

My web server has been on the air since Ubuntu 5.x, which makes it over a decade now.
I have not noticed any instability thus far.

I recently installed (automatically, with no user intervention) a new version of certbot. I have no idea whether this is connected.


#5

Seems to be totally down now. Not sure if you’re rebooting your router or whatever.


#6

What URL did you try?
I just tried https://alpha.mike-r.com from an external (different network, different location) and it seemed OK.


#7

You definitely have some hugely overzealous stateful firewall dropping connections. mod_security? fail2ban?

I’ve gone through 5 different internet connections.

First two requests succeed, and then all of a sudden, your server is permanently dropping all traffic from that IP. Like clockwork.

I bet a bunch of my IPs are currently in your firewall.

iptables -L -n

Edit: I also see this message in the markup on the final request that succeeds before I’m blocked:

<UL><LI> A system error was returned when seeking the document
<BR>OR<BR>
<LI> Access controls placed on that document by the HTTP server deny access.
<!-- REMOTE_HOST = "MY_IPS_PRT_RECORD" -->

#8

I have fail2ban set to catch addresses making five (or more) hits in 120 seconds. This is the first time i’ve heard (read) complaints.

I’v set all the addresses that accessed “/.well-known/acme-challenge/” as “ignoreip” in fail2ban, and just checked iptables – none of them are banned.

Let me know your IP and I’ll unban it; next time try more slowly.


#9

Your rules are not working properly.

Here’s a server where I got banned after a single request (I had never connected to your server from that host before):

https://asciinema.org/a/V8tKPxbIXJ32vzsVZgZrezuyM

In any case, the cause is now evident - it’s self inflicted with fail2ban.


#10

@_az

  1. That request was refused due to attempting to access a non-existent directory.

  2. As it seems you have no concrete advice to offer, would you please leave, with my thanks.


#11

Hi @mikeR,

I think the concrete advice here is to fix or remove your fail2ban rules. I agree with @_az’s assessment of the problem: You’re likely to run into continued difficulty using HTTP-01 challenges with this system in place as it is today.

I’m unfamiliar with fail2ban, is it possible to make an exception for the /.well-known/ path that the ACME challenges are placed under?


#12

OK I’m persuaded.
This approaching week-end I will zero all the iptables rules, stop fail2ban and try
sudo certbot --apache renew --dry-run
again,

I will report back with any results.
Here’s hoping!!


#13

There’s a saying “If two people tell you you’drunk – go to bed!!”

I got impatient, and tried removing all the iptables chains, and tried
sudo certbot --apache renew --dry-run again

I was wrong, you were right!! the certbot command ended successfully!

Unfortunately to restore everything I had to reboot. But now I have a workaround until I find which fail2ban rule (there are several hundred) is getting in the way,

Marking this item Solved, And thank you _az and cpu!

Mike


#14

Glad to hear you got things working! :tada:


#15

Found the problem.
After a spate of hacking attempts from AWS sites last year I blocked all Amazon addresses.
Well, it seems as if certbot renew uses some of those addresses.

c.f. http://paste.ubuntu.com/p/ZrwW98CnM4/


#16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.