HTTP challenge access log shows 200 (success) but certbot says timeout

I run the certbot renew --dry-run and I can see the challenge files written into the /var/lib/letsencrypt/http_challenges. I can also see via the access.log that those files were successfully served by the web server:

outbound1.letsencrypt.org - - [16/Aug/2019:14:27:58 -0500] “GET /.well-known/acme-challenge/XMGGPxospcGE9oDH3CsB0jnBjZWF7i67JWDZBYedK68 HTTP/1.1” 200 308 “-” “Mozilla/5.0 (compatible; Let’s Encrypt validation server; +https://www.letsencrypt.org)”
ec2-3-122-105-36.eu-central-1.compute.amazonaws.com - - [16/Aug/2019:14:27:59 -0500] “GET /.well-known/acme-challenge/XMGGPxospcGE9oDH3CsB0jnBjZWF7i67JWDZBYedK68 HTTP/1.1” 200 308 “-” “Mozilla/5.0 (compatible; Let’s Encrypt validation server; +https://www.letsencrypt.org)”
outbound1.letsencrypt.org - - [16/Aug/2019:14:28:02 -0500] “GET /.well-known/acme-challenge/GyPacDy3sYwy9FDVGMNbEJG7uqbpw2fHL1ZNq09oVQA HTTP/1.1” 200 308 “-” “Mozilla/5.0 (compatible; Let’s Encrypt validation server; +https://www.letsencrypt.org)”
ec2-3-122-105-36.eu-central-1.compute.amazonaws.com - - [16/Aug/2019:14:28:06 -0500] “GET /.well-known/acme-challenge/GyPacDy3sYwy9FDVGMNbEJG7uqbpw2fHL1ZNq09oVQA HTTP/1.1” 200 308 “-” “Mozilla/5.0 (compatible; Let’s Encrypt validation server; +https://www.letsencrypt.org)”
ec2-54-245-186-160.us-west-2.compute.amazonaws.com - - [16/Aug/2019:14:29:16 -0500] “GET /.well-known/acme-challenge/dB7hnb4O_gofaOs29aLltV2mZjHncPfoOnVgoSDEmzs HTTP/1.1” 200 308 “-” “Mozilla/5.0 (compatible; Let’s Encrypt validation server; +https://www.letsencrypt.org)”

Yet certbot still says it times out (see error below). I also placed a file in the http_challenges folder with the same permissions/owner as the challenge files and pulled it up externally (from another network) to verify that the rewrite is working and files from that folder can be served externally.

My domain is:www.leeburch.com

I ran this command:certbot renew --dryu-run

It produced this output:
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My web server is (include version): Apache 2.4
The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):certbot 0.31.0

That's one authorization at 14:27:58 - 14:27:59.

That's another authz at 14:28:02-14:28:06.

That's a third one at 14:29:16.

It looks like requests were timing out -- each validation normally results in 4 HTTP requests in the staging environment, but those validations only logged 1-2.

I am not so sure it is me. I during testing to capture how long service took, without any changes suddenly it succeeded. Perhaps the servers are very busy (mine isn’t) and it only appears to be timing out due to busy servers on the Lets Encrypt end?

Hi @ltburch

there is a timeout. Check the output of https://check-your-website.server-daten.de/?q=leeburch.com

Non-www + /.well-known/acme-challenge/random-filename + 24.148.31.169 has a (good) http status 404 - not found.

But www + /.well-known/acme-challeng/random-filename + 24.148.29.215 and 24.148.31.169 - both have a timeout.

The 24.148.29.215 may be the old / wrong ip address. But 24.148.31.169 is new.

1 Like

Excellent catch, I will check my DNS records. I may well have updated one and not the other when they changed my IP a while back. Thanks for the feedback.

1 Like

No, it's not an ip address problem.

The 24.148.29.215 may be the old address, ignore it.

But your 24.148.31.169 - non www answers, www doesn't answer. Looks like

  • a blocking firewall with a timeout (or)
  • a wrong configured webserver
<VirtualHost OldIp:80>

is wrong, use something like

<VirtualHost *:80>
1 Like

Yes, turns out you are correct about that.

I also corrected by virtual hosts for 80 and 443.

If I go and pull up

http://owncloud.leeburch.com/.well-known/acme-challenge/iamhere

it works fine (from an external network)

Yet 9/10 times when I run certbot I still am getting:

http://owncloud.leeburch.com/.well-known/acme-challenge/S-JDPCRSIKZRbDubFpcHyKVFq5q-gTSBKgpLbkXHmao:
Timeout during connect (likely firewall problem)

Yet 1/10 times or so, it actually works. with no config change on my part.

I even see the request from the certbot servers and see that I am serving a 200 return code on the request, yet the certbot still usually reports timeout.

I am wondering, is this a problem on my end or that I can pull up files in the challenge directory and that sometimes certbot can too mean the problem is happening elsewhere?

FYI, the validation servers are operated by Let’s Encrypt, not the Certbot development team.

Like I said, Let’s Encrypt sends multiple validation requests. If you’re only receiving one, that is evidence that something is wrong.

(Though, in the production environment, only one validation request matters.)

The validation servers run in Let’s Encrypt’s two data centers and three(?) Amazon regions. It’s most likely that the problem is with your infrastructure, or a regional issue affecting your ISP.

Edit:

I cannot access the URL you posted from four ISPs in five locations (including one of the Amazon regions Let’s Encrypt uses). Attempting to connect times out.

FWIW, I don’t think the AAAA record with the ::ffff:0:0/96 IP will be used by clients. And connecting to the IPv4 IP in that AAAA record also seems to time out.

Edit 2:

I can ping your site, though.

1 Like

How do you test with the other ISPs? For me I was testing “external access” by VPNing out and then making calls back knowing they originate from the VPN endpoint. Maybe if I could test some alternate locations I might find the issue.

Still it is weird that it works sometimes, and just about 2 hours ago I did get the validation to work - again no changes on my end.

I just ran curl on some VPSes.

There are some good web services for accessing an URL from multiple locations; this is the one I can remember:

https://pulse.turbobytes.com/

Doesn't work with my browser.

And there is a new certificate - https://check-your-website.server-daten.de/?q=owncloud.leeburch.com/.well-known/acme-challenge/iamhere

Issuer not before not after Domain names LE-Duplicate next LE
Let's Encrypt Authority X3 2019-08-22 2019-11-20 leeburch.com, owncloud.leeburch.com, www.leeburch.com - 3 entries duplicate nr. 1

Definitely seems to be something happening in between me and others. This output is from a server I have shell access to in California (I am in Chicago IL)

drfurter@discovery ~ $ wget -O - http://owncloud.leeburch.com/.well-known/acme-challenge/iamhere
–2019-08-23 16:49:42-- http://owncloud.leeburch.com/.well-known/acme-challenge/iamhere
Resolving owncloud.leeburch.com… 24.148.31.169, ::ffff:24.148.29.215
Connecting to owncloud.leeburch.com|24.148.31.169|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 22
Saving to: ‘STDOUT’

  •                                                                          0%[                                                                                                                                                                                            ]       0  --.-KB/s               I am serving properly
    
  •                                                                        100%[===========================================================================================================================================================================================>]      22  --.-KB/s    in 0s      
    

2019-08-23 16:49:42 (3.71 MB/s) - written to stdout [22/22]

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.