Failed http-01 challenge

Hi there.

I work with a hosting company and we use Let's Encrypt quite a lot.
By the way, here is a chance to thank all the people involved in this.

There is a particular server and domaine name that caused an unusual amout of frustration.

TL;DR :

  • http-01 challenge never worked, even if direct GET on the challenge URL is working perfectly
  • dns-01 challenge worked

When we try to make a certificate with an http-01 challenge, in verbose mode, we can see the challenge file being created, permissions are OK, we can GET the challenge URL without firewall issues (and no special firewall permissions on source IP) and we have the proper content (same as file name).
But the challenge ends in a timeout error :

Failed authorization procedure. yest.app (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://yest.app/.well-known/acme-challenge/ufaExU4ADFLBRpz48ONumaM9xFOkWB0o_wxGo1U9640: Timeout during connect (likely firewall problem)
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: yest.app
   Type:   connection
   Detail: Fetching
   http://yest.app/.well-known/acme-challenge/ufaExU4ADFLBRpz48ONumaM9xFOkWB0o_wxGo1U9640:
   Timeout during connect (likely firewall problem)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

During the challenge (before the timeout) we were able to GET the challenge URL from multiple sources.
The Apache log file showed our GET requests but also Let's Encrypt requests :

$ grep ufaExU4ADFLBRpz48ONumaM9xFOkWB0o_wxGo1U9640 /home/yest/log/*.log
/home/yest/log/access.log:18.222.145.89 - - [19/Jul/2021:16:14:23 +0200] "GET /.well-known/acme-challenge/ufaExU4ADFLBRpz48ONumaM9xFOkWB0o_wxGo1U9640 HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
/home/yest/log/access.log:18.236.228.243 - - [19/Jul/2021:16:14:23 +0200] "GET /.well-known/acme-challenge/ufaExU4ADFLBRpz48ONumaM9xFOkWB0o_wxGo1U9640 HTTP/1.1" 200 292 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
/home/yest/log/access.log:31.x.y.z - - [19/Jul/2021:16:14:32 +0200] "GET /.well-known/acme-challenge/ufaExU4ADFLBRpz48ONumaM9xFOkWB0o_wxGo1U9640 HTTP/1.1" 200 273 "-" "curl/7.64.0"

I've put a (fake) persistent file at http://yest.app/.well-known/acme-challenge/HWAQkCfF9F4C9smqhb2s1dZLu3ULLHpuMwVZ4GCkpDo in case anyone would like to investigate "from outside". You should get a "200 OK" return code like this :

$ curl -v http://yest.app/.well-known/acme-challenge/HWAQkCfF9F4C9smqhb2s1dZLu3ULLHpuMwVZ4GCkpDo
* […]
*   Trying 163.172.20.152...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x56524ddbcfb0)
* Connected to yest.app (163.172.20.152) port 80 (#0)
> GET /.well-known/acme-challenge/HWAQkCfF9F4C9smqhb2s1dZLu3ULLHpuMwVZ4GCkpDo HTTP/1.1
> Host: yest.app
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 19 Jul 2021 14:28:01 GMT
< Server: Apache
< Last-Modified: Mon, 19 Jul 2021 14:16:51 GMT
< ETag: "2c-5c77a96af37b7"
< Accept-Ranges: bytes
< Content-Length: 44
< 
HWAQkCfF9F4C9smqhb2s1dZLu3ULLHpuMwVZ4GCkpDo
* Connection #0 to host yest.app left intact

We arrived at the conclusion that there might be a connectivity issue between Let's Encrypt servers and ours.

If anyone has the lightest clue regarding this mystery we would appreciate greatly.

Thanks.

1 Like

I should add that we've tried a few other domain names on the same server. The behaviour is the same, so it's nothing related to the domain itself.

1 Like

Note: the reason why you're seeing validation requests in your log file and having a failed authorization due to time outs is probably because part of the 4 requests are actually succeeding, but some other don't. See this thread about these multiple requests: ACME v1/v2: Validating challenges from multiple network vantage points

2 Likes

How many connections to your server from the "Let's Encrypt validation server" do you generally see? There should be four, I believe, 1 from the primary system and 3 from the secondaries. Your log above only shows 2, which makes me think that yes, not everywhere on the Internet can get to your server.

3 Likes

Well I don't.
I get:

curl -I http://yest.app/.well-known/acme-challenge/HWAQkCfF9F4C9smqhb2s1dZLu3ULLHpuMwVZ4GCkpDo
curl: (56) Recv failure: Connection reset by peer
2 Likes

Thanks for the feedback. It probably is a network issue, independent from Let's Encrypt.
To confirm this, I've set an http check from many places around the world using UpDown.io

2 Likes

Yeah, it definitely seems inconsistent. I can't connect from my home, but it works from my server in AWS.

2 Likes

We're working on this with the network team. Any idea where I could find the full list of IP Let's Encrypt uses to validate challenges ? I'd like to run an "mtr" between the server and them.

They don't provide a list of IPs, since they check from several vantage points on the Internet that occasionally change. (And they're trying to validate that you own the name as seen from everywhere.) I do believe that (at least some of) their secondary systems are in AWS, but you seem to be having a problem with the primary.

2 Likes

I've forced a certificate renew on a different server to get the 4 current IP and I've managed to execute the "mtr" :

$ for IP in 18.184.29.122 52.39.4.59 66.133.109.36 3.142.122.14; do echo "" && echo "Test Let's Encrypt $IP" && mtr -r $IP; done

Test Let's Encrypt 18.184.29.122
Start: 2021-07-20T14:42:24+0200
HOST: yest-www00                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 163-172-20-1.rev.poneytel  0.0%    10    0.5   0.5   0.4   0.6   0.1
  2.|-- 195.154.1.226              0.0%    10    0.4   0.5   0.4   0.6   0.0
  3.|-- 51.158.8.62                0.0%    10    0.6   0.6   0.4   0.8   0.1
  4.|-- amazon-th2.par.franceix.n  0.0%    10    0.7   1.6   0.6   4.7   1.6
  5.|-- 52.46.95.132               0.0%    10    1.1   1.7   1.0   4.2   1.0
  6.|-- 52.93.16.143               0.0%    10    0.9   0.9   0.7   1.1   0.2
  7.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0

Test Let's Encrypt 52.39.4.59
Start: 2021-07-20T14:42:40+0200
HOST: yest-www00                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 163-172-20-1.rev.poneytel  0.0%    10    0.4   0.6   0.4   2.1   0.5
  2.|-- 195.154.1.226              0.0%    10    0.5   0.5   0.4   0.6   0.1
  3.|-- 51.158.8.62                0.0%    10    0.6   0.7   0.4   1.0   0.2
  4.|-- be4751.rcr21.b022890-0.pa  0.0%    10    0.7   0.8   0.7   1.0   0.1
  5.|-- be2152.ccr32.par04.atlas.  0.0%    10    1.4   1.6   1.4   3.1   0.5
  6.|-- be2102.ccr41.par01.atlas.  0.0%    10    1.8   1.8   1.7   2.0   0.1
  7.|-- be2315.ccr31.bio02.atlas.  0.0%    10   13.9  14.1  13.9  15.3   0.4
  8.|-- be2331.ccr41.dca01.atlas.  0.0%    10   81.3  81.3  81.1  81.7   0.2
  9.|-- be2112.ccr41.atl01.atlas.  0.0%    10   92.1  92.1  91.9  92.2   0.1
 10.|-- be2687.ccr41.iah01.atlas.  0.0%    10  108.8 108.9 108.6 109.6   0.3
 11.|-- be2927.ccr21.elp01.atlas.  0.0%    10  121.4 121.4 121.3 121.5   0.0
 12.|-- be2929.ccr31.phx01.atlas.  0.0%    10  132.4 132.5 132.3 132.7   0.1
 13.|-- be2931.ccr41.lax01.atlas.  0.0%    10  144.2 144.3 144.1 144.5   0.1
 14.|-- be3271.ccr41.lax04.atlas.  0.0%    10  144.9 144.5 144.2 144.9   0.2
 15.|-- 38.88.196.90               0.0%    10  140.9 141.3 140.9 141.9   0.3
 16.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 17.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 18.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 19.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 20.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 21.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 22.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 23.|-- 52.93.130.73               0.0%    10  161.4 161.2 160.9 161.6   0.3
 24.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0

Test Let's Encrypt 66.133.109.36
Start: 2021-07-20T14:42:59+0200
HOST: yest-www00                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 163-172-20-1.rev.poneytel  0.0%    10    0.5   0.5   0.4   0.6   0.0
  2.|-- 195.154.1.228              0.0%    10    0.7   0.6   0.5   0.8   0.1
  3.|-- 51.158.8.64                0.0%    10    0.7   0.6   0.6   0.7   0.1
  4.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0

Test Let's Encrypt 3.142.122.14
Start: 2021-07-20T14:43:15+0200
HOST: yest-www00                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 163-172-20-1.rev.poneytel  0.0%    10    0.4   0.5   0.4   0.6   0.1
  2.|-- 195.154.1.226              0.0%    10    0.4   0.5   0.4   0.7   0.1
  3.|-- 51.158.8.62                0.0%    10    0.7   0.6   0.5   0.9   0.1
  4.|-- be4751.rcr21.b022890-0.pa  0.0%    10    0.7   0.7   0.6   0.9   0.1
  5.|-- be2152.ccr32.par04.atlas.  0.0%    10    1.4   1.5   1.4   1.7   0.1
  6.|-- be2102.ccr41.par01.atlas.  0.0%    10    1.7   1.8   1.7   1.9   0.1
  7.|-- be12497.ccr41.lon13.atlas  0.0%    10    8.3   8.4   8.3   8.7   0.1
  8.|-- be2099.ccr31.bos01.atlas.  0.0%    10   70.6  70.6  70.5  70.6   0.1
  9.|-- 38.140.158.98              0.0%    10   70.2  70.9  70.2  72.8   0.9
 10.|-- 52.93.76.134               0.0%    10   71.0  72.6  70.9  84.5   4.2
 11.|-- 52.93.76.145               0.0%    10   70.3  70.4  70.2  71.0   0.3
 12.|-- 52.93.135.185              0.0%    10   93.4  94.5  93.2  98.9   2.1
 13.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 14.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 15.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 16.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 17.|-- 52.93.132.165              0.0%    10   99.1  92.0  90.6  99.1   2.6
 18.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 19.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 20.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 21.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 22.|-- 52.93.133.87               0.0%    10   98.6  94.9  93.1 102.8   3.2
 23.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 24.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 25.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 26.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 27.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 28.|-- 52.95.3.38                 0.0%    10   94.1  95.0  93.7 100.7   2.2
 29.|-- ???                        0.0%     0    0.0   0.0   0.0   0.0   0.0

I've also tried this with IP addresses used by UpDown.

Investigation is in progress by the network team.