Unable to get Let's Encrypt certificates on spl89.hosting.reg.ru

Hello!

I encountered a problem that it is impossible to get Let's Encrypt certificates on one of the servers due to an error:

Invalid response from https://acme-v02.api.letsencrypt.org/acme/authz-v3/131625315396.

Details:

Type: urn:ietf:params:acme:error:connection

Status: 400

Detail: During secondary validation: 31.31.198.201: Fetching http://shtakesnyjbogperla.ru/.well-known/acme-challenge/Invd7Ycd4XrlhiU3caNz1yidfKHpIfhpW1SKg5k992E: Timeout during connect (likely firewall problem)

The specified domain is located on server spl89.hosting.reg.ru. Server running CentOS 7, Plesk Obsidian panel.
I checked the server and domain address via https://letsdebug.net/ and got these errors:

There are no blocks in the firewall, server available from other countries:
https://ping-admin.com/free_test/result/16581280137ty1x89casi868zo3hwri.html

When checking another server, the problem is not observed:

I compared requests that come to the servers and found that requests from the server in Linode do not reach spl89.hosting.reg.ru:

spl90.hosting.reg.ru:

172.104.24.29 - - [18/Jul/2022:10:11:32 +0300] spl90.hosting.reg.ru GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1 404 64721 "-" "Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)" "-" 0.131-0.009
172.104.24.29 - - [18/Jul/2022:10:11:32 +0300] spl90.hosting.reg.ru GET / HTTP/1.1 200 64153 "-" "Go-http-client/1.1" "-" 0.130-0.008
66.133.109.36 - - [18/Jul/2022:10:11:33 +0300] spl90.hosting.reg.ru GET /.well-known/acme-challenge/g-nRu0uab7ZWa07THcE33CpuxAy3oggUg8DCihVQoMo HTTP/1.1 404 45491 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 0.183-0.010

spl89.hosting.reg.ru:

66.133.109.36 - - [18/Jul/2022:10:10:43 +0300] spl89.hosting.reg.ru GET /.well-known/acme-challenge/zCAgrg64HUQwp1afJBLYudfn0D0iXmv2i-TuyHxuE38 HTTP/1.1 404 45491 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 0.189-0.011

Yesterday logs:

spl90.hosting.reg.ru:

172.104.24.29 - - [17/Jul/2022:20:54:31 +0300] spl90.hosting.reg.ru GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1 404 64721 "-" "Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)" "-" 0.137-0.012
52.28.77.66 - - [17/Jul/2022:20:54:32 +0300] spl90.hosting.reg.ru GET /.well-known/acme-challenge/CiXg786K2zTvDvcWA33GHExfam-Odk7aHPPURkIxNMQ HTTP/1.1 404 44103 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 0.058-0.009
172.104.24.29 - - [17/Jul/2022:20:54:32 +0300] spl90.hosting.reg.ru GET / HTTP/1.1 200 64153 "-" "Go-http-client/1.1" "-" 0.127-0.008
66.133.109.36 - - [17/Jul/2022:20:54:32 +0300] spl90.hosting.reg.ru GET /.well-known/acme-challenge/CiXg786K2zTvDvcWA33GHExfam-Odk7aHPPURkIxNMQ HTTP/1.1 404 45491 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 0.177-0.008

spl89.hosting.reg.ru:

52.28.77.66 - - [17/Jul/2022:20:53:45 +0300] spl89.hosting.reg.ru GET /.well-known/acme-challenge/mKNdhLna5zkjIkkAESka_Uor2K0EwH_Z-OZvNWaywjY HTTP/1.1 404 44103 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 0.062-0.010
66.133.109.36 - - [17/Jul/2022:20:53:45 +0300] spl89.hosting.reg.ru GET /.well-known/acme-challenge/mKNdhLna5zkjIkkAESka_Uor2K0EwH_Z-OZvNWaywjY HTTP/1.1 404 45491 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" 0.171-0.007

➜ ~ whois 172.104.24.29 | grep NetName
NetName: LINODE-US

According to mtr result, the problem should be on the side of the letsencrypt/letsdebug server located in Linode, or on the side of Linode infrsatructure:
mtr from spl90.hosting.reg.ru:

                                            My traceroute  [v0.85]
spl90.hosting.reg.ru (0.0.0.0)                                                       Mon Jul 18 10:35:08 2022
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                     Packets               Pings
 Host                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. kiae-r1.hosting.reg.ru (31.31.194.4)                            0.0%   274    0.9   1.8   0.2  60.8   7.0
 2. ???
 3. ???
 4. 198.18.9.152                                                    0.0%   274    0.7   1.0   0.6  15.7   1.7
 5. 10.1.92.1                                                       0.0%   274    1.3   1.4   0.7  13.7   1.7
 6. mow-b4-link.ip.twelve99.net (213.248.96.104)                    0.0%   274    2.1   1.5   0.7  28.3   3.0
 7. mow-b1-link.ip.twelve99.net (62.115.142.176)                    0.0%   274    2.2   2.3   1.6   8.8   1.0
 8. s-bb2-link.ip.twelve99.net (62.115.141.22)                      0.0%   274   19.6  19.8  19.6  20.4   0.0
 9. kbn-bb2-link.ip.twelve99.net (62.115.139.173)                   0.4%   274  105.6  66.5  43.1 677.5  80.2
10. nyk-bb2-link.ip.twelve99.net (80.91.254.91)                     2.2%   274  123.1 122.2 118.7 135.2   1.8
11. nyk-b15-link.ip.twelve99.net (62.115.113.71)                    0.0%   274  120.7 121.1 120.3 123.5   1.1
12. linode-ic342729-nyk-b2.ip.twelve99-cust.net (62.115.172.131)    0.0%   274  117.1 117.9 117.0 147.3   3.5
13. ???
14. ???
15. ???
16. letsdebug.net (172.104.24.29)                                   0.0%   274  113.2 112.6 110.3 113.9   1.0

mtr from spl89.hosting.reg.ru

                                            My traceroute  [v0.85]
spl89.hosting.reg.ru (0.0.0.0)                                                       Mon Jul 18 10:34:27 2022
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                     Packets               Pings
 Host                                                              Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. kiae-r1.hosting.reg.ru (31.31.194.4)                            0.0%   225    0.3   2.4   0.2  70.8   8.7
 2. ???
 3. ???
 4. 198.18.9.152                                                    0.4%   225    2.0   2.4   1.2  24.6   2.3
 5. 10.1.92.1                                                       0.0%   225    1.6   2.3   1.3  17.9   1.7
 6. mow-b4-link.ip.twelve99.net (213.248.96.104)                    0.0%   225    2.0   2.5   1.4  17.9   1.9
 7. mow-b1-link.ip.twelve99.net (62.115.142.176)                    0.0%   225    3.0   3.6   2.3  13.4   1.4
 8. s-bb2-link.ip.twelve99.net (62.115.141.22)                      0.0%   225   21.0  20.9  20.3  21.9   0.1
 9. kbn-bb2-link.ip.twelve99.net (62.115.139.173)                   0.0%   225   47.9  67.4  44.8 676.5  84.3
10. nyk-bb2-link.ip.twelve99.net (80.91.254.91)                     6.7%   225  114.8 117.5 114.8 119.5   0.9
11. nyk-b15-link.ip.twelve99.net (62.115.113.71)                    0.0%   225  114.7 117.2 114.3 118.8   0.9
12. linode-ic342729-nyk-b2.ip.twelve99-cust.net (62.115.172.131)    0.0%   225  115.0 116.1 114.9 151.4   3.9
13. ???

Can someone from Let's Encrypt team check if ip 31.31.198.201 blocked on Let's Encrypt servers or it is Linode's problem?

The IP is not blocked. That would cause a problem for you connecting to the LE servers but this is a problem of the LE servers contacting you.

That said, it is odd. You are seeing some acme challenge requests but not the number expected. And, if I try to request a cert for your domain I consistently get an http 404 failure - not a timeout. This means my tests reached your server consistently. Of course, I won't be able to get a cert but this test shows the communications are not consistent.

One guess is you might have a firewall that has some sort of "smart blocking" feature that is blocking repeated requests. As you see, Let's Encrypt will make the identical request from different IPs around the world to check if you control your domain. Maybe your firewall is seeing these repeated requests as a DDoS attack and blocking some requests.

Output from my test request. I show it only to prove to other volunteers I am not hallucinating :slight_smile: The 404 is expected since I am not running this on your server.

sudo certbot certonly --webroot -w /test/folder -d shtakesnyjbogperla.ru 

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: shtakesnyjbogperla.ru
  Type:   unauthorized
  Detail: 31.31.198.201: Invalid response from http://shtakesnyjbogperla.ru/.well-known/acme-challenge/SeJQQX6RjpHwbpoxt9VOnmvDJPtVjtJulJZI6nyVG9g: 404

EDIT:
Oh, I can't reach your site doing simple curl requests from an AWS region on US East Coast. This "feels" like a comms routing problem on your end more like a firewall rule.

curl -I -m20 http://shtakesnyjbogperla.ru
curl: (28) Connection timed out after 20001 milliseconds
6 Likes

@MikeMcQ

It isn't firewall problem, i checked it many times. Not only http request - any traffic from 172.104.24.29 not reaching spl89.hosting.reg.ru

Oh, I can't reach your site doing simple curl requests from an AWS region on US East Coast.

Please, could you send mtr result from your server to shtakesnyjbogperla.ru? And if possible, the ip address of your server (maybe in PM?), so that I can run mtr in the opposite direction to determine which part of the network the problem is.

EDIT:
And, if possoble - please attach mtr result to spl90.hosting.reg.ru, for comparison.

Debugging comms routing problems is pretty far from helping get and use a Let's Encrypt cert. But, below is an mtr to your domain. Other info sent privately,

                                My traceroute  [v0.92] TO shtakesnyjbogperla.ru
ip-172-31-61-226.ec2.internal (172.31.61.226)                2022-07-18T15:14:18+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                             Packets               Pings
 Host                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 216.182.229.207                         0.0%   264   20.0  18.0   0.8 113.7  22.8
 2. ???
 3. ???
 4. ???
 5. 241.0.4.140                             0.0%   263    0.4   0.4   0.3   2.3   0.2
 6. 240.0.28.0                              0.0%   263    0.4   0.4   0.4   3.9   0.3
 7. 243.254.3.1                             0.0%   263    0.5   0.4   0.3   4.5   0.3
 8. 240.192.10.99                           0.0%   263    0.4   0.4   0.3   1.8   0.1
 9. 240.0.28.22                             0.0%   263    0.4   0.4   0.3   3.9   0.3
10. 240.0.28.4                              0.0%   263    0.4  31.7   0.4 583.9  92.4
11. 242.0.146.33                            0.0%   263    0.5   1.5   0.3  32.6   3.8
12. 52.93.28.215                            0.0%   263    1.2   1.5   0.4  25.3   2.9
13. 100.100.2.12                            0.0%   263    0.5   1.1   0.4   9.4   1.6
14. ash-b2-link.ip.twelve99.net             0.8%   263    0.9   1.0   0.8   8.2   0.5
15. ash-bb2-link.ip.twelve99.net            0.0%   263    2.0   1.8   1.6   2.7   0.1
16. prs-bb1-link.ip.twelve99.net           23.2%   263   87.9  88.0  87.9  89.3   0.1
17. adm-bb1-link.ip.twelve99.net            0.0%   263   93.4  93.7  93.4  99.4   0.5
18. adm-b10-link.ip.twelve99.net            3.8%   263   92.1  92.8  91.8 138.1   4.5
19. ???

And, to spl90.hosting.reg.ru

                                My traceroute  [v0.92]
ip-172-31-61-226.ec2.internal (172.31.61.226)                2022-07-18T15:24:01+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                             Packets               Pings
 Host                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 244.5.1.121                             0.0%   113    1.5  13.5   0.7 110.0  22.1
 2. 240.3.20.97                             0.0%   113    0.5   0.4   0.3   0.6   0.1
 3. 243.254.31.137                          0.0%   113    0.5   0.4   0.4   2.5   0.2
 4. 240.192.194.240                         0.0%   113    0.4   0.4   0.3   1.9   0.2
 5. 240.3.20.31                             0.0%   113    0.4   0.4   0.3   1.1   0.1
 6. 240.3.20.1                              0.0%   113    0.4   0.4   0.3   0.9   0.1
 7. 240.0.60.5                              0.0%   113    0.4   0.5   0.4   5.8   0.6
 8. 243.254.3.141                           0.0%   113    0.5   0.4   0.4   1.5   0.1
 9. 240.192.18.115                          0.0%   113    0.4   0.4   0.4   1.8   0.1
10. 240.0.60.55                             0.0%   113    0.4   0.4   0.4   1.7   0.1
11. 240.0.60.34                             0.0%   113    0.4   0.4   0.4   1.1   0.1
12. 240.0.44.14                             0.0%   113    0.5   0.5   0.4   1.1   0.1
13. 243.254.4.133                           0.0%   113   10.2  11.2   0.9  23.0   6.6
14. 240.192.14.147                          0.0%   113    0.4   0.5   0.4   4.4   0.4
15. 240.0.44.25                             3.6%   113  708.3 710.3 609.5 752.0  26.0
16. 240.0.44.7                              0.0%   113   16.6  15.8   0.6 207.4  25.4
17. 242.0.179.161                           0.0%   113    0.5   1.9   0.4  24.5   4.1
18. 52.93.28.253                            0.0%   113    0.5   1.7   0.4  22.8   3.3
19. 100.100.34.32                           0.0%   113    0.5   1.4   0.5  16.5   2.9
20. ash-b1-link.ip.twelve99.net             0.0%   113    0.9   1.0   0.8   1.3   0.1
21. ash-bb2-link.ip.twelve99.net           37.2%   113    1.4   1.4   1.2   4.1   0.3
22. prs-bb1-link.ip.twelve99.net           20.5%   113   89.1  89.1  89.0  90.6   0.2
23. adm-bb1-link.ip.twelve99.net            0.0%   113   94.8  94.8  94.5  95.2   0.2
24. adm-b10-link.ip.twelve99.net            0.0%   112   93.2  95.9  93.0 138.0   9.8
25. ???
4 Likes

That IP is only for "Let's Debug"
It is NOT used by LE for authentication.

Only two current LE IPs are shown:

It should be three (or four) LE IPs.
Something is blocking some of the LE IPs:

4 Likes

The problem arose due to errors in routing on my side, found and corrected. Thank you all for your help.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.