Can't connect to acme-v02.api.letsencrypt.org

I'm sorry this problem continues. I can check again, but nothing should be changed on our side. Please DM any IPs that are having problems including IPv6.

6 Likes

Well, there was a happy plot twist :grin: . I've just checked again before sending the DM and now it is working :partying_face: I haven't touched the server since this morning so it seems that whatever was avoiding the connection, it was temporary.

Thank you all for your help and your suggestions this week :relaxed: @rg305 @jillian @bruncsak @JimPas @MikeMcQ You saved me from insanity :smiley:

Regards!

7 Likes

Hi there! We unfortunately started seeing what appears to be the same problem as CBImag was hitting, starting at 2021-12-03T20:05:21.264492Z, ramping up pretty sharply from there, and still ongoing. Our connections to acme-v02.api.letsencrypt.org are immediately getting closed, before starting the TLS handshake. Works fine from an identical setup running in a different VPC, so we suspect it's some IP-based filtering/blocking going on?

~ $ openssl version                                                            
OpenSSL 1.1.1f  31 Mar 2020
~ $ openssl s_client -connect acme-v02.api.letsencrypt.org:443 \                                     
> -servername acme-v02.api.letsencrypt.org -debug
CONNECTED(00000003)
write to 0x55c149122b90 [0x55c149132f10] (320 bytes => 320 (0x140))
0000 - 16 03 01 01 3b 01 00 01-37 03 03 83 17 2a 8a 0a   ....;...7....*..
0010 - 68 b4 36 11 c8 10 81 ae-4d 81 92 9f a7 f6 ec 3a   h.6.....M......:
0020 - d7 b5 d6 5a 98 1e 46 ff-5d c4 5b 20 b3 f6 3f 7d   ...Z..F.].[ ..?}
0030 - ec 2a 94 dd 48 5e 97 ac-96 e2 78 fa 53 6d 66 cb   .*..H^....x.Smf.
0040 - 31 79 3b ca 37 e1 4a e6-aa 40 7d 14 00 3e 13 02   1y;.7.J..@}..>..
0050 - 13 03 13 01 c0 2c c0 30-00 9f cc a9 cc a8 cc aa   .....,.0........
0060 - c0 2b c0 2f 00 9e c0 24-c0 28 00 6b c0 23 c0 27   .+./...$.(.k.#.'
0070 - 00 67 c0 0a c0 14 00 39-c0 09 c0 13 00 33 00 9d   .g.....9.....3..
0080 - 00 9c 00 3d 00 3c 00 35-00 2f 00 ff 01 00 00 b0   ...=.<.5./......
0090 - 00 00 00 21 00 1f 00 00-1c 61 63 6d 65 2d 76 30   ...!.....acme-v0
00a0 - 32 2e 61 70 69 2e 6c 65-74 73 65 6e 63 72 79 70   2.api.letsencryp
00b0 - 74 2e 6f 72 67 00 0b 00-04 03 00 01 02 00 0a 00   t.org...........
00c0 - 0c 00 0a 00 1d 00 17 00-1e 00 19 00 18 00 23 00   ..............#.
00d0 - 00 00 16 00 00 00 17 00-00 00 0d 00 2a 00 28 04   ............*.(.
00e0 - 03 05 03 06 03 08 07 08-08 08 09 08 0a 08 0b 08   ................
00f0 - 04 08 05 08 06 04 01 05-01 06 01 03 03 03 01 03   ................
0100 - 02 04 02 05 02 06 02 00-2b 00 05 04 03 04 03 03   ........+.......
0110 - 00 2d 00 02 01 01 00 33-00 26 00 24 00 1d 00 20   .-.....3.&.$... 
0120 - e7 07 d7 bc 25 97 f1 5b-62 be ed 84 95 bd d9 f4   ....%..[b.......
0130 - b2 30 b3 76 86 e4 00 2e-6b 19 1e f1 24 d0 31 6f   .0.v....k...$.1o
read from 0x55c149122b90 [0x55c149129c73] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
read from 0x55c149122b90 [0x55c149117f60] (8192 bytes => 0 (0x0))

Public IPs for the impacted VPC's NAT gateways:

34.193.80.114
34.193.80.31
34.193.31.190
34.193.80.117

Anything we can do to get these IPs unblocked? Thanks!

2 Likes

Please show the outputs of:

echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
echo | openssl s_client -connect google.com:443 | head

[while we wait for the IPs to be unblocked]

5 Likes

Thanks for taking a look!

~ $ echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
write:errno=104
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
~ $ echo | openssl s_client -connect google.com:443 | head                     
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:CN = *.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
DONE
3 Likes

That seems to confirm the IP block.

5 Likes

Hi, @david-at-heroku,

We had blocked two of these IPs; I've unblocked them for you. We were observing many, frequent, persistent attempts to validate hostnames that were always failing validation.

7 Likes

Thank you! I can confirm that things are working for us again now, and really appreciate the quick response!

The system in question generates certificates for apps hosted on Heroku. We certainly have a number of customers who attach a custom domain to their app without actually setting the DNS entries for that domain up correctly to point at us. The system does a number of pre-flight validations to try to confirm that everything looks good before it calls LetsEncrypt, sounds like maybe something is/was slipping through that filter? We'll dig into the logs on our side and see if we can find anything.

8 Likes

Hi,
I post on this topic because I think we have also our public IP blocked when we try re-new certificate : like @david-at-heroku , the return of my command

openssl s_client -connect acme-v02.api.letsencrypt.org:443

is

write:errno=104
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported

@JamesLE please can you check if this IP is blocked and unlock it if needed :

54.36.114.229

Thanks a lot.

4 Likes

Yes, this IP had been blocked because of many, repeated requests that were always failing validation. I've now unblocked it.

5 Likes

@JamesLE I post here because I think we have also our public IP blocked when we try re-new certificate. please can you check if this IP is blocked and unlock it if needed:

Pub IP Address: 51.15.121.96

Thanks a lot.

1 Like

Thank you @JamesLE for processing my request, I can renew certificate now, and it work well !

2 Likes

@JamesLE We have a suspicion that our IPs were blocked as well 18.246.31.224/28 and 35.162.54.42,
35.161.3.151, 35.162.23.98 We are getting connection timeouts starting at 7:00 UTC. Is it possible to check?

1 Like

It actually miraculously recovered. I initially suspected that some of our IPs from 18.246.31.224/28 range were blocked. It was difficult for me to test every one of them using your command above

1 Like

@JamesLE I post here because I think we have also our public IP blocked when we try re-new certificate. please can you check if this IP is blocked and unlock it if needed:

Pub IP Address: 51.15.121.96

Thanks a lot.

1 Like

@Sivakumar Please post the actual error message first, thank you.

5 Likes

We are seeing a similar problem.

[root@aisa ~]# echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
write:errno=104
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 254 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
[root@aisa ~]# echo | openssl s_client -connect google.com:443 | head
depth=3 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify return:1
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:/CN=*.google.com
   i:/C=US/O=Google Trust Services LLC/CN=GTS CA 1C3
 1 s:/C=US/O=Google Trust Services LLC/CN=GTS CA 1C3
   i:/C=US/O=Google Trust Services LLC/CN=GTS Root R1
 2 s:/C=US/O=Google Trust Services LLC/CN=GTS Root R1
   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
---
DONE

Server addresses:
5.8.79.138
5.8.79.140

2 Likes

We were blocking 5.8.79.138 (it's now unblocked), but none of the other new IP addresses in this thread.

6 Likes

A post was split to a new topic: Is my IP being blocked?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.