I'm sorry this problem continues. I can check again, but nothing should be changed on our side. Please DM any IPs that are having problems including IPv6.
Well, there was a happy plot twist . I've just checked again before sending the DM and now it is working I haven't touched the server since this morning so it seems that whatever was avoiding the connection, it was temporary.
Thank you all for your help and your suggestions this week @rg305 @jillian @bruncsak @JimPas @MikeMcQ You saved me from insanity
Regards!
Hi there! We unfortunately started seeing what appears to be the same problem as CBImag was hitting, starting at 2021-12-03T20:05:21.264492Z, ramping up pretty sharply from there, and still ongoing. Our connections to acme-v02.api.letsencrypt.org are immediately getting closed, before starting the TLS handshake. Works fine from an identical setup running in a different VPC, so we suspect it's some IP-based filtering/blocking going on?
~ $ openssl version
OpenSSL 1.1.1f 31 Mar 2020
~ $ openssl s_client -connect acme-v02.api.letsencrypt.org:443 \
> -servername acme-v02.api.letsencrypt.org -debug
CONNECTED(00000003)
write to 0x55c149122b90 [0x55c149132f10] (320 bytes => 320 (0x140))
0000 - 16 03 01 01 3b 01 00 01-37 03 03 83 17 2a 8a 0a ....;...7....*..
0010 - 68 b4 36 11 c8 10 81 ae-4d 81 92 9f a7 f6 ec 3a h.6.....M......:
0020 - d7 b5 d6 5a 98 1e 46 ff-5d c4 5b 20 b3 f6 3f 7d ...Z..F.].[ ..?}
0030 - ec 2a 94 dd 48 5e 97 ac-96 e2 78 fa 53 6d 66 cb .*..H^....x.Smf.
0040 - 31 79 3b ca 37 e1 4a e6-aa 40 7d 14 00 3e 13 02 1y;.7.J..@}..>..
0050 - 13 03 13 01 c0 2c c0 30-00 9f cc a9 cc a8 cc aa .....,.0........
0060 - c0 2b c0 2f 00 9e c0 24-c0 28 00 6b c0 23 c0 27 .+./...$.(.k.#.'
0070 - 00 67 c0 0a c0 14 00 39-c0 09 c0 13 00 33 00 9d .g.....9.....3..
0080 - 00 9c 00 3d 00 3c 00 35-00 2f 00 ff 01 00 00 b0 ...=.<.5./......
0090 - 00 00 00 21 00 1f 00 00-1c 61 63 6d 65 2d 76 30 ...!.....acme-v0
00a0 - 32 2e 61 70 69 2e 6c 65-74 73 65 6e 63 72 79 70 2.api.letsencryp
00b0 - 74 2e 6f 72 67 00 0b 00-04 03 00 01 02 00 0a 00 t.org...........
00c0 - 0c 00 0a 00 1d 00 17 00-1e 00 19 00 18 00 23 00 ..............#.
00d0 - 00 00 16 00 00 00 17 00-00 00 0d 00 2a 00 28 04 ............*.(.
00e0 - 03 05 03 06 03 08 07 08-08 08 09 08 0a 08 0b 08 ................
00f0 - 04 08 05 08 06 04 01 05-01 06 01 03 03 03 01 03 ................
0100 - 02 04 02 05 02 06 02 00-2b 00 05 04 03 04 03 03 ........+.......
0110 - 00 2d 00 02 01 01 00 33-00 26 00 24 00 1d 00 20 .-.....3.&.$...
0120 - e7 07 d7 bc 25 97 f1 5b-62 be ed 84 95 bd d9 f4 ....%..[b.......
0130 - b2 30 b3 76 86 e4 00 2e-6b 19 1e f1 24 d0 31 6f .0.v....k...$.1o
read from 0x55c149122b90 [0x55c149129c73] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
read from 0x55c149122b90 [0x55c149117f60] (8192 bytes => 0 (0x0))
Public IPs for the impacted VPC's NAT gateways:
34.193.80.114
34.193.80.31
34.193.31.190
34.193.80.117
Anything we can do to get these IPs unblocked? Thanks!
Please show the outputs of:
echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
echo | openssl s_client -connect google.com:443 | head
[while we wait for the IPs to be unblocked]
Thanks for taking a look!
~ $ echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
write:errno=104
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
~ $ echo | openssl s_client -connect google.com:443 | head
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
0 s:CN = *.google.com
i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
DONE
That seems to confirm the IP block.
Hi, @david-at-heroku,
We had blocked two of these IPs; I've unblocked them for you. We were observing many, frequent, persistent attempts to validate hostnames that were always failing validation.
Thank you! I can confirm that things are working for us again now, and really appreciate the quick response!
The system in question generates certificates for apps hosted on Heroku. We certainly have a number of customers who attach a custom domain to their app without actually setting the DNS entries for that domain up correctly to point at us. The system does a number of pre-flight validations to try to confirm that everything looks good before it calls LetsEncrypt, sounds like maybe something is/was slipping through that filter? We'll dig into the logs on our side and see if we can find anything.
Hi,
I post on this topic because I think we have also our public IP blocked when we try re-new certificate : like @david-at-heroku , the return of my command
openssl s_client -connect acme-v02.api.letsencrypt.org:443
is
write:errno=104
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
@JamesLE please can you check if this IP is blocked and unlock it if needed :
54.36.114.229
Thanks a lot.
Yes, this IP had been blocked because of many, repeated requests that were always failing validation. I've now unblocked it.
@JamesLE I post here because I think we have also our public IP blocked when we try re-new certificate. please can you check if this IP is blocked and unlock it if needed:
Pub IP Address: 51.15.121.96
Thanks a lot.
Thank you @JamesLE for processing my request, I can renew certificate now, and it work well !
@JamesLE We have a suspicion that our IPs were blocked as well 18.246.31.224/28 and 35.162.54.42,
35.161.3.151, 35.162.23.98 We are getting connection timeouts starting at 7:00 UTC. Is it possible to check?
It actually miraculously recovered. I initially suspected that some of our IPs from 18.246.31.224/28 range were blocked. It was difficult for me to test every one of them using your command above
@JamesLE I post here because I think we have also our public IP blocked when we try re-new certificate. please can you check if this IP is blocked and unlock it if needed:
Pub IP Address: 51.15.121.96
Thanks a lot.
We are seeing a similar problem.
[root@aisa ~]# echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
write:errno=104
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 254 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
[root@aisa ~]# echo | openssl s_client -connect google.com:443 | head
depth=3 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify return:1
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
0 s:/CN=*.google.com
i:/C=US/O=Google Trust Services LLC/CN=GTS CA 1C3
1 s:/C=US/O=Google Trust Services LLC/CN=GTS CA 1C3
i:/C=US/O=Google Trust Services LLC/CN=GTS Root R1
2 s:/C=US/O=Google Trust Services LLC/CN=GTS Root R1
i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
---
DONE
Server addresses:
5.8.79.138
5.8.79.140
We were blocking 5.8.79.138 (it's now unblocked), but none of the other new IP addresses in this thread.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.