Renewal failure: EOF occured in violation of protocol

Just to rule out proxy or other issues... can you confirm your IP address matches the IP address external services see? for example, Your IP is - IP And Computer Lookup

1 Like

I do see that too.
Sometimes resolves to IPv4 only.
Sometimes resolves to IPv6 only
Sometimes resolves to both.
[weird]

2 Likes

Yes, sends the Client Hello to acme-v02 but does not receive the Server hello back like when connecting to community.letsencrypt.org (below)

Should probably show result of this:

curl --version

And what about trying this. Does it fail same way?

curl --tlsv1.2 -v https://acme-v02.api.letsencrypt.org/directory
1 Like

try curl also individually (with -4 and -6)

1 Like

So, Your IP is - IP And Computer Lookup reports 188.25.160.26
My router shows: 188.25.160.26

The linux OS is running inside a VMWare shell on a PC connected directly to the router, and port 80 is forwarded (via router config) directly to port 8000 of the linux machine, on which my python HTTP server is running on.

This config has not changed in the past years, ever since I started getting the certificates.

If I run

I can see the requests in my python HTTP server:

172.104.24.29 - - [20/Apr/2022 20:19:09] code 404, message File not found
172.104.24.29 - - [20/Apr/2022 20:19:09] "GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1" 404 -
3.65.2.107 - - [20/Apr/2022 20:19:10] code 404, message File not found
3.65.2.107 - - [20/Apr/2022 20:19:10] "GET /.well-known/acme-challenge/IgT5f5e_e6vcN_pljaJrnVHo3fPn-olOSJIwmhyUrGw HTTP/1.1" 404 -
3.141.38.206 - - [20/Apr/2022 20:19:10] code 404, message File not found
3.141.38.206 - - [20/Apr/2022 20:19:10] "GET /.well-known/acme-challenge/IgT5f5e_e6vcN_pljaJrnVHo3fPn-olOSJIwmhyUrGw HTTP/1.1" 404 -
54.202.126.141 - - [20/Apr/2022 20:19:10] code 404, message File not found
54.202.126.141 - - [20/Apr/2022 20:19:10] "GET /.well-known/acme-challenge/IgT5f5e_e6vcN_pljaJrnVHo3fPn-olOSJIwmhyUrGw HTTP/1.1" 404 -
66.133.109.36 - - [20/Apr/2022 20:19:10] code 404, message File not found
66.133.109.36 - - [20/Apr/2022 20:19:10] "GET /.well-known/acme-challenge/IgT5f5e_e6vcN_pljaJrnVHo3fPn-olOSJIwmhyUrGw HTTP/1.1" 404 -
172.104.24.29 - - [20/Apr/2022 20:19:22] "GET / HTTP/1.1" 200 -

2 Likes

Quick question regarding the let's debug test: should I be worried about the rate limit?

asuscomm.com is the domain offered by Asus for their routers, it's not my own domain.

RateLimit

ERROR

scviper.asuscomm.com is currently affected by Let's Encrypt-based rate limits (Rate Limits - Let's Encrypt). You may review certificates that have already been issued by visiting crt.sh | %asuscomm.com . Please note that it is not possible to ask for a rate limit to be manually cleared.

The 'Certificates per Registered Domain' limit (50 certificates per week that share the same Registered Domain: asuscomm.com) has been exceeded. There is no way to work around this rate limit. The next non-renewal certificate for this Registered Domain should be issuable after 2022-04-20 20:19:05 +0000 UTC (0s from now).

Yes. You are subject to the rate limit for anyone who uses the apex domain (asuscomm.com). Asus should apply for a rate limit exception form on this page.

Personally, I would get a new domain name and set it up with a DNS provider that has an API so you could use it with DNS acme challenge if you ever want a wildcard cert. Such as Cloudflare

UPDATE: But, this rate limit is not causing the connection error to acme-v02 endpoint

2 Likes

I see no difference between requests with --tlsv1.2 and without.

curl --version

curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

curl --tlsv1.2 -v4 https://acme-v02.api.letsencrypt.org/directory

  • Trying 172.65.32.248:443...
  • TCP_NODELAY set
  • Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443
  • Closing connection 0
    curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443

curl --tlsv1.2 -v6 https://acme-v02.api.letsencrypt.org/directory

  • Trying 2606:4700:60:0:f53d:5624:85c7:3a2c:443...
  • TCP_NODELAY set
  • Immediate connect fail for 2606:4700:60:0:f53d:5624:85c7:3a2c: Network is unreachable
  • Closing connection 0
    curl: (7) Couldn't connect to server
1 Like

This is probably the last bit of ideas I have...

What do you see from dig?

  dig acme-v02.api.letsencrypt.org

I get:

;; ANSWER SECTION:
acme-v02.api.letsencrypt.org. 5112 IN CNAME prod.api.letsencrypt.org.
prod.api.letsencrypt.org. 300 IN CNAME 
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com. 284 IN A 172.65.32.248

What do you see from ping, the ip and domain name?

ping 172.65.32.248
ping acme-v02.api.letsencrypt.org

Mine go though:

iMac-314:~ jvanasco$ ping acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248): 56 data bytes
64 bytes from 172.65.32.248: icmp_seq=0 ttl=54 time=34.264 ms

And what about traceroute?

traceroute acme-v02.api.letsencrypt.org

Hopefully the above will confirm that you do have an ip based block, like @MikeMcQ suggested, and traceroute may show us where your traffic seems to be blocked.

3 Likes

I think you are right @jvanasco

Ping seems to be fine, but traceroute seems to be blocked, see results below:

dig acme-v02.api.letsencrypt.org

; <<>> DiG 9.16.1-Ubuntu <<>> acme-v02.api.letsencrypt.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44754
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;acme-v02.api.letsencrypt.org. IN A

;; ANSWER SECTION:
acme-v02.api.letsencrypt.org. 4424 IN CNAME prod.api.letsencrypt.org.
prod.api.letsencrypt.org. 235 IN CNAME ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com. 197 IN A 172.65.32.248

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Apr 20 20:58:39 UTC 2022
;; MSG SIZE rcvd: 155

ping acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=58 time=7.49 ms

traceroute acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 router.asus.com (192.168.1.1) 0.229 ms 0.299 ms 0.332 ms
2 StamAcasa.rdsnet.ro (10.0.0.1) 1.457 ms 1.503 ms 1.478 ms
3 qr90.bucuresti.rdsnet.ro (172.19.212.241) 1.780 ms 1.806 ms 1.786 ms
4 10.220.188.218 (10.220.188.218) 2.185 ms 2.334 ms 2.484 ms
5 10.220.149.27 (10.220.149.27) 2.187 ms static-10-220-142-135.rdsnet.ro (10.220.142.135) 2.221 ms 2.315 ms
6 86-120-70-185.rdsnet.ro (86.120.70.185) 3.356 ms 2.230 ms 2.346 ms
7 * * *
...
30 * * *

3 Likes

Wow this looks odd to me. It looks like you're getting through to their systems, but something on ISRG's server(s) is blocking you.

The traceroute output is expected. it often gets murky at the end because UDP gets blocked.

Can you try traceroute -I acme-v02.api.letsencrypt.org (uppercase i) to force ICMP? I just want to be sure before flagging staff again.

2 Likes

Certainly, here you go:
(the I is uppercase, but doesn't show properly here when pasted)

traceroute -I acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 router.asus.com (192.168.1.1) 0.253 ms 0.440 ms 0.471 ms
2 StamAcasa.rdsnet.ro (10.0.0.1) 1.650 ms 1.728 ms 1.724 ms
3 qr90.bucuresti.rdsnet.ro (172.19.212.241) 18.320 ms 18.346 ms 18.347 ms
4 10.220.188.218 (10.220.188.218) 2.138 ms 2.719 ms 2.808 ms
5 10.220.184.206 (10.220.184.206) 2.166 ms 2.164 ms 2.204 ms
6 * * *
7 172.65.32.248 (172.65.32.248) 1.814 ms 1.497 ms 1.564 ms

2 Likes

If you use ``` surrounding your paste, it will render in a monospace font where this distinction is also easier to see.

4 Likes

Good tip, thank you! I edited my post to reflect this :slight_smile:

3 Likes

well, everything looks to me like you're getting as far as ISRG's CDN (Cloudflare), and something beyond that point is dropping the connection. An IP block should block your pings, but maybe it was implemented differently.

I'm tagging @lestaff again, because the traceroute and ping might be informative. There's really nothing that any community volunteers can help with at this point, because we don't have access to the networks in question.

4 Likes

Is there any Anti-Virus at play?

1 Like

Thanks @jvanasco , @rg305 , @MikeMcQ , everyone chiming in on this. This one was wacky.

I've done some more digging and this IP was actually blocked by Let's Encrypt, but it was blocked by a netmask of a /29 and thus eluded the earlier searches. Our automated tools don't block by networks, just individual IPs,. I actually didn't realize we had any network-blocks at all in the list, but I was frustrated unable to figure out what was going on and combed through the output with my eyes and not with my grep.

This block didn't have any notes associated with it as to when it got blocked or why, so I think we must have done it rapidly in response to something bad in the logs. Typically that means a client that isn't respecting HTTP code 429 after hitting a rate limit. Could easily have been someone else with that IP a long time ago, since I've no metadata to go off of.

It's now unblocked. I'm sorry for the inconvenience.

11 Likes

I had a funny feeling earlier today there could be an ip range block in play. Glad this got sorted out. It is wacky indeed… so wacky I kept checking back on this site today to see WTF was happening!

3 Likes

@jcjones: thank you very much, this solved my issue. I renewed the certificate successfully.

Just to clear a bit of the mystery on why the IP range could have blocked: my provider, Digi Romania (also known as RDS-RCS) does not provide static IPs for private users, it only provides dynamic DNS.

It is very possible someone in this range of IPs spammed the renewal in the past and got blocked. After this their IP changed and they kept spamming, to the point where a netmask ban was applied. At least that's my theory.

@MikeMcQ, @jvanasco, @rg305, @schoen and everyone from staff who helped, thank you so very much, you guys are great!

And just so that I leave no question unanswered:

@rg305: there's no antivirus at play :slight_smile:

6 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.