Renewal failure: EOF occured in violation of protocol

Help
1

Hello eveyone!

I am having some trouble renewing my certificate and after lots of attempts, upgrading certbot, updating everything else that I could, I am stumped.

My server is available in a browser, nothing has changed (except the updates, but those were done after it failed the first time) since the last renewal a few months ago.

I can see my HTTP server trying to process the request but returning 400:

205.210.31.138 - - [19/Apr/2022 12:56:29] code 400, message Bad HTTP/0.9 request type ('\x16\x03\x01\x00\xca\x01\x00\x00\xc6\x03\x035A\xd3\xcb\xf4\xf7\xf6?\x16I\x14"\xc5?mK%\xf1\xd0\xd6\xca\xf6pa\x1bF\x08\xefH\x8c\xb6\x17\x00\x00h\xcc\x14\xcc\x13\xc0/\xc0+\xc00\xc0,\xc0\x11\xc0\x07\xc0'\xc0#\xc0\x13\xc0')
205.210.31.138 - - [19/Apr/2022 12:56:29] "��5A�����?I"�?mK%�����p�H��h���/�+�0�,���'�#�� �(�$��" 400 -

Here's the rundown of the situation:

My domain is:
scviper.asuscomm.com

I ran this command:
sudo certbot certonly --webroot -w /home/viper/Downloads/ -d scviper.asuscomm.com

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
python SimpleHTTPServer

The operating system my web server runs on is (include version):
Ubuntu 20.04.4 LTS

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.26.0

2

Welcome to the community @scviper

I cannot reach your http server and neither can Let's Debug

Right now you are rate limited due to too many failures so you need to wait an hour before more certbot tests. You could add --dry-run to your command to use the testing system right now (and whenever testing).

A curl request gets this:

curl -I scviper.asuscomm.com
curl: (7) Failed to connect to scviper.asuscomm.com port 80 after 215 ms: Connection refused
3 Likes
3

Hi Mike,

I apologize, I had stopped the server out of habit, I usually only leave it on for certificate renewals.

I have started it again now and will leave it on until this is solved.

1 Like
4

OK. I see your server now. What happens when you try the command with --dry-run on the end?

Note the log entry you show was not from Let's Encrypt server. After trying dry-run check your logs again for a request with a path of /.well-known/acme-challenge/...

2 Likes
5

I tried with --dry-run, but the request is not reaching my server.

This is the result:

sudo certbot certonly --webroot -w /home/viper/Downloads/ -d scviper.asuscomm.com --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

6

Thanks. Your DNS points to IP 188.25.160.26

Can you confirm that is your current public IP? Your IP may be blocked. Can run this to confirm your IP. Let us know.

curl -4 ifconfig.co
2 Likes
7

curl -4 ifconfig.co
188.25.160.26

The IP is correct!

1 Like
8

@lestaff Can you check whether this IP is blocked? Thanks

2 Likes
9

This looks like you are trying to send a HTTPS request to a HTTP server.

2 Likes
10

This is not a solution to your integration, but a possible emergency bandage:

If you want to try and get things running ASAP... consider turning off your webserver and running certbot in standalone mode (User Guide — Certbot 1.26.0 documentation). That will have Certbot bind to port 80 and use it's own webserver to handle the challenge. If that works, the issue is on your webserver integration and you'll have a certificate that gets you back up and running.

If it doesn't work, then the error is elsewhere and someone else may have an idea. Considering you're on Ubuntu20.4 and Certbot 1.26, though - and the general nature of your ssl error - I think the issue is generally in your webserver integration.

2 Likes
11

@jvanasco thank you for the suggestion.

This is what happens when I try standalone:

sudo certbot certonly -d scviper.asuscomm.com --dry-run --standalone
Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

12

I still think it's possible your IP is blocked. I am surprised LEstaff has not responded yet (either way).

As further diagnosis, what do these do?

curl -I https://acme-v02.api.letsencrypt.org/directory

curl -I https://google.com
3 Likes
13

Google is fine, but letsencrypt not. Luckily, I am not in a rush to get this solved, but it would be nice if it happened :slight_smile:

curl -I https://acme-v02.api.letsencrypt.org/directory
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443

curl -I https://google.com
HTTP/2 301
location: https://www.google.com/
content-type: text/html; charset=UTF-8
date: Tue, 19 Apr 2022 20:22:20 GMT
expires: Tue, 19 Apr 2022 20:22:20 GMT
cache-control: private, max-age=2592000
server: gws
content-length: 220
x-xss-protection: 0
x-frame-options: SAMEORIGIN
set-cookie: CONSENT=PENDING+037; expires=Thu, 18-Apr-2024 20:22:20 GMT; path=/; domain=.google.com; Secure
p3p: CP="This is not a P3P policy! See P3P and Google's cookies - Google Account Help for more info."
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

1 Like
14

If that's the case, @MikeMcQ is probably right and it's an IP issue. The request looks like a protocol mismatch, so I wonder how LE implemented the IP blocking to create a 400 error like that.

3 Likes
15

I believe the 400 error in the log was an ill-behaved "bot" trying their server and not LE (see my post #4). We should have seen cleaner acme challenges in the log and never did (even test ones from my server). Also, the originating IP in the log pointed to google domains :slight_smile:

3 Likes
16

This IP is not blocked by Let's Encrypt.

5 Likes
17

If the IP is not blocked I don't have any good ideas on how to proceed. Hopefully another volunteer will. You might try using -v with curl and check diffs between the failing LE and google or other working URLs - like even this forum site. Please post anything curious.

2 Likes
18

agreed, next step:
curl -v https://acme-v02.api.letsencrypt.org/directory

acme-v02.api.letsencrypt.org should only resolve to an IPv4 address...
While google.com has both (IPv4 and IPv6)...
So maybe there is an IPv4 problem?
To test that, try:
curl -I4 https://google.com

3 Likes
19

Looks like it has IPv6 now but agree worth trying both explicitly (and maybe diff TLS versions too)

curl -v6 https://acme-v02.api.letsencrypt.org/directory
*   Trying 2606:4700:60:0:f53d:5624:85c7:3a2c:443...
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
( and proceeds normally from here )
2 Likes
20

First of all: I want to thank everyone for the help and suggestions :slight_smile:

Now for the curls:

goole.com returns 301:

curl -I4 https://google.com
HTTP/2 301
location: https://www.google.com/
content-type: text/html; charset=UTF-8
date: Wed, 20 Apr 2022 19:16:43 GMT
expires: Wed, 20 Apr 2022 19:16:43 GMT
cache-control: private, max-age=2592000
server: gws
content-length: 220
x-xss-protection: 0
x-frame-options: SAMEORIGIN
set-cookie: CONSENT=PENDING+695; expires=Fri, 19-Apr-2024 19:16:43 GMT; path=/; domain=.google.com; Secure
p3p: CP="This is not a P3P policy! See P3P and Google's cookies - Google Account Help for more info."
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

www.google.com returns 200:

curl -I4 https://www.google.com
HTTP/2 200
content-type: text/html; charset=ISO-8859-1
p3p: CP="This is not a P3P policy! See P3P and Google's cookies - Google Account Help for more info."
date: Wed, 20 Apr 2022 19:17:47 GMT
server: gws
x-xss-protection: 0
x-frame-options: SAMEORIGIN
expires: Wed, 20 Apr 2022 19:17:47 GMT
cache-control: private
set-cookie: AEC=AakniGMZZIivaoXE-jR0YwFNrmKuorm5Utwy1u2DMPGtT_mAQkqsWuT46w; expires=Mon, 17-Oct-2022 19:17:47 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
set-cookie: CONSENT=PENDING+945; expires=Fri, 19-Apr-2024 19:17:47 GMT; path=/; domain=.google.com; Secure
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

letsencrypt api encounters SSL_ERROR_SYSCALL somewhere along the way:

curl -v https://acme-v02.api.letsencrypt.org/directory

  • Trying 172.65.32.248:443...
  • TCP_NODELAY set
  • Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443
  • Closing connection 0
    curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443

community.letsencrypt.org is successful and returns the entire page:

curl -v https://community.letsencrypt.org

  • Trying 64.71.144.203:443...
  • TCP_NODELAY set
  • Connected to community.letsencrypt.org (64.71.144.203) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=community.letsencrypt.org
  • start date: Mar 12 00:00:02 2022 GMT
  • expire date: Jun 10 00:00:01 2022 GMT
  • subjectAltName: host "community.letsencrypt.org" matched cert's "community.letsencrypt.org"
  • issuer: C=US; O=Let's Encrypt; CN=R3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x55e5058f7e30)

GET / HTTP/2
Host: community.letsencrypt.org
user-agent: curl/7.68.0
accept: /

... page content removed for brevity...

* Connection #0 to host community.letsencrypt.org left intact
1 Like