Can't connect to acme-v02.api.letsencrypt.org

CBImag · November 29, 2021, 11:35am

Hello, we have several domains in our server, and when we try to create a certificate for a new one with certbot we're getting a handshake problem. We created the server a few weeks ago and until today we requested new certificates without any problems.

I've seen this link, and it seems that we are having that problem, so let me kindly tag you @lestaff

I don't know if it's because a problem with the previous owner of the ip, or if it's because of the several domains we have in our server.

Details:

My domain is: One of them is emblematic.es, but there are several ones and more to come. Our ip is 188.166.113.247

I ran this command:

We detected the problem using "certbot --apache", but the main problem is when requesting to the url:

curl -v https://acme-v02.api.letsencrypt.org/

It produced this output:

*   Trying 172.65.32.248:443...
* TCP_NODELAY set
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443

My web server is (include version): Apache 2.4.41

The operating system my web server runs on is (include version): Ubuntu 20.04.1

My hosting provider, if applicable, is: Digital Ocean

I can login to a root shell on my machine: Yes

I'm using a control panel to manage my site: No

The version of my client is: curl 7.68

Thanks!

rg305 · November 29, 2021, 1:48pm

Hi @CBImag and welcome to the LE community forum

I don't see the exact situation as in the other posted topic.
From your server, please show the outputs of:
echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
curl -Ii https://acme-v02.api.letsencrypt.org/directory

CBImag · November 29, 2021, 2:28pm

Hello @rg305, thanks!

Here are the outputs of those commands:

> echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head

write:errno=104
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)

> curl -Ii https://acme-v02.api.letsencrypt.org/directory

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443

Also, the output of the certbot command:

> certbot

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
An unexpected error occurred:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
    cnx.do_handshake()
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1640, in _raise_ssl_error
    raise SysCallError(-1, "Unexpected EOF")
OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
    raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')")))

During handling of the above exception, another exception occurred:

requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError("bad handshake: SysCallError(-1, 'Unexpected EOF')")))
Please see the logfiles in /var/log/letsencrypt for more details.

rg305 · November 29, 2021, 2:41pm

What say?:
certbot --version
openssl version

CBImag · November 29, 2021, 3:00pm

For cerbot:
certbot 0.40.0

For OpenSSL:
OpenSSL 1.1.1f 31 Mar 2020

rg305 · November 29, 2021, 4:18pm

Try switching to certbot via snap
See: Certbot | Certbot (eff.org)
[make sure you uninstall that version first]

CBImag · November 29, 2021, 6:30pm

Hello @rg305! I made a snapshot of the server and cloned it into another droplet, and there both curl and certbot commands are working so it seems to be something at network level, the only difference between the two servers is the ip.

jillian · November 29, 2021, 8:30pm

I'll check for the IP being blocked and any reason why we might have blocked it. Some times we block clients that are broken and stuck in request loops without any successful issuance.

jillian · November 30, 2021, 9:04pm

This IP is not blocked in our configurations.

CBImag · December 1, 2021, 12:57pm

Thanks @jillian for checking it!

I contacted Digital Ocean but they don't see either any reason for this blocking on their side. In fact they cloned the server, as I did the other day, and they confirmed that it works with same configuration but another ip They told me to ask you but I already did

bruncsak · December 1, 2021, 1:14pm

You may want to do one more test. Swap the IP addresses of the original and the cloned system. Then check if the problem is moving with the IP, or not.

CBImag · December 1, 2021, 2:40pm

Thanks for the advice @bruncsak

Unfortunately I think I can't put the original IP in the cloned server to test if there it doesn't work, as I can only swap floating ips, and that ip is not one.

However, I've assigned a floating ip to the original server and if I force curl to use that interface, it is working correctly So it seems to be something at network level with that ip

bruncsak · December 1, 2021, 3:32pm

Is outbound HTTPS working in general for other hosts? What does curl -v https://letsencrypt.org/ give?

CBImag · December 1, 2021, 3:47pm

Yep, for other hosts it's working without any problems, for example for letsencrypt.org:

*   Trying 18.159.128.50:443...
* TCP_NODELAY set
* Connected to letsencrypt.org (18.159.128.50) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=lencr.org
*  start date: Oct 10 03:00:44 2021 GMT
*  expire date: Jan  8 03:00:43 2022 GMT
*  subjectAltName: host "letsencrypt.org" matched cert's "letsencrypt.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5630cee73860)
> GET / HTTP/2
> Host: letsencrypt.org
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200 
< cache-control: public, max-age=0, must-revalidate
< content-security-policy: default-src 'none'; font-src 'self'; style-src 'unsafe-inline' 'self'; script-src 'unsafe-eval' 'unsafe-inline' 'self' data: https://www.google-analytics.com https://www.googleadservices.com https://www.googletagmanager.com https://googleads.g.doubleclick.net https://donorbox.org https://js.stripe.com/v3/ https://sdks.shopifycdn.com ; img-src 'self' data: blob: https://www.google-analytics.com https://www.paypal.com https://www.paypalobjects.com https://ak2s.abmr.net https://ak1s.abmr.net https://www.google.com https://cdn.shopify.com https://v.shopify.com ; frame-src https://donorbox.org https://www.youtube.com https://www.youtube-nocookie.com https://bid.g.doubleclick.net https://js.stripe.com/v3/ https://js.stripe.com/v2/ ; connect-src 'self' https://d4twhgtvn0ff5.cloudfront.net/ https://letsencrypt-merch.myshopify.com https://monorail-edge.shopifysvc.com ;
< content-type: text/html; charset=UTF-8
< date: Tue, 30 Nov 2021 23:02:58 GMT
< etag: "2510200eef2605575b5f53da49fcb4fa-ssl"
< permissions-policy: geolocation=(), midi=(), notifications=(), push=(), sync-xhr=(), microphone=(), camera=(), magnetometer=(), gyroscope=(), speaker=(self), vibrate=(), fullscreen=(self), interest-cohort=()
< referrer-policy: no-referrer
< strict-transport-security: max-age=31536000
< x-xss-protection: 1; mode=block
< age: 60106
< server: Netlify
< x-nf-request-id: 01FNVAPS8CBKQR6CMQZHXMCS45
< content-length: 34074
< x-content-type-options: nosniff
< x-frame-options: DENY
< 
and here goes the html content...

bruncsak · December 1, 2021, 4:12pm

Is the error immediate, or delayed (like time-out)?

rg305 · December 1, 2021, 4:59pm

7 posts were split to a new topic: Unable to validate certificate

CBImag · December 1, 2021, 4:30pm

Hello @bruncsak ! The error is immediate

JimPas · December 1, 2021, 5:10pm

Can you try this path? This is the starting point.

https://acme-v02.api.letsencrypt.org/directory

You should see this:

|||
|---|---|
|D32_oKU6bYU|https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417|
|keyChange|https://acme-v02.api.letsencrypt.org/acme/key-change|
|meta||
|caaIdentities||
|0|letsencrypt.org|
|termsOfService|https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf|
|website|https://letsencrypt.org|
|newAccount|https://acme-v02.api.letsencrypt.org/acme/new-acct|
|newNonce|https://acme-v02.api.letsencrypt.org/acme/new-nonce|
|newOrder|https://acme-v02.api.letsencrypt.org/acme/new-order|
|revokeCert|https://acme-v02.api.letsencrypt.org/acme/revoke-cert|

CBImag · December 2, 2021, 7:04am

Hello @JimPas ! I get the same error

> curl -v https://acme-v02.api.letsencrypt.org/directory
*   Trying 172.65.32.248:443...
* TCP_NODELAY set
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to acme-v02.api.letsencrypt.org:443

rg305 · December 2, 2021, 5:44pm

Please show:
ls -ltr /etc/ssl/certs/