Please restore TLSv1 access for now

mirabilos · August 22, 2021, 6:53pm

acme-v02.api.letsencrypt.org stopped accepting TLSv1 https connections. I’m not currently in a position to connect with TLSv1.2 from two or three systems but have installed all necessary mitigations and ciphersuite restrictions for TLSv1 to have a sufficient security level on my site. I’ll be working on porting a newer SSL library, but right now porting a newer SSH is more pressing because OpenSSH 8.8 will disable interop with the version these systems are still using. It’s a volunteer one-man project in spare time, so I need some time.

So please restore TLSv1 access for now, until, say, at least the end of the year (hoping I’ll manage to port things until then). Also, please mail subscribers before doing this kind of change in the future.

webprofusion · August 23, 2021, 1:31am

As workaround, I'd suggest you switch to renewing your certs from another machine (using DNS validation), then deploy your certs from there (via ssh?). This removes the need for your individual services to renew their own certs directly via the the Let's Encrypt API.

Since you appear to have an outdated OS are you certain you want these machines to be directly connected to the internet?

rg305 · August 23, 2021, 2:43am

What you need is a better design.

That says it all.
You could easily use even a Raspberry Pi and run up-to-date software on it.
Even if you chose to keep your existing system, you could simply use the RPi to proxy to the old one.
If you can't afford to even do that, then you could at least use Cloudflare CDN and turn on the firewalls to only allow trusted IPs in.

mirabilos · August 23, 2021, 2:51am

Hobbyist, not outdated. I also use SSL for SMTP and things like that, not just web stuff.

DNS… argh, no, I don’t have the time right now to invent a mechanism in which that one Debian rented VM I have (which is therefore much less trusted) can trigger updates for djbdns on my main BSD system…

I’ve already put porting a newer SSL library on the TODO, it’s just SSH is more pressing right now and I had to postpone that as well because I also am a Debian Developer and we kinda worked hard to produce a release the last few months, and I so don’t need this right now…

(Also, no, there was no need for newer SSL, I managed to avoid most the exploits because the code in question, such as DTLS, wasn’t even there in the first place, ported or wrote stuff for the rest, including SHA-2 certificate support. Right now, and for quite a while longer, if you’re not a bank or something, and especially if you don’t even serve per-user content or have a thing to even login, TLSv1 with the usual mitigations is definitely secure enough.)

ski192man · August 23, 2021, 3:48pm

There has been a huge coordinated effort by the entire security community to deprecate TLS versions prior to 1.2. The idea that Let's Encrypt would or should re-enable this and allow outdated software to continue to connect is quite frankly insane and would be a huge step backwards for internet security in general.

The easiest route would be to setup an outbound proxy that can speak tls 1.0 and 1.2 in your network for legacy compatibility.

petercooperjr · August 23, 2021, 5:13pm

I do think regardless of whether or not TLSv1.0 should still be around for some cases, I am very surprised that there may have been a change without anything being put in the API Announcements for it. I wonder if something upstream of them (Cloudflare maybe?) started blocking it without Let's Encrypt being aware.

@mirabilos, do you know when you last had a working TLSv1.0 connection to the API? I don't think it's something that I've ever tested, myself.

@jple, hoping it's okay to call you out: Can you comment on the communication plan, or lack thereof, on TLSv1.0 no longer working to connect to the Let's Encrypt API? Usually for changes like this there's an effort to see what portion of connections/clients are still using an old protocol that's going away and notifying affected users, isn't there?

jvanasco · August 23, 2021, 5:27pm

This happened with Python's PYPI in 2018. Their CDN announced plans to drop 1.0 and 1.1, but gave notice. They used rolling brownouts to prod users into upgrading.

That being said, I know several industries have outright banned 1.0 and 1.1 for security issues. I would not be surprised if there were a CA/B requirement, or LetsEncrypt did this for their own security concerns.

Phil · August 23, 2021, 5:33pm

I'm taking a look at this. We performed some loadbalancer work last week, but hadn't touched the served up cipher suites or protocols.

Phil · August 23, 2021, 8:11pm

We have a fix that'll end up in production within the week. This was an unintended side effect of system upgrades.

Phil · August 23, 2021, 9:20pm

@mirabilos The change is live in our staging environment. Can you confirm that one of your domains is able to successfully retrieve a certificate from there?

mirabilos · August 23, 2021, 10:05pm

@Phil first, thanks for confirming this was unintended, and getting back to me! Unfortunately, it does not work:

_acme@fish:/home/acme $ curl https://acme-v02.api.letsencrypt.org/directory
curl: (35) error:1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 alert protocol version
35|_acme@fish:/home/acme $ curl https://acme-staging-v02.api.letsencrypt.org/directory
curl: (35) error:1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 alert protocol version

(First one is to still check the other one.)

Maybe you can make something out of this:

$ openssl s_client -CApath /etc/ssl/certs -connect acme-staging-v02.api.letsencrypt.org:443 -debug
CONNECTED(00000004)
write to A1129EC0 [AA4CE000] (70 bytes => 70 (0x46))
0000 - 16 03 01 00 41 01 00 00-3d 03 01 61 24 1a b6 a7   ....A...=..a$...
0010 - 76 0f 92 8b 53 8c b4 cb-2d c1 bb 47 d2 eb 57 02   v...S...-..G..W.
0020 - b7 0d 30 42 58 0b e0 27-32 d7 bc 00 00 16 00 39   ..0BX..'2......9
0030 - 00 38 00 33 00 32 00 16-00 13 00 35 00 2f 00 0a   .8.3.2.....5./..
0040 - 00 05 00 04 01                                    .....
0046 - <SPACES/NULS>
read from A1129EC0 [AA4D3000] (5 bytes => 5 (0x5))
0000 - 15 03 01 00 02                                    .....
read from A1129EC0 [AA4D3005] (2 bytes => 2 (0x2))
0000 - 02 46                                             .F
29476:error:1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 alert protocol version:/usr/src/lib/libssl/ssl/../src/ssl/s3_pkt.c:1027:SSL alert number 70
29476:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:/usr/src/lib/libssl/ssl/../src/ssl/s3_pkt.c:490:

If it’s Debian, I have this…

tg@caas:~ $ tail -3 /etc/ssl/openssl.cnf 
[system_default_sect]
MinProtocol = TLSv1.0
CipherString = DEFAULT@SECLEVEL=1

… but as far as I remember, this would also work with the default seclevel of 2 (since I have longer DH parameters and/or this affects only the other side of the traffic), as long as the MinProtocol is lowered from TLSv1.2 to TLSv1.0.

@ski192man as far as I understand, LE requests are additionally signed/encrypted using per-user keys, so there should really be no concerns for traffic to the CA itself.

I’ve considered the proxy setup, and it indeed does seem to be the easiest one, but I had hoped to be able to avoid this until I can get another SSL library ported and working well and fitting the quality standards.

@petercooperjr I have a certificate with…

Not Before: Aug 15 07:15:58 2021 GMT

… so at least then. My cronjob is weekly and only alerted me the night I wrote this request.

petercooperjr · August 23, 2021, 10:19pm

I can confirm that TLSv1.0 is still not working in staging for me either.

$ curl --tls-max 1.0 https://acme-staging-v02.api.letsencrypt.org/directory
curl: (35) error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version

rg305 · August 23, 2021, 10:56pm

Back to back IPv4 requests:

curl -v4 --tlsv1.0 https://acme-staging-v02.api.letsencrypt.org/directory
*   Trying 172.65.46.172...
* TCP_NODELAY set
* Connected to acme-staging-v02.api.letsencrypt.org (172.65.46.172) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (IN), TLS alert, Server hello (2):
* error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version



 curl -v4 --tlsv1.0 https://acme-staging-v02.api.letsencrypt.org/directory
*   Trying 172.65.46.172...
* TCP_NODELAY set
* Connected to acme-staging-v02.api.letsencrypt.org (172.65.46.172) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Server hello (2):
* TLSv1.0 (IN), TLS handshake, Certificate (11):
* TLSv1.0 (IN), TLS handshake, Server key exchange (12):
* TLSv1.0 (IN), TLS handshake, Server finished (14):
* TLSv1.0 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.0 (OUT), TLS change cipher, Client hello (1):
* TLSv1.0 (OUT), TLS handshake, Finished (20):
* TLSv1.0 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.0 / ECDHE-RSA-AES256-SHA
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=acme-staging.api.letsencrypt.org
*  start date: Aug 17 17:39:27 2021 GMT
*  expire date: Nov 15 17:39:25 2021 GMT
*  subjectAltName: host "acme-staging-v02.api.letsencrypt.org" matched cert's "acme-staging-v02.api.letsencrypt.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55be23d18600)
> GET /directory HTTP/2
> Host: acme-staging-v02.api.letsencrypt.org
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< server: nginx
< date: Mon, 23 Aug 2021 22:55:20 GMT
< content-type: application/json
< content-length: 724
< cache-control: public, max-age=0, no-cache
< x-frame-options: DENY
< strict-transport-security: max-age=604800
<
{
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert",
  "vTRA8siEaNA": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
* Connection #0 to host acme-staging-v02.api.letsencrypt.org left intact

rg305 · August 23, 2021, 11:01pm

Now also happening on IPv6:

curl -v6 --tlsv1.0 https://acme-staging-v02.api.letsencrypt.org/directory
*   Trying 2606:4700:60:0:f41b:d4fe:4325:6026...
* TCP_NODELAY set
* Connected to acme-staging-v02.api.letsencrypt.org (2606:4700:60:0:f41b:d4fe:4325:6026) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (IN), TLS alert, Server hello (2):
* error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version



curl -v6 --tlsv1.0 https://acme-staging-v02.api.letsencrypt.org/directory
*   Trying 2606:4700:60:0:f41b:d4fe:4325:6026...
* TCP_NODELAY set
* Connected to acme-staging-v02.api.letsencrypt.org (2606:4700:60:0:f41b:d4fe:4325:6026) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Server hello (2):
* TLSv1.0 (IN), TLS handshake, Certificate (11):
* TLSv1.0 (IN), TLS handshake, Server key exchange (12):
* TLSv1.0 (IN), TLS handshake, Server finished (14):
* TLSv1.0 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.0 (OUT), TLS change cipher, Client hello (1):
* TLSv1.0 (OUT), TLS handshake, Finished (20):
* TLSv1.0 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.0 / ECDHE-RSA-AES256-SHA
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=acme-staging.api.letsencrypt.org
*  start date: Aug 17 17:39:27 2021 GMT
*  expire date: Nov 15 17:39:25 2021 GMT
*  subjectAltName: host "acme-staging-v02.api.letsencrypt.org" matched cert's "acme-staging-v02.api.letsencrypt.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56263ec00600)
> GET /directory HTTP/2
> Host: acme-staging-v02.api.letsencrypt.org
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< server: nginx
< date: Mon, 23 Aug 2021 23:00:21 GMT
< content-type: application/json
< content-length: 724
< cache-control: public, max-age=0, no-cache
< x-frame-options: DENY
< strict-transport-security: max-age=604800
<
{
  "Zs0zShQjgKg": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host acme-staging-v02.api.letsencrypt.org left intact

mirabilos · August 23, 2021, 11:53pm

It does seem to work occasionally, only to then break again:

_acme@fish:/home/acme $ curl -v4 https://acme-staging-v02.api.letsencrypt.org/directory
* About to connect() to acme-staging-v02.api.letsencrypt.org port 443 (#0)
*   Trying 172.65.46.172... connected
* Connected to acme-staging-v02.api.letsencrypt.org (172.65.46.172) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES128-SHA
* Server certificate:
*        subject: /CN=acme-staging.api.letsencrypt.org
*        start date: 2021-08-17 17:39:27 GMT
*        expire date: 2021-11-15 17:39:25 GMT
*        subjectAltName: acme-staging-v02.api.letsencrypt.org matched
*        issuer: /C=US/O=Let's Encrypt/CN=R3
* SSL certificate verify ok.
> GET /directory HTTP/1.1
> User-Agent: curl/7.18.2 (i386-ecce-mirbsd10) libcurl/7.18.2 OpenSSL/0.9.7m zlib/1.2.8.f-MirOS
> Host: acme-staging-v02.api.letsencrypt.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Mon, 23 Aug 2021 23:51:31 GMT
< Content-Type: application/json
< Content-Length: 724
< Connection: keep-alive
< Cache-Control: public, max-age=0, no-cache
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
<
{
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert",
  "tH1sdu1rdQs": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
* Connection #0 to host acme-staging-v02.api.letsencrypt.org left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
_acme@fish:/home/acme $ curl -v4 https://acme-staging-v02.api.letsencrypt.org/directory
* About to connect() to acme-staging-v02.api.letsencrypt.org port 443 (#0)
*   Trying 172.65.46.172... connected
* Connected to acme-staging-v02.api.letsencrypt.org (172.65.46.172) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS alert, Server hello (2):
* error:1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 alert protocol version
* Closing connection #0
curl: (35) error:1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 alert protocol version

Same with v6, one out of six back-to-back attempts worked.

Phil · August 24, 2021, 2:41am

I've got a proper fix deployed to the staging environment. It took a lot of back and forth to figure out the interaction between openssl and nginx. Most of the time was spent wrangling the openssl configuration itself. Order definitely matters in that file. The breakage/working was due to being round robined between loadbalancers and me iterating on a change in dev and then staging. Perhaps I should have dropped that staging LB out of service so no-one would be wise to it . This will function the same on IPv4 and IPv6 due to how we build our vhosts.

$ curl -L https://acme-staging-v02.api.letsencrypt.org/directory --verbose --tlsv1.0 --tls-max 1.1
*   Trying 172.65.46.172:443...
* Connected to acme-staging-v02.api.letsencrypt.org (172.65.46.172) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.1 (OUT), TLS handshake, Client hello (1):
* TLSv1.1 (IN), TLS handshake, Server hello (2):
* TLSv1.1 (IN), TLS handshake, Certificate (11):
* TLSv1.1 (IN), TLS handshake, Server key exchange (12):
* TLSv1.1 (IN), TLS handshake, Server finished (14):
* TLSv1.1 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.1 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.1 (OUT), TLS handshake, Finished (20):
* TLSv1.1 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.1 / ECDHE-RSA-AES256-SHA
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=acme-staging.api.letsencrypt.org
*  start date: Aug 17 17:39:27 2021 GMT
*  expire date: Nov 15 17:39:25 2021 GMT
*  subjectAltName: host "acme-staging-v02.api.letsencrypt.org" matched cert's "acme-staging-v02.api.letsencrypt.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x64b2d84ea110)
> GET /directory HTTP/2
> Host: acme-staging-v02.api.letsencrypt.org
> user-agent: curl/7.69.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Tue, 24 Aug 2021 02:40:10 GMT
< content-type: application/json
< content-length: 724
< cache-control: public, max-age=0, no-cache
< x-frame-options: DENY
< strict-transport-security: max-age=604800
< 
{
  "Glsvow850MI": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host acme-staging-v02.api.letsencrypt.org left intact
}```

mirabilos · August 24, 2021, 3:08am

Thanks, that looks like it will solve the problem. I really appreciate that you took the time to help with this; is there a donation pot I can tip somewhere?

rg305 · August 24, 2021, 3:12am

Donate - Let's Encrypt (letsencrypt.org)
https://letsencrypt.org/donate/

jcjones · August 24, 2021, 4:20pm

Excellent. That said, just be aware we are going to require at least TLS v1.2, probably early-to-mid in Q4. We'll announce a timeline soon in the API Announcements category.

mirabilos · August 24, 2021, 4:52pm

Right. I’d really appreciate if you could postpone that to 2022Q1 though. Less pressure makes for more attention to detail.