Verify return code: 67 (CA certificate key too weak)

Hi all,

When running certbot renew, I'm getting a failure in talking to acme-v02.api.letsencrypt.org:443.

Certbot logs show:

Failed to renew certificate mail.crc.id.au with error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1000)')))

If I try to access the directory URL with curl, it fails with a similar error:

$ curl -v https://acme-v02.api.letsencrypt.org/directory
* Host acme-v02.api.letsencrypt.org:443 was resolved.
* IPv6: 2606:4700:60:0:f53d:5624:85c7:3a2c
* IPv4: 172.65.32.248
*   Trying [2606:4700:60:0:f53d:5624:85c7:3a2c]:443...
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, bad certificate (554):
* SSL certificate problem: EE certificate key too weak
* Closing connection
curl: (60) SSL certificate problem: EE certificate key too weak
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I'm doing all this on Fedora 40 - and can't say I've run into this before...

Has anyone come across this and managed to fix it somehow?

1 Like

could you try openssl s_client -connect acme-v02.api.letsencrypt.org:443 on that server and post result for this?

2 Likes
openssl s_client -connect acme-v02.api.letsencrypt.org:443
Connecting to 2606:4700:60:0:f53d:5624:85c7:3a2c
CONNECTED(00000003)
depth=0 CN=acme-v02.api.letsencrypt.org
verify error:num=66:EE certificate key too weak
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=R10
verify error:num=67:CA certificate key too weak
verify return:1
depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=R10
verify return:1
depth=0 CN=acme-v02.api.letsencrypt.org
verify return:1
---
Certificate chain
 0 s:CN=acme-v02.api.letsencrypt.org
   i:C=US, O=Let's Encrypt, CN=R10
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 16:03:46 2024 GMT; NotAfter: Dec  3 16:03:45 2024 GMT
 1 s:C=US, O=Let's Encrypt, CN=R10
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFqjCCBJKgAwIBAgISBL4p77MSVouOsQZQINLffau6MA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTAwHhcNMjQwOTA0MTYwMzQ2WhcNMjQxMjAzMTYwMzQ1WjAnMSUwIwYDVQQD
ExxhY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAoVc5mou51gqcaVsHsopoMyKpwqxF7O9ZIXYcni1yl8gk
A7/i+RDm3b2EixnhEbcFDMcjiPgsoHbbWzj3g4GRXFJVFDQnr9w+VL0qXvU9A3Sm
wOg/s6Vph7y0KymfS25h0scM8S8o+nUhuyoeqK5T9YuZjmSNnRI5nMGWeramPU1p
JUVeKp4x6YmP9bpOQs8yGqS8UZ/xlcXF4P21Oz2fI2KjIIg48782U5R7M+XoguIV
SEPHeYfwpGFnojFEb3jHYWaSORx9vvn1E01r0LOuHn7Udx5z4XzcRggMtPmuGNCi
cw8sLb7nGo+FQzAzYX28zAAf9TJAMH1u51SQ9IAK6QIDAQABo4ICwjCCAr4wDgYD
VR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNV
HRMBAf8EAjAAMB0GA1UdDgQWBBR50/M0MZvkR5623tTs8+Pyq/vT/TAfBgNVHSME
GDAWgBS7vMNHpeS8qcbDpHIMEI2iNeHI6DBXBggrBgEFBQcBAQRLMEkwIgYIKwYB
BQUHMAGGFmh0dHA6Ly9yMTAuby5sZW5jci5vcmcwIwYIKwYBBQUHMAKGF2h0dHA6
Ly9yMTAuaS5sZW5jci5vcmcvMIHJBgNVHREEgcEwgb6CHmFjbWUtdjAyLTEuYXBp
LmxldHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItMi5hcGkubGV0c2VuY3J5cHQub3Jn
gh5hY21lLXYwMi0zLmFwaS5sZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAyLTQuYXBp
LmxldHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItNS5hcGkubGV0c2VuY3J5cHQub3Jn
ghxhY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnMBMGA1UdIAQMMAowCAYGZ4EM
AQIBMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHUASLDja9qmRzQP5WoC+p0w6xxS
ActW3SyB2bu/qznYhHMAAAGRvfymUwAABAMARjBEAiAt70lY2Z21hM/S4cmqHEIl
2b6i7CWuPctvTsCrosOBjgIgMi3KJplu84s+zmXYWXUUGdL1+Ucl+PzCcQw0Yttp
tooAdgB2/4g/Crb7lVHCYcz1h7o0tKTNuyncaEIKn+ZnTFo6dAAAAZG9/KZ1AAAE
AwBHMEUCIQCsBsU/OytLQxJFyjXveQem/Ye/sGKk8/PyR3pTrgMlDAIgWV4/DksD
Sq+EDkK4M2hT+Nd2/RdGpR77sgDvthSLpo0wDQYJKoZIhvcNAQELBQADggEBABsr
FG1nLmPQdCne3KrGpw+ns7tS8hlCDK11sjIqR/svVvE5uLIuEOmroUXBd99bF5M8
j7CAHt3iYyFVQ7QUeU+sX7W0hk09eq8z7CA2iRpvYPsbbEgnet3Gk5JGgyjF1T1u
p9QYLhHh/7Cu7B0ySijD01ctvPyF8cl0/Wj2sl7cSq8hKCLECGzo/VYSFt+kvQCy
5uxcsgIzayh/WQlBrm1xwK/vdvch8DpWcCHnI7yg0cJdpjMyw/Mi1vLzbClJP6HK
MjL0OYhfS2fJbGeCwhaKCc56aRWwrxfuxj+5Afo+ZgxvujIr6qipvu0YpRTdITec
LjfE+3Tm8m32v7H3HiY=
-----END CERTIFICATE-----
subject=CN=acme-v02.api.letsencrypt.org
issuer=C=US, O=Let's Encrypt, CN=R10
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3308 bytes and written 359 bytes
Verification error: CA certificate key too weak
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 67 (CA certificate key too weak)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 3D88D1A1C7F3E25E71888A0522A58352490F6FE95B2C06A96FC35980575E561F
    Session-ID-ctx: 
    Resumption PSK: 5CADB028F8DB671B49D59ACE5F418E1F37C6FC490779EC11ED236EED54CA69B97AA620615DC565E25B8DD86CDBDE0AED
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - ab dd 5a 47 90 4d 0e 46-f0 b9 99 2d 3d a3 11 d9   ..ZG.M.F...-=...
    0010 - d8 1e 59 7c 42 32 81 76-5b af 35 d4 51 bb d9 07   ..Y|B2.v[.5.Q...

    Start Time: 1730016957
    Timeout   : 7200 (sec)
    Verify return code: 67 (CA certificate key too weak)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 2A6F775B294FE951AA3DFBE74C893E696C5E35ABF41D78EFECD32A9E2EB0FD31
    Session-ID-ctx: 
    Resumption PSK: 8BE52C7D229B5CA341D0B9EA478EA737C2F887CBD29B9783465B7EF6E944CCC3F792BA115DBCDD298123774106E50750
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - af 23 94 fe 5d f7 cf 59-25 36 1a be af c9 17 2b   .#..]..Y%6.....+
    0010 - a0 ac c1 3d 75 4a 60 97-4f 71 4e 13 13 89 f4 f7   ...=uJ`.OqN.....

    Start Time: 1730016957
    Timeout   : 7200 (sec)
    Verify return code: 67 (CA certificate key too weak)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
curl --insecure -vvI https://acme-v02.api.letsencrypt.org

it doens't look like curl is using using openssl
which curl ?

2 Likes

It's just the system package:

# which curl
/usr/bin/curl
# curl --version
curl 8.6.0 (x86_64-redhat-linux-gnu) libcurl/8.6.0 OpenSSL/3.2.2 zlib/1.3.1.zlib-ng brotli/1.1.0 libidn2/2.3.7 libpsl/0.21.5 libssh/0.10.6/openssl/zlib nghttp2/1.59.0 OpenLDAP/2.6.7
Release-Date: 2024-01-31
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets

can you try that insecure call of curl too?

curl --insecure -vvI https://acme-v02.api.letsencrypt.org
2 Likes

I do get headers back using --insecure:

$ curl --insecure -vvI https://acme-v02.api.letsencrypt.org
* Host acme-v02.api.letsencrypt.org:443 was resolved.
* IPv6: 2606:4700:60:0:f53d:5624:85c7:3a2c
* IPv4: 172.65.32.248
*   Trying [2606:4700:60:0:f53d:5624:85c7:3a2c]:443...
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=acme-v02.api.letsencrypt.org
*  start date: Sep  4 14:46:39 2024 GMT
*  expire date: Dec  3 14:46:38 2024 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify result: EE certificate key too weak (66), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://acme-v02.api.letsencrypt.org/
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: acme-v02.api.letsencrypt.org]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.6.0]
* [HTTP/2] [1] [accept: */*]
> HEAD / HTTP/2
> Host: acme-v02.api.letsencrypt.org
> User-Agent: curl/8.6.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200 
HTTP/2 200 
< server: nginx
server: nginx
< date: Sun, 27 Oct 2024 08:21:51 GMT
date: Sun, 27 Oct 2024 08:21:51 GMT
< content-type: text/html
content-type: text/html
< content-length: 1540
content-length: 1540
< last-modified: Thu, 20 Jun 2024 17:20:07 GMT
last-modified: Thu, 20 Jun 2024 17:20:07 GMT
< etag: "667464c7-604"
etag: "667464c7-604"
< x-frame-options: DENY
x-frame-options: DENY
< strict-transport-security: max-age=604800
strict-transport-security: max-age=604800

< 
* Connection #0 to host acme-v02.api.letsencrypt.org left intact

looks like openssl seclevel is above 3 and not like LE's rsa 2048 (as this is 112bits)

you can change it in openssl config, but not sure it this is you consciously configed for:

2 Likes

Hunting further, it seems that the system crypto policies are set higher than LetsEncrypt support.

# update-crypto-policies --show
FUTURE

I managed to fix this setting the policy back to DEFAULT via:

# update-crypto-policies --set DEFAULT

At this point, running certbot renew works correctly.

Something must have changed in the crypto policy to exclude the certs that LetsEncrypt use on their systems.

The down side is if you want to use a higher policy than LetsEncrypt supports, then you'll have to configure each application manually and not rely on the system cipher set.

4 Likes

Note that this is not a Let's Encrypt support issue per se, but simply that the FUTURE policy requires "RSA keys size: >= 3072" where the intermediate certs are 2048 bits.

I'm not sure why OpenSSL didn't simply accept the ECDSA cert though, as the intermediate certs have ECDSA keys of 384 bits? The FUTURE policy says "Curves: all prime >= 255 bits (including Bernstein curves)", but also "TLS Ciphers: >= 256-bit key (…)". Is that latter bit size the "symmetric equivalent" perhaps?

3 Likes

I didn't think the Let's Encrypt API servers used ECDSA (yet).

2 Likes

:scream:

Why not? :thinking:

I don't know why; I just only see the RSA intermediates in its history which I would think indicates they're only using RSA leafs too. I'm guessing it's because RSA is still the most compatible and they wouldn't want to change it (even to using dual RSA/ECDSA certs) without warning, and changing may be more of a hassle than it's worth.

3 Likes

There's no deeper reason beyond "Certbot defaulted to RSA and we haven't changed anything".

4 Likes

At the risk of speaking out of turn here, given that the FUTURE crypto policies required better than the RSA that is already enabled, would it be a good idea to start working towards a dual RSA / ECDSA certs on these systems to further future-proof them?

ie get ahead of the issue before it becomes more pressing over time.

Frankly I don’t think disallowing 2048 bit RSA on the internet today is feasible, but we do have a ticket in the backlog to enable ECDSA and clean up a bit of cruft in our accepted cipher suites. It’s not at the top of the backlog and may not get handled for some months yet, though.

4 Likes

You might be right. In fact, you probably are right.

That's the whole point of the FUTURE cipher suite policy - is that its future looking.

At least having something in process now will allow time for this to filter through without being in a panic - which is a much nicer and lower stress option :slight_smile:

I don't think anyone is going to care that its a 6-12 month implementation timeline - as even that would probably be just fine for everyday users / adoptions.

Most of my tests are running the FUTURE cipher set and TLS1.3 only - which again, would cause major issues for a lot of everyday users at this point.