Help, let'sencrypt does not work for Mail

Hi folks,
I would like to use my certificates for mail. The FAQ states that

Let’s Encrypt certificates are standard Domain Validation certificates, so you can use them for any server that uses a domain name, like web servers, mail servers, FTP servers, and many more.

But then, in contrast, the certificate itself states:

        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Extended Key Usage: 
            TLS Web Server Authentication, TLS Web Client Authentication

This says clearly that it is only for Web Servers, not for mail servers. (I did look for the correct category for mail servers in the IANA RFCs, but there isn't even one mentioned. Not sure where to look further for that.)

Then, I thought, what the heck, just give it a try, and installed it on the mailserver nevertheless. And indeed, it does not work for mailservers. (The same cert runs on the Web server just fine.)

Apr 28 23:26:19 <mail.info> gate sm-mta[71345]: STARTTLS=client, relay=intra.daemon.contact., version=TLSv1.3, verify=FAIL

After a lot of hassle, trying out my own CA, shooting myself in the foot a couple of times, etc.etc., I found somebody to ask about how to debug sendmail. And so I got further information:

2022-04-28T23:18:46.704257+02:00 <mail.info> intra.daemon.contact sm-mta[69293] STARTTLS: TLS cert verify: depth=3 /O=Digital Signature Trust Co./CN=DST Root CA X3, state=0, reason=certificate has expired
2022-04-28T23:18:46.705103+02:00 <mail.info> intra.daemon.contact sm-mta[69293] STARTTLS=server, get_verify: 10 get_peer: 0x801c3af00
2022-04-28T23:18:46.705122+02:00 <mail.info> intra.daemon.contact sm-mta[69293] STARTTLS=server, relay=gate.intra.daemon.contact [xx.xx.xx.xx], version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
2022-04-28T23:18:46.705131+02:00 <mail.info> intra.daemon.contact sm-mta[69293] STARTTLS=server, cert-subject=/CN=moon.daemon.contact, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=certificate has expired

So the certificate is expired for mail!

Let's look closer at this. That is what's in the fullchain I obtained:

    Issuer: C = US, O = Let's Encrypt, CN = R3
    Validity
        Not Before: Mar 22 05:04:24 2022 GMT
        Not After : Jun 20 05:04:23 2022 GMT
    Subject: CN = moon.daemon.contact

    Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
    Validity
        Not Before: Sep  4 00:00:00 2020 GMT
        Not After : Sep 15 16:00:00 2025 GMT
    Subject: C = US, O = Let's Encrypt, CN = R3

    Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
    Validity
        Not Before: Jan 20 19:14:03 2021 GMT
        Not After : Sep 30 18:14:03 2024 GMT
    Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

None of these appears to be expired. Let's check the storage in /etc/ssh/certs:

    Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
    Validity
        Not Before: Jun  4 11:04:38 2015 GMT
        Not After : Jun  4 11:04:38 2035 GMT
    Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

    Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
    Validity
        Not Before: Sep 30 21:12:19 2000 GMT
        Not After : Sep 30 14:01:15 2021 GMT
    Subject: O = Digital Signature Trust Co., CN = DST Root CA X3

Both of them are there, and one is indeed outdated.
(BTW, how can I figure out which release of a cert was used to sign another?)

I was wondering what is going on - checked the web and found this article:

Now this is really a bad joke: "older devices and browsers".
This device is:

$ uname -a
FreeBSD 13.1-RC2 FreeBSD 13.1-RC2 n250132-697a6a902ab[697a6a902ab=c2e15a08fd9+28]

This is so new, it is not even released yet.

So, now, would somebody please be so kind and explain to me what is going on, and, most importantly, how this would be supposed to work?

The id-kp-serverAuth EKU is suitable for any TLS server.

Web as in "World Wide Web", not Web as in "Website". It is suitable for mail servers. There are an enormous number of mail (IMAPS/POP3S/SMTPS) servers out there secured by Let's Encrypt certificates.

Certificates are signed by a private key, not by another certificate. Clients should be able to build a trust path to the self-signed, unexpired "ISRG Root X1", even if the cross-signed "ISRG Root X1" certificate chains to the expired "DST Root CA X3".

There is more information about that in this post: OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates. Modern versions of OpenSSL/GnuTLS/NSS/SecureChannel/whatever shouldn't encounter issues.

Do you know what TLS library+version your deployment of sendmail is built with?

4 Likes

Okay, understood.

Ah, there was something... the key may or may not be done away with when the cert needs renewal... Thank you for reminding me. :slight_smile:

Okay, this translates to: the bug is in sendmail, it should search exhaustively instead returning with error.
This is what I needed to know.

That is a good question. It might be the system's openssl which would be:

# openssl version
OpenSSL 1.1.1n-freebsd  15 Mar 2022
# ldd sendmail 
sendmail:
        libutil.so.9 => /lib/libutil.so.9 (0x801126000)
        libssl.so.111 => /usr/lib/libssl.so.111 (0x80113e000)
        libcrypto.so.111 => /lib/libcrypto.so.111 (0x8011d8000)
        libwrap.so.6 => /usr/lib/libwrap.so.6 (0x8014da000)
        libc.so.7 => /lib/libc.so.7 (0x8014e7000)
        libthr.so.3 => /lib/libthr.so.3 (0x8018ff000)
# strings /usr/lib/libssl.so.111 | grep freebsd
OpenSSL 1.1.1n-freebsd  15 Mar 2022

That looks like it is the system's openssl.

and these logs came from the sendmail which is built against OpenSSL 1.1.1n?

And are you 1000000% sure that sendmail is currently serving the renewed certificate, and not the one which expired on 2022-04-18? Sorry, it's worth double checking what's actively being served and not just what's on the filesystem.

4 Likes

I have no sendmail installed, neither an SSL library. So both are the OS integrated ones.
And everything gets installed through a deploy chain here - there is no library chaos here unless it's created upstream.

Near to that. I took only one of Your certs, the "moon", to bring it into this play. And I fetched it from the live webserver, where it was installed on march 22.
The certbot is running on the kerberos server, that's quite a different place, and only there are the old ones.

You're absolutely right. And indeed I found a strangeness, a sendmailer that reported mysterious things because it hadn't been restarted as intended.
So I did a good housecleen, and started from fresh - and the strangeness is still here, and is reproducible.

There are two nodes ("gate" and "webs"), and currently both have Your "moon" cert installed:

-rw-r--r--  1 root  wheel  5973 Apr 28 23:17 cacert.pem
-rw-r--r--  1 root  wheel  5973 Apr 28 23:17 host.cert
-rw-------  1 root  wheel  3272 Apr 28 23:17 host.key

This is the fullchain twice; I tried different things before but it didn't seem to improve things.

All sendmailers are restarted.

Sending a mail.

On the client "gate":

Apr 29 03:16:41 <mail.info> gate sm-mta[29139]: STARTTLS: TLS cert verify: depth=3 /O=Digital Signature Trust Co./CN=DST Root CA X3, state=0, reason=certificate has expired
Apr 29 03:16:41 <mail.info> gate sm-mta[29139]: STARTTLS=client, relay=webs-e.intra.daemon.contact., version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Apr 29 03:16:41 <mail.info> gate sm-mta[29139]: STARTTLS=client, cert-subject=/CN=moon.daemon.contact, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=certificate has expired

On the server "webs":

2022-04-29T03:16:41.989891+02:00 <mail.info> webs.intra.daemon.contact sm-mta[29140] STARTTLS: TLS cert verify: depth=3 /O=Digital Signature Trust Co./CN=DST Root CA X3, state=0, reason=certificate has expired
2022-04-29T03:16:41.992999+02:00 <mail.info> webs.intra.daemon.contact sm-mta[29140] STARTTLS=server, relay=gate-e.intra.daemon.contact [**.**.**.**], version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
2022-04-29T03:16:41.993019+02:00 <mail.info> webs.intra.daemon.contact sm-mta[29140] STARTTLS=server, cert-subject=/CN=moon.daemon.contact, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=certificate has expired

The mail gets routed back.

On the client "webs":

2022-04-29T03:17:34.299284+02:00 <mail.info> webs.intra.daemon.contact sm-mta[29486] STARTTLS: TLS cert verify: depth=3 /O=Digital Signature Trust Co./CN=DST Root CA X3, state=0, reason=certificate has expired
2022-04-29T03:17:34.320753+02:00 <mail.info> webs.intra.daemon.contact sm-mta[29486] STARTTLS=client, relay=gate-e.intra.daemon.contact., version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
2022-04-29T03:17:34.320781+02:00 <mail.info> webs.intra.daemon.contact sm-mta[29486] STARTTLS=client, cert-subject=/CN=moon.daemon.contact, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=certificate has expired

On the server "gate":

Apr 29 03:17:34 <mail.info> gate sm-mta[29487]: STARTTLS: TLS cert verify: depth=3 /O=Digital Signature Trust Co./CN=DST Root CA X3, state=0, reason=certificate has expired
Apr 29 03:17:34 <mail.info> gate sm-mta[29487]: STARTTLS=server, relay=webs-e.intra.daemon.contact [192.168.98.11], version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Apr 29 03:17:34 <mail.info> gate sm-mta[29487]: STARTTLS=server, cert-subject=/CN=moon.daemon.contact, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=certificate has expired

Now for something different - I install one of my own certs in the "gate" node (that one is only for server-usage, so it will fail to verify on the server side):

-rw-r--r--  1 root  wheel  2204 Apr 29 03:31 cacert.pem
-rw-r--r--  1 root  wheel  7503 Apr 29 03:31 host.cert
-rw-------  1 root  wheel  3272 Apr 29 03:31 host.key

And restart the sendmailer on "gate". (The "webs" node is left completely untouched!)

Mail is sent. And now there is this:

On the client "gate":

Apr 29 03:35:43 <mail.info> gate sm-mta[36828]: STARTTLS: TLS cert verify: depth=0 /CN=moon.daemon.contact, state=1, reason=ok
Apr 29 03:35:43 <mail.info> gate sm-mta[36828]: STARTTLS=client, relay=webs-e.intra.daemon.contact., version=TLSv1.3, verify=OK, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Apr 29 03:35:43 <mail.info> gate sm-mta[36828]: STARTTLS=client, cert-subject=/CN=moon.daemon.contact, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=ok

On the server "webs":

2022-04-29T03:35:43.404305+02:00 <mail.info> webs.intra.daemon.contact sm-mta[36829] STARTTLS: TLS cert verify: depth=0 /O=Phase23/OU=mailtest-7/CN=loetzinn, state=0, reason=unsupported certificate purpose
2022-04-29T03:35:43.405843+02:00 <mail.info> webs.intra.daemon.contact sm-mta[36829] STARTTLS=server, relay=gate-e.intra.daemon.contact [**.**.**.**], version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
2022-04-29T03:35:43.405850+02:00 <mail.info> webs.intra.daemon.contact sm-mta[36829] STARTTLS=server, cert-subject=/O=Phase23/OU=mailtest-7/CN=loetzinn, cert-issuer=/O=Phase23/OU=Phase23+20CA/emailAddress=*****************************/CN=Phase23+20Root+20CA+20#2, verifymsg=unsupported certificate purpose

And the mail is routed back:

On the client "webs":

2022-04-29T03:38:34.354201+02:00 <mail.info> webs.intra.daemon.contact sm-mta[37543] STARTTLS: TLS cert verify: depth=0 /O=Phase23/OU=mailtest-7/CN=loetzinn, state=1, reason=ok
2022-04-29T03:38:34.418918+02:00 <mail.info> webs.intra.daemon.contact sm-mta[37543] STARTTLS=client, relay=gate-e.intra.daemon.contact., version=TLSv1.3, verify=OK, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
2022-04-29T03:38:34.418938+02:00 <mail.info> webs.intra.daemon.contact sm-mta[37543] STARTTLS=client, cert-subject=/O=Phase23/OU=mailtest-7/CN=loetzinn, cert-issuer=/O=Phase23/OU=Phase23+20CA/emailAddress=*****************************/CN=Phase23+20Root+20CA+20#2, verifymsg=ok

On the server "gate":

Apr 29 03:35:43 <mail.info> gate sm-mta[36828]: STARTTLS: TLS cert verify: depth=0 /CN=moon.daemon.contact, state=1, reason=ok
Apr 29 03:35:43 <mail.info> gate sm-mta[36828]: STARTTLS=client, relay=webs-e.intra.daemon.contact., version=TLSv1.3, verify=OK, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Apr 29 03:35:43 <mail.info> gate sm-mta[36828]: STARTTLS=client, cert-subject=/CN=moon.daemon.contact, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=ok

I find this strange. So I swapped both certs. And the behaviour is the same, i.e. things appear to work.

I refrain from further comments because I would find this hard to believe if somebody tells me.

Uhm...

% openssl s_client -connect moon.daemon.contact:25 -starttls smtp
F8A45BDA7B000000:error:8000006E:system library:BIO_connect:Connection timed out:crypto/bio/bio_sock2.c:125:calling connect()
F8A45BDA7B000000:error:10000067:BIO routines:BIO_connect:connect error:crypto/bio/bio_sock2.c:127:
1 Like

Ahh, @9peppe, test+evaluation is usually not done in the production environment. Would You think me a good sysop when I did these experiments on the live mailserver?

I fact I am doing them on the live environment :wink: - but moon is not the name it actually answers to for smtp. (You could have looked for the mx of daemon.contact.)

Let me help You out:

# openssl s_client -connect webs-e.intra.daemon.contact:25 -starttls smtp
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = moon.daemon.contact
verify return:1
---
Certificate chain
 0 s:CN = moon.daemon.contact
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
 3 s:O = Digital Signature Trust Co., CN = DST Root CA X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

But we knew that already.

What's more interesting is the -rarely read- documentation of sendmail:

When acting as a serv er, sendmail requires X.509 certificates to support ST ARTTLS: one
as certificate for the server (ServerCertFile and corresponding pri vate Serv erKeyFile) at least
one root CA (CACertFile), i.e., a certificate that is used to sign other certificates, and a path to a
directory which contains (zero or more) other CAs (CA CertPath). The file specified via CA C-
ertFile can contain se veral certificates of CAs. The DNs of these certificates are sent to the
client during the TLS handshake (as part of the CertificateRequest) as the list of acceptable CAs.
However, do not list too many root CAs in that file, otherwise the TLS handshake may fail; e.g.,
error:14094417:SSL routines:SSL3_READ_BYTES:
sslv3 alert illegal parameter:s3_pkt.c:964:SSL alert number 47
You should probably put only the CA cert into that file that signed your own cert(s), or at least
only those you trust.

What I learned, is, that of these three files

cacert.pem
host.cert
host.key

the key is the key, the cert is the cert, and the cacert.pem is the chain, i.e. the collection of intermediate certificates that are needed for the adversary to get to one of their known root ca certs, which obviousely must be sent to the adversary during handshake.

But here the doc talks about something different: it appears to say that the cacert.pem contains a "list of acceptable CAs".
The outdated DST ROOT should proably not be in such a list.

This is expected and fine.

The mess with DST Root X3 would ignore ISRG Root X1 as a trust anchor and evaluate DST Root X3, failing to do so because it's expired. It was the behavior for OpenSSL 1.0.1 and 1.0.2.

What's the openssl version of the client that's failing?

1 Like

OpenSSL 1.1.1n-freebsd 15 Mar 2022

But read what I added above! (I hit shift-enter and hoped for a newline, instead it sent the posting prematurely)

The nomenclature is confusing and I cannot find the documentation easily, but yes. It looks like that's supposed to be chain.pem

That's a choice you can make. Just run (if you use certbot, check docs otherwise):

certbot --preferred-chain "ISRG Root X1" [the rest of your command]

and certbot will obtain a chain.pem without any reliance on DST Root X3 (it will do so if you renew, force-renew, or issue a new certificate)

1 Like

Let me elaborate: read this page before you switch chains,

1 Like

I agree wholeheartedly. :slight_smile:
(The documentation is inside the source distribution in a *.me format that would need eqn and nroff to get readable. This is Berkeley as it was in 1988. :wink: )

Thanks for the confirmation. This is the hint I found from the article @_az kindly linked to, but the precise syntax to use wasn't described there. This indeed looks more nice than using sed on my own. :wink:

Nevertheless, since sendmail once was the mta of the internet and is still fairly widespread, I would have expected the engineers of Let'sEncrypt to mention that not only openssl 1.0.x and whatever needs this adaption, but also when using sendmail StartTLS.

You're probably the first one encountering this. :smiley:

You have a more fun alternative, you can use an ECDSA certificate issued via ISRG Root X2, and that never uses the DST root :smiley:

(You have to get your acme account allowlisted to do that)

1 Like

I nailed it down now.

You say in

The behavior that causes this incompatibility was fixed over 4 years ago with the release of OpenSSL 1.1.0.

That is wrong. It was not fixed.

Or, more specifically, it was fixed only for the simple case of one-sided authentication (like it happens usually in web servers), but not for the case of mutual authentication (like it commonly happens with smtp).

You can reproduce this with openssl s_client. Do the following:

  1. Have a mailserver running with Let'sEncrypt certificate installed. This certificate can be obtained with --preferred-chain "ISRG Root X1" so it does not contain the offending intermediate.

  2. Get yourself another (or the same) Let'sEncrypt certificate, this time with the complete default fullchain. And get yourself the two root ca certs in question.

It should look like this then:

# ls -l
total 45
drwxr-xr-x  2 root  wheel     6 Apr 29 17:44 CADIR
-rw-r--r--  1 root  wheel  5973 Apr 29 16:08 cacert.pem
-rw-r--r--  1 root  wheel  2224 Apr 29 18:01 host.cert
-rw-------  1 root  wheel  3272 Apr 29 17:50 host.key
# ls -l CADIR/
total 13
lrwxr-xr-x  1 root  wheel    18 Apr 29 17:18 2e5ac55d.0 -> DST_Root_CA_X3.pem
lrwxr-xr-x  1 root  wheel    16 Apr 29 17:18 4042bcee.0 -> ISRG_Root_X1.pem
-rw-r--r--  1 root  wheel  4665 Apr 29 17:14 DST_Root_CA_X3.pem
-rw-r--r--  1 root  wheel  7461 Apr 29 17:14 ISRG_Root_X1.pem
  1. run the s_client in mutual authentication mode:

# openssl s_client -connect flowm.daemon.contact:25 -starttls smtp -CAfile cacert.pem -chainCAfile cacert.pem -key host.key -cert host.cert -CApath CADIR -chainCApath CADIR

This does now nicely verify on the server. But on the client (where the openssl s_client is running) it looks like this:

depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
notAfter=Sep 30 18:14:03 2024 GMT
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
notAfter=Sep 15 16:00:00 2025 GMT
verify return:1
depth=0 CN = moon.daemon.contact
notAfter=Jun 20 05:04:23 2022 GMT
verify return:1
[etc.etc.]

SSL handshake has read 5744 bytes and written 6530 bytes
Verification error: certificate has expired
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 10 (certificate has expired)
---
250 HELP
---
Post-Handshake New Session Ticket arrived:
[etc. etc.]

    Start Time: 1651248347
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
[etc. etc.]

    Start Time: 1651248347
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)
    Extended master secret: no
    Max Early Data: 0
---

I'm not sure if I hit this exact problem, but maybe I did:

Is this this issue? And do you have a solution to this?

Yeah, apparently, it is, and the solution is to request a certificate with --preferred-chain "ISRG Root X1" :wink:

This gets more and more fun. I checked back with a FreeBSD contributor if we shouldn't simply remove that outdated DST X3 caroot from the ca-store that is delivered with FreeBSD - so to resolve the issue for the FreeBSD users without further effort.

But that doesn't work!

Lets go back to our test environment:
Alice is running a mailserver with a Let'sEncrypt cert. (It doesn't matter what she does in detail.)
Bob is running a FreeBSD box and sends a mail to Alice. He has a Let'sEncrypt cert with the long chain including the DST Root CA X3 intermediate. So Bob gets the error "certificate expired".

Now Bob starts to search for the expired cert, finds the caroot as delivered by FreeBSD, it is /etc/ssl/certs/2e5ac55d.0 - and simply deletes it.

What happens - now we get this error:

STARTTLS=client, version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
STARTTLS=client, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=unable to get issuer certificate

I am more and more convinced that we have a solid bug in the openssl. And there are only two ways to deal with it:

  • use the --preferred-chain switch, or
  • get somebody at openssl to fix it.

Those at openssl do already know about the issue, but they try to push blame to "external callbacks" (whatever that means):

The problem happens with a web server too.
Try "www.letsencrypt.org"

We have the CA as trusted:

openssl x509 -text -in /etc/ssl/certs/ISRG_Root_X1.pem | grep CN.=
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

With "DST Root CA X3" in /etc/ssl/certs/ :

openssl s_client -connect www.letsencrypt.org:443 -CAfile /usr/local/etc/letsencrypt/live/example.com/chain.pem -CApath /etc/ssl/certs/ < /dev /null
[...]
Verification error: certificate has expired
[...]
Verify return code: 10 (certificate has expired)
[...]
    Verify return code: 10 (certificate has expired)```
[...]

With "DST Root CA X3" removed from /etc/ssl/certs/ :

openssl s_client -connect www.letsencrypt.org:443 -CAfile /usr/local/etc/letsencrypt/live/example.com/chain.pem  -CApath /etc/ssl/certs/ </dev /null
[...]
Verification error: unable to get issuer certificate
[...]
Verify return code: 2 (unable to get issuer certificate)
[...]
    Verify return code: 2 (unable to get issuer certificate)
[...]