Curl does not trust LE certs on plain Debian

Tl;Dr:
Tools such as curl don’t trust let’s encrypt certificates on vanilla Debian systems, even though both root certs let’s encrypt uses are included by default
One has to add the intermediate LE cert to make curl trust LE certs

Hey everyone,

I have encountered an issue with Let’s Encrypt’s certs several times in the past and was wondering if the ISRG has plans to deal with that issue or I am just missing something here.

There are several distributions that include the Mozilla root CA store. Debian for example has the ca-certificates package, which includes both of the root certs LE uses: ISRG Root X1 and Identrust DST Root CA X3.

When using wget or curl to download something via HTTPS on Debian from a host with a Let’s Encrypt certificate they both throw an error though, because they don’t trust the certificate. One has to use the --insecure flag with curl, otherwise it just won’t work.

To fix this, one has to manually add the intermediate cert “Let’s Encrypt Authority X3” to the CA store, to make curl and wget trust Let’s Encrypt certificates.

I’ve read somewhere that browsers actually don’t include the intermediate cert either but do some sort of magic to still trust LE certs…
Are there any plans by the ISRG to include the LE intermediate in distributions? Or is this just a “known issue”?

I know that commercial CAs also use intermediates, why do those not have this problem?

Best regards,
Valentin

You should never add the intermediates to your root trust store. The trust anchors are the ISRG and DST roots, and the intermediates are signed/cross-signed by them.

The DST root should already be present in any Debian distribution you are using, and therefore you shouldn’t run into trust issues. The intermediates are cross-signed by the DST root for the precise reason of ensuring compatibility.

I would guess that the hosts you are experiencing issues with are not properly bundling the intermediates along with their certificates.

Are you able to successfully fetch the following from the servers where you experience this issue?

$ curl https://helloworld.letsencrypt.org
1 Like

The issue is that the web servers need to send the end-entity certificate and all necessary intermediates (currently just Let’s Encrypt Authority X3 for Let’s Encrypt certificates).

Browsers usually but do not always work around missing intermediates (by caching them from connections to other websites, or downloading them on demand). Simpler non-browser clients usually don’t.

What ACME client are you using? What sites are affected? What web servers are they using? How are they configured?

As an example, https://incomplete-chain.badssl.com/ is currently using a DigiCert certificate, and you won’t be able to access it with curl either.


Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

Thanks for the quick replies!

Interestingly, my Firefox does not complain at all when I visit incomplete-chain.badssl.com, so I guess it’s doing some arcane caching stuff as you pointed out.

I am using acme-tiny and my Apache should serve the intermediate correctly, at least my config seems good to me:

        SSLCertificateFile /etc/univention/letsencrypt/signed.crt
        SSLCertificateKeyFile /etc/univention/letsencrypt/domain.key
        SSLCACertificateFile /etc/univention/ssl/ucsCA/CAcert.pem   
        SSLCertificateChainFile /etc/univention/letsencrypt/intermediate.pem

I’ve had this issue on internal systems so I set up a test system on AWS for you:

My domain is: le-test.ddns.net

My web server is (include version): Apache 2.4.10

The operating system my web server runs on is (include version): UCS 4.2-3 (based on Debian 8)

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

SSLCACertificateFile doesn’t do anything outside of the context of client authentication, but otherwise it seems fine.

On which version of Debian do you have issues with curl? Do you have ca-certificates installed on the host that is running curl?

I think the most straightforward way to chase this down would be to run this against your internal host and show the full output:

openssl s_client -connect le-test.ddns.net:443 -servername le-test.ddns.net -showcerts

I’m using Univention Corporate Server 4.2-3 which is based on Debian 8. ca-certificates is installed. When I do a openssl verify with either of the root CAs Let’s Encrypt uses I get the following error:

openssl verify -CAfile /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt /etc/univention/letsencrypt/signed.crt
/etc/univention/letsencrypt/signed.crt: CN = le-test.ddns.net
error 20 at 0 depth lookup:unable to get local issuer certificate
openssl verify -CAfile /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt /etc/univention/letsencrypt/signed.crt
/etc/univention/letsencrypt/signed.crt: CN = le-test.ddns.net
error 20 at 0 depth lookup:unable to get local issuer certificate

But when I use the intermediate it works:

openssl verify -CAfile /etc/univention/letsencrypt/intermediate.pem /etc/univention/letsencrypt/signed.crt
/etc/univention/letsencrypt/signed.crt: OK

When I open my certificate with openssl I see that the certificate is issued by:

Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3

So openssl verify is entirely right to complain that it does not find the issuer cert, because it is not installed.
I have the root CAs on the system as you can see in my last post but the root is not the issuer.

Nowadays, root certificates never issue end-entity (leaf) certificates directly. End-entity certificates are always issued by an intermediate. So, you will see the same phenomenon with any CA's certificates if you export them in your web browser.

Comodo for example gives admins a cabundle to use as certchain.
I created a cabundle using DST_Root_CA_X3 and Let’s Encrypt Authority X3 and verifying against that bundle works perfectly.

Why doesn’t Let’s Encrypt give admins a cabundle by default? Or is that just different for every client

The ACME protocol does provide the intermediate certificate to the client, and clients can decide what to do with it. For example, Certbot saves it in chain.pem and fullchain.pem. Serving the root certificate over the wire is unnecessary and not recommended.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.