Where can I download the trusted root CA certificates for Let's Encrypt

Server
1

Where can I download the trusted root CA certificates for Let’s Encrypt?

sudo openssl s_client -connect helloworld.letsencrypt.org:443 -showcerts
Start Time: 1493743196
Timeout   : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)

which, accordingly to this page http://movingpackets.net/2015/03/16/five-essential-openssl-troubleshooting-commands/ :

 error:num=20:unable to get local issuer certificate
 This can be fixed by adding the -CAfile option pointing to a file containing all the trusted root certificates   but where to get those?

Note that I would really like to download those files and not install it via packages – I need them to debug further problems I will have with the certificates of the application I am writing.

1 Like
2

hi @ribamar-santarosa

The chain of trust is described here: https://letsencrypt.org/certificates/

There is a link to both Root Certificates and Intermediates that LetsEncrypt uses.

If you are looking at broader CACert Bundle for example for wide range troubleshooting I suggest you review the Mozilla CA Bundle

https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/

Note that for the chain of trust to work you should have both the roots and the intermediates. The Server should serve up the intermediates.

A list of all the included roots can be found here: https://mozillacaprogram.secure.force.com/CA/IncludedCACertificateReport

Andrei

1 Like
3

I think you might have been confused about what you were getting there because you did not use SNI (Server Name Indication). Therefore, the certs that you got from that server are totally different from those that apply to helloworld.letsencrypt.org and that users would see when visiting https://helloworld.letsencrypt.org/.

To remedy this, you should probably add -servername helloworld.letsencrypt.org to your s_client command line. This makes s_client behave closer to the way that a browser would for the TLS negotiation. You will then have a very different set of certs.

As @ahaw021 said, you can download certs from Chain of Trust - Let's Encrypt but most people should not need to do this for most purposes, because their OS or browser CA bundle will typically already include IdenTrust's DST X3 root, which is the root that we customarily chain to for certificates that are being issued today.

4

thanks @ahaw021 and @schoen.
With the output of openssl I could get that I was interested in the X3 certificate:

openssl s_client -connect helloworld.letsencrypt.org:443 -showcerts    
# ...
# issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
#  ...
# Verify return code: 20 (unable to get local issuer certificate)

Then I downloaded the self signed one:

wget "https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt" --output-document=letsencryptauthorityx3.pem

And then I could verify the helloworld certificate:

openssl s_client -connect helloworld.letsencrypt.org:443 -showcerts   -CAfile letsencryptauthorityx3.pem 
# Verify return code: 0 (ok)

I also tried with the X-signed certificate:

openssl s_client -connect helloworld.letsencrypt.org:443 -showcerts   -CAfile  lets-encrypt-x3-cross-signed.pem 
#   Verify return code: 0 (ok)

Then I tried all the commands with my application’s host (that I am calling here my.example.com) instead of helloworld.letsencrypt.org, and, as I expected, they always returned the same return codes for the same test caes, because they’re signed with the same certificates.

However there is still something quite weird: With wget, I can’t download from my host:

wget https://my.example.com/   --output-document=- 
ERROR: The certificate of ‘my.example.com.com’ is not trusted.
ERROR: The certificate of ‘my.example.com’ hasn't got a known issuer.

That is “solved” if I add the certificate file:

wget https://my.example.com/   --output-document=-  --ca-certificate=letsencryptauthorityx3.pem
# Welcome to my application

But, to download from helloworld.letsencrypt.org, I don’t need a certificate:

wget https://helloworld.letsencrypt.org --output-document=- 
# ... 
# /body>
# </html>

The same behaviour is observed if I use curl instead of wget.

now the question is: why do I need to add --ca-certificate for wget to work with my certificate but not to work with helloworld.letsencrypt.org? Aren’t they signed in the same chain of trust?

1 Like
5

@ribamar-santarosa

Review the chain of trust carefully. There are roots and intermediates. Your original question was about root certificates but intermediate certificates also play an important part.

The main difference most likely is that you are not serving up an intermediate with your web server configuration.

Specifying the --ca-certificate=letsencryptauthorityx3.pem solves this issue as WGET knows about the intermediate

For your own learning - don’t jump around between tools as well

If you are going to use openssl - do it for both the letsencrypt reference site and your site (as this gives you a baseline) which you can compare.

A good list of commands can be found here: https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html

Using OpenSSL for one function and then jumping to wget tests creates confusion.

The ways I would have run the tests

a) run against your site, letsencrypt reference site and one other site
b) run SNI and non SNI versions
c) compare result sets and see what is different.

Andrei

6

a couple of other notes

A) If you are interested in another site to test feel free to use https://tls-sni.firecube.xyz/
B) CURL keeps a PEM version of the Mozilla CA bundle here: https://curl.haxx.se/docs/caextract.html
C) When Testing with OpenSSL I prefer to specify the CA bundle in B manually so I am not relying on Operating System to have the bunde available. Example syntax is below.

The server I have configured requires SNI (this is a good thing to test) so if you do not specify a servername no certificate will be returned

With SNI

image.png849×297 13.3 KB

Without SNI

image.png708×354 8.85 KB

Andrei

7

@ahaw021

Ok, now I got it:

openssl s_client -connect helloworld.letsencrypt.org:443 -showcerts    | grep "BEGIN CERTIFICATE"
# -----BEGIN CERTIFICATE-----
# -----BEGIN CERTIFICATE-----
# -----BEGIN CERTIFICATE-----  

openssl s_client -connect my.example.com:443 -showcerts    | grep "BEGIN CERTIFICATE"
# -----BEGIN CERTIFICATE-----

So it was actually true that my application wasn’t serving all the certificates needed. I was somehow confused by the name /etc/letsencrypt/live/scatologies.com/fullchain.pem, I was thinking it would contain all the certificate needed for verifying the host, but that’s not true.

For ruby developers, it is basically:

   server = TCPServer.new nil,  listening_port
    sslContext = OpenSSL::SSL::SSLContext.new
    sslContext.cert = begin OpenSSL::X509::Certificate.new File.open("/etc/letsencrypt/live/my.example.com/fullchain.pem") rescue nil end
    sslContext.key = begin OpenSSL::PKey::RSA.new File.open("/etc/letsencrypt/live/my.example.com/privkey.pem") rescue nil end
    sslContext.extra_chain_cert = ["/etc/letsencrypt/live/my.example.com/chain.pem"].map {|extra_cert_pem_file| begin  OpenSSL::X509::Certificate.new  File.open(extra_cert_pem_file) rescue nil end }
    # note: I didn't test this code after I made some substitutions
   sslServer = OpenSSL::SSL::SSLServer.new(server, sslContext)
   loop do
          connection = sslServer.accept
          Thread.new {
             # here you can connection.gets to receive data already plain until it returns nil, connection.puts string to send string using https
          }
   end

After setting the missing sslContext.extra_chain_cert, I could ensure that both certificates were served:

openssl s_client -connect my.example.com:443 -showcerts    | grep "BEGIN CERTIFICATE"
# -----BEGIN CERTIFICATE-----
# -----BEGIN CERTIFICATE-----

And wget download data without complaining:

wget https://my.example.com/   --output-document=-  --ca-certificate=letsencryptauthorityx3.pem
# Welcome to my application

I still don’t know how to set/force SNI with ruby/if it is even supported by http://ruby-doc.org/stdlib-2.1.2/libdoc/openssl/rdoc/OpenSSL/SSL/SSLContext.html .

Andrei: Many thanks for the quick and involved answers!

Ribamar

8

Hi @ribamar-santarosa

Glad to know it worked

So it was actually true that my application wasn't serving all the certificates needed. I was somehow confused by the name /etc/letsencrypt/live/scatologies.com/fullchain.pem, I was thinking it would contain all the certificate needed for verifying the host, but that's not true.

FullChain.pem is exactly that a combination of the domain certificate + intermediate, however depending on your system it may require (and I think ruby does) you to separate these out. There is not hard and fast rule about this so it's one of those things that needs to be figured out

Andrei

9

It does! As @ahaw021 deduced, your SSLContext library is apparently unwilling to load a multiple-certificate chain from a single file. But they're both present within the file. :slight_smile:

I'm glad that your service is now successfully serving the complete chain.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.