Questions re: Beginning Issuance from R3

@ritturoy as @petercooperjr suggests, your internal service is not serving the full certificate chain. I'd go as far as to say this is 99% likely to be the problem, so fix it on the internal service. You could technically add the R3 intermediate into the trust store of whatever app is using the service so that your service doesn't need to serve the full chain, but that just moves the problem around.

4 Likes

Thank you folks! Like you all mentioned, the problem was with our internal service. It did not serve the new intermediate cert. We have a job that automatically renews the certs every three months. Somehow it did not get the new intermediate. We had to manually download it.

4 Likes

You probably have a client that hardcoded the intermediate certificate. You should ensure your client is able to adapt to changing intermediates, as this will likely break again in the future.

5 Likes

I concur:

You need to remove anything that requires "manual" interaction from your renewal process.

3 Likes

My ssl is not working on Android.

Issuer R3

TLS Certificate is not trusted

Now my app is useless. Thank you guys for breaking things that were working the day before.

Help!!! I want the old issuer back, this new one is not trusted and broke my app. I'm lost.

1 Like

That's not a problem caused by Let's Encrypt, as the intermediate certificate could have changed any time due to a major incident. It's bad practice to hardcode an intermediate certificate.

That's not possible.

That's bad design on the app designers part.

As stated in the thread in the API announcement section:

This change should be a non-event for you and your site's users.

Emphesis on "should be".

If you have adhered to the Let's Encrypt recommendation of renewing 30 days before the end of the lifetime of the certificate, you should be able to put back the previous certificate and you'll have 30 days to fix your app until the certificate expires.

5 Likes

Problem solved.

Replaced .ca file with the new pem file in server and it is now working.

The problem was virtualmin, they didn't update their software to work with the new issuer and broke my app with "Untrusted" incomplete certificate.

Sorry. Everything is good now.
Thank you.

3 Likes

One of the virtualmin developers has told me he will update the Virtualmin certificate software to not pin the intermediate to remedy this kind of issues. Currently, the R3 intermediate is hard-pinned, they updated Virtualmin 12 days ago (see the linked Github commit). However, there hasn't been a new Virtualmin release since. I'm hopeful this hard-pinning is a thing of the past in the future, as promised.

4 Likes

I was hours struggling and thinking my server broke and trying to figure it out how or when, or that I was hacked or something, that's why I came in a rant when I though it was your fault after testing the new certificate. I'm truly sorry.

Thank you for the information regarding Virtualmin and the update. You are son kind.

We are now back in business!!!

Have a nice day!

4 Likes

Hi folks! I just ran into an issue which I'd like to share here:

I am using Let's Encrypt certs for my personal Dovecot IMAP service. I did not face any issue in Apple Mail on iOS 14.2, nor using Thunderbird on MacOSX Big Sur.
But I ran into an issue when I launched Thunderbird on my Windows 10 PC; which is a rare event, since that box is usually only used for gaming, nowadays. I did not get any Error message or anything; the sync just ended and no new mail was fetched.
I then tried it with K-9 Mail app on my old Android device: Same issue: No error but no new mail, either.

I then jumped into the journald of the Mailserver and found:

Dez 21 19:32:56 zzzzz.zen-net.de dovecot[3331742]: imap-login: Disconnected 
(no auth attempts in 0 secs): user=<>, rip=xxx.xxx.xxx.xxx, lip=yyy.yyy.yyy.yyy,
TLS handshaking: SSL_accept() failed: error:14094416:SSL routines:ssl3_read_bytes:sslv3
alert certificate unknown: SSL alert number 46, session=<D1nY...

I also tried to check the cert using OpenSSL on the server's shell:

~ # openssl s_client -connect mail.marc-richter.info:993 -quiet | echo
depth=0 CN = mail.marc-richter.info
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = mail.marc-richter.info
verify error:num=21:unable to verify the first certificate
verify return:1
~ #

Then I found your blog entry, pointing to https://crt.sh/?id=3470671161. I downloaded the Certificate and imported it to Windows 10, Android and the Thunderbird certificate store manually. This fixed it for me.

I described the details in a blog entry here.

I do not know if this is of interest for you or if it was just me doing anything wrong, but I thought I should let you know.

BR,
Marc

1 Like

Hi Marc, and welcome to the forum!

Unfortunately, the solution you found is not the best way to solve your problem. You've added R3, an intermediate certificate which is subject to change at any time without warning, to your root trust stores. This certainly works, but it isn't good practice and (as you noticed) requires changes from every client (your thunderbird, your android, your friend's android, etc...).

Instead, you should fix the mail server to provide the correct information. OpenSSL shows that your server is only providing one certificate (your end-entity cert, issued from R3), and not providing an intermediate at all:

$ openssl s_client -connect mail.marc-richter.info:993
CONNECTED(00000003)
depth=0 CN = mail.marc-richter.info
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = mail.marc-richter.info
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:CN = mail.marc-richter.info
   i:C = US, O = Let's Encrypt, CN = R3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFMTCCBBmgAwIBAgISA6TZLXHIlse5fzvavMy554IVMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMDEyMTEwMDA1NDJaFw0yMTAzMTEwMDA1NDJaMCExHzAdBgNVBAMT
Fm1haWwubWFyYy1yaWNodGVyLmluZm8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCtBMJa5KsV00ZdDb5oUCx08J9UOxVe92+CK2t6XJqbtTdMxjtQ8B98
Z0aSKAnkiksHr+aAqT5DjnmhgZUl7edIJl81IiaUMY6pO3re+LgB0JLCUADjC3+n
jJQUQSzD4Z0z86/N4agqmnvxeEmj9aecZJFkKL3L50A6+yeB0+rTrzWVOHNAVV3F
2Ib0OOM9/IJUZFwytvvp6Rg/DMbtPGrYWTtxyBTWGxvDYNAyyrqyLE71voZj0n65
fDVx9g76oM5k+K7+kJyUYWDZeHWYurj85EqIKce2aigihgHUq7ZmURNoqLz4l764
Z30RkMSlEQW67BlD1jv83PPfGlC66doFAgMBAAGjggJQMIICTDAOBgNVHQ8BAf8E
BAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQC
MAAwHQYDVR0OBBYEFO1jHRE7HG8BZ7X5U0zfcw2EzwhwMB8GA1UdIwQYMBaAFBQu
sxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYV
aHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5s
ZW5jci5vcmcvMCEGA1UdEQQaMBiCFm1haWwubWFyYy1yaWNodGVyLmluZm8wTAYD
VR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYa
aHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEDBgorBgEEAdZ5AgQCBIH0BIHx
AO8AdgBvU3asMfAxGdiZAKRRFf93FRwR2QLBACkGjbIImjfZEwAAAXZPVQ0qAAAE
AwBHMEUCICekt0PdjwP+c0/BIGSxmN4YE0PW3o+y6+4QjP3foSEzAiEAmCJflLjk
PLoYHt4yN2z/ySGqGJrcSiCAKpnhPIC9u/AAdQD2XJQv0XcwIhRUGAgwlFaO400T
GTO/3wwvIAvMTvFk4wAAAXZPVQ0WAAAEAwBGMEQCIFt5bi4fpMTkufR6zmcd7QZA
zO6rvB22eMJCmii3s75KAiBVWMyGnHEgZHaObYBXwgQh6NRsJ8butFG1PbKg3WPl
nzANBgkqhkiG9w0BAQsFAAOCAQEASfRe78Sjf+xbXBqoY+q+5S3MsNTWP2BMyRyk
swZQJNnDjso6ps6YBaNeczOJr2UJf0pMXZfXXK3qT7h1DX4Myc5zoZmDpTzkprQ0
JUKpbI+jXQf7WDOoiUQ5rhV3Tpw59YE0s8ce/7GYI6dO1eMxfsj3wKNe4bNORQwE
YwctA5cK3z4yQJPLPA4D5G3jzcvkwaixMdz4FHyASDZemnHiULSuhtc37wUvnlks
xRoUurOMthehT7+AgYyevLbrq7ncGDIschhWaJ912kufqKKNRTgODePZ0hbxfsmA
Y7SJn7/LBeJ9o/h76eFjuJNqPBP5KqyVPO2et0nZgp0O6Ak9Xw==
-----END CERTIFICATE-----
subject=CN = mail.marc-richter.info

issuer=C = US, O = Let's Encrypt, CN = R3
<snip>

Because your server is not providing the intermediate, if the client doesn't have the R3 intermediate cached already, it can't build and verify a chain from your certificate to a trusted root.

You fixed this problem by making the intermediate a trusted root directly, but really it should be fixed by having the server provide the intermediate so the client can build a chain to an already-trusted root.

Instead of downloading the R3 certificate and installing it in the root store of all your client devices, download the R3 certificate and append it to the certificate file which your server is configured to provide during handshakes.

6 Likes

Hey Aaron; thanks for the note!

But when the R3 intermediate may change at any given time: What's the appropriate way to keep the chain complete on the server then? I mean: Providing it just once on a central point is already way easier than deploying it everywhere. But: So far, Let's Encrypt root certs were part of most modern chains without anything left to do for users and server admins. Downloading and appending it on any cert certbot might re-issue in the future needs a hack but still is fine. But when this might change from time to time, I do not see a resiliency solution to configure for site administrators.

Can you give me a hint on the idea here?

Thanks a lot!

1 Like

It's a good question. All correct ACME clients should (and certbot in particular does) provide the intermediate by default. For example, when certbot runs, it outputs two different files: cert.pem (which contains your newly-issued end entity certificate) and fullchain.pem (which contains both the new cert and the intermediate). Just make sure that your mail server is configured to use the latter file instead of the former.

6 Likes

Hello All,

Apologies if this question has already been asked and answered.

I ran in to some issues two weeks ago due to the intermediate change to R3 and managed to get the issue sorted and new certificate and chain installed in all systems affected. Problem was with the systems not being aware of R3 intermediate.

My chain is now | R3 | ISRG Root X1. It is similar to the one referred as the "Alternative Chain" here - https://letsencrypt.org/2020/12/21/extending-android-compatibility.html

For the renewal of certificates I use https://github.com/acmesh-official/acme.sh. In my scenario, application requires that I provide certificates file that contains the correct intermediate CA and also the ca.cer to include the appropriate root CA(due to application requirements/limitations).

Since there is a mention of an another upcoming change in January:

  1. Does that mean more changes coming to people using alternate chain ?
  2. Where should I refer to for the Intermediate CA and Root CA certificates as a best practice ? This is so I can script some/all of it for future.

Thank you in advance

2 Likes

Hi @yajith,

The certificates made by the R3 intermediate are still valid and unchanged regardless of which path your clients use to validate them. If you use the existing ISRG X1-signed version of the R3 intermediate in the chain you present to clients, that will continue to be valid for new issuance under R3 indefinitely (until the ISRG X1 root expires or until Let's Encrypt stops issuance under R3).

The announcement today said that the new chain will become the default one served to ACME clients:

Instead, we will be switching to provide this new chain by default in late January or early February.

That means that if your ACME clients followed the recommended default behavior and served the chain suggested by the certificate authority, your intermediate certificate would change at that time. That doesn't mean that your current intermediate certificate would become invalid (it would still be a valid path to the ISRG root!), but just that server software that follows the default would not use the X1 chain.

However, as you are already using an "alternative chain", it should be possible for you to continue using this alternative the way you do now: it will still be considered an alternative, but it will still be valid just as it is now.

You might have seen discussions here on the forum where people discouraged this, describing it as fragile. In particular, intermediate certificates tend to shift over time for various reasons. Let's Encrypt has usually been able to give some advance warning of these shifts and discuss it with the community, but in principle it could shift abruptly without prior notice in response to a disaster or a crisis. In any case, it will probably change at least every few years even in the absence of a crisis. Therefore, people on this forum and other PKI experts have strongly recommended not pinning or hard-coding intermediates either on the server side or the client side.

While Let's Encrypt obviously has no way to require or enforce this (and the people criticizing this practice also understand that not everyone has been given a choice in the matter!), people are mentioning it because they would like to discourage users from getting themselves into a position where their applications will break abruptly in the future due to another change of intermediate. There's unfortunately simply no way that Let's Encrypt can promise not to change the intermediate certificate on any particular timescale, so pinning or hard-coding it is taking a gamble about the timetable on which it will end up changing.

There's detailed documentation and a diagram maintained at

This is current as of the recent change to R3 (which you mentioned), but hasn't yet been updated for the new cross-signature that was announced earlier today (probably because the technical work necessary for that cross-signature hasn't even been completed yet). If you keep an eye on that page, you should find good and up-to-date information about Let's Encrypt's issuance practices and the different chains that may be available or in use on the Internet at any given time.

Just to reiterate, the purpose of that page is not to encourage people to pin intermediates (or to manually download them from there instead of using automated chain-updating features in their ACME client software!). These details are provided in the intention of helping people test and assess software compatibility, or debug validation problems, and not as a promise or prediction that the information there will be stable for any particular period of time.

I would also like to emphasize that I'm not trying to criticize you in any way for trying to deal with the requirements you've been given by other people. I appreciate your questions and I hope this reply will be helpful to you. I've reiterated the warning about pinning intermediates here because I'd like the substance and rationale behind that warning to be as clear as possible to anyone reading this information now or in the future. :slight_smile:

7 Likes

Thank you for taking the time for detailed explanation @schoen It is much appreciated.

I will test and switch to "New Default Chain".

Totally agreed. It is just that specific application suite I'm dealing with requires that the provide the full chain (including the intermediates + Root CA). acme.sh just seem(at least used to) to give me only up to the next up Intermediate CA certificate. I will check more if this is just due to the way I have been using it and not a limitation on the tool itself :slight_smile:. Since this is something I have to work work even going forward regardless of which chain/path is used to validate the certs, I was wondering if there is a better place to point as a source where to grab them. As of now I have grabbed these via the links in https://letsencrypt.org/certificates/ just during the troubleshooting/fixing when things when the issue was experienced.

2 Likes

No, ACME clients in general probably will not provide the chain up to the root certificate. The chain saved by ACME clients is the one provided by the CA, which doesn't include the root, only the intermediate or intermediates.

The argument for doing it this way is that, since root certificates are self-signed, being told about them by an outside (untrusted) party never changes anyone's decision about whether to accept a particular certificate that chains to them.

  • If you don't already trust that root, just receiving a self-signed certificate that claims to be a root is no reason to accept it (as anyone in the world can produce a self-signed certificate claiming to be a root CA at any time).
  • If you already trust that root, then you already know about it and don't have to be told about it.

Another way to phrase this is that roots are qualitatively different from intermediate and leaf certificates in how and why verifiers come to accept them. For a root certificate, a verifier should accept it only because of an out-of-band trust decision such as the deliberations of a browser's or OS vendor's root program. For an intermediate or leaf certificate, a verifier accepts it because it was validly issued by another trusted entity. Thus the root certificate acceptance decision is unaffected by information provided in online interactions, while the intermediate or end-entity acceptance decision is based directly on such information.

This is certainly different if your application isn't trying to use a set of public CAs in an open-ended way the web PKI does. But in that case, it might be simpler and more reliable in some ways not to use a public CA at all and instead to use your own private CA. If you are actively involved in the selection and distribution of roots, it might make more sense to take advantage of that to select and distribute (private) roots that you control rather than roots that someone else controls! For example, if you create a private CA and then tell this application about its root and intermediate, not only will you not have to use an Internet-based validation process like ACME challenges to get new certificates (from your own CA), but you can guarantee that nobody else will unexpectedly change the intermediates, because you control them yourself!

The ability to make unanticipated or periodic changes in the trust chain is a practical operational requirement for publicly-trusted CAs in a public PKI, but it doesn't have to be for your own private PKI.

4 Likes

It might be nice to put together an "Introduction to WebPKI" page in the documentation, describing the kinds of things that @schoen mentions here (about when you might want a private vs. public CA), and how the standard is for servers to send intermediates, and for clients to only have roots in trust stores, and what should go into a trust store if you need to make one yourself, and that kind of thing.

I feel like we repeat this kind of thing a lot on the forums (especially lately with people trying to understand how Android versions do things, as well as why the R3 intermediate switch broke their workflow for whatever reason), and while yet another link buried in the "advanced documentation" section may not get read much on its own, it'd be nice to be able to point people to it when there are questions. (Unless there's some other nice intro already out there on the net somewhere?)

@griffin, maybe if you're planning on continuing to overhaul the documentation pages :slight_smile:, this could be another project on your to-do list? I may end up tackling a draft myself if I get some time, but I don't know how likely I am to be able to get the time for it myself.

4 Likes

If the cross-signed ISRG Root X1 certificate were issued, I would already be on that PR. :grin:

2 Likes

I've certainly been considering it for a while. It kinda became a tentacle off of the belated certbot handbook (that turned-up as many questions as answers).

3 Likes