@aarongable I'm a little confused. Is the staging environment currently broken, or working as intended? We need to use it to avoid throttling limits from your prod environment while testing. We are using cert-manager, and by default it is getting the already expired root, though it is unclear if that is a cert-manager bug or a Let's Encrypt bug. What root of trust is supposed to be used for staging? Your link doesn't make it clear. We found "(STAGING) Pretend Pear X1" but it is not self-signed and thus incompatible with certain libraries.
Lovely, looks like this is the right technical solution that I wasn't aware of. Thanks for this wonderful discussion here everyone!
The API Announcement is so confues.
On 18 Feb
(STAGING) Artificial Apricot R3and chain to our new Staging root
(STAGING) Doctored Durian Root CA X3
But on 20 Feb
Addtionally, we noticed our test hierarchy has the wrong AIA CA Issuers URI and CRL Distribution Point URI on the certificate:
(STAGING) Artificial Apricot R3signed by
(STAGING) Doctored Durian Root CA X3. That fix will be done in the next few days too.
Apparently, both mentioning
(STAGING) Doctored Durian Root CA X3, so is the expired root cert will be corrected in the next few days?
Or a more simple question waiting to be answered, is the staging env are able or will be able to get a fake cert with the root cert is not expired?
I can see the issue is now has been fixed,
(STAGING) Doctored Durian Root CA X3 is not the root cert anymore, the new root cert
(STAGING) Pretend Pear X1 is come with a valid date.
So much thanks!
There are two root certs, my friend. (Three if you want to be really technical.)
This was the root for the primary chain of issuance:
(STAGING) Doctored Durian Root CA X3 self-signed
This is now the root for the primary chain of issuance:
(STAGING) Pretend Pear X1 signed by
(STAGING) Doctored Durian Root CA X3 self-signed
This was/is the root for the alternate chain of issuance:
(STAGING) Pretend Pear X1 self-signed
I think a key thing here was that the expired root in the chain is intentional to test out the planned production strategy for handling Android etc. The alternate chain is for those who don't want/need the expired root in their chain, I believe so anyway?
"Doctored Durian Root CA X3 self-signed" is not yet removed from the chain. It is still there. Those who don't want the expired cert should request for an alternate chain that has
(STAGING) Pretend Pear X1 self-signed?
I see that the text in Staging Environment - Let's Encrypt was updated. Thanks for that!
Quoting, for posterity:
The staging environment intermediate certificate ("(STAGING) Artificial Apricot R3") is issued by a root certificate not present in browser/client trust stores. If you wish to modify a test-only client to trust the staging environment for testing purposes you can do so by adding the "(STAGING) Pretend Pear X1" certificate to your testing trust store. Important: Do not add the staging root or intermediate to a trust store that you use for ordinary browsing or other activities, since they are not audited or held to the same standards as our production roots, and so are not safe to use for anything other than testing.
Thanks to the entire Let's Encrypt team for the discussion and support in this thread. The technical solution to simply trust
Pretend Pear X1 instead of
Doctored Durian Root CA X3 is of course super nice and clean. I have used the opportunity to read up on the cross certification concept used by the Let's Encrypt cert chains (for the record: super nicely documented here: Chain of Trust - Let's Encrypt). As far as I am concerned, this 'issue' is resolved to my satisfaction, and it's also fair to say that it never really was a technical issue -- but just a lack of understanding, at least on my side. I feel like I've learned a lot about LE in the process of thinking this here through! Thanks again.
That's correct. The alternate chain should not be changing for awhile to my knowledge. Thus, both the primary and alternate R3 intermediate certificates (used for signing user leaf certificates) will soon be signed by an X1 root certificate. The only difference between the two chains will be that the X1 root certificate for the primary chain will be signed by Root CA X3 while the X1 root certificate for the alternate chain will be self-signed.
Please note that either root X1 certificate can be used to authenticate the R3 intermediate certificate served by either chain. That's because both root X1 certificates use the same private key. The very definition of cross-signing is having multiple certificates for the same subject using the same private key that are signed by different certificates (Root CA X3 and Root X1 in this case).
agree we use staging to not hit rate limits for production, maybe there should be another level of environment for breaking changes? at the moment this has broken our environment builds using terraform/azure as the certificate is not allowed to upload to azure due to the expiration of the root?
@mattduguid, there was a recent thread where other people (including me) proposed something similar to what you suggest here.
This led to this statement from Let's Encrypt staff:
You can post your experience and opinions in the former thread, but following the latter thread, it seems like Let's Encrypt's current intentions are not to guarantee that staging and production will maintain a particular level of equivalence or parallelism with each other such that you can reliably condition a production issuance decision on a successful staging issuance.
we use terraform resource "acme_certificate" to request the certificate from "https://acme-staging-v02.api.letsencrypt.org/directory"
a number of comments advise "Those who don't want the expired cert should request for an alternate chain" is there any further information on how to achieve that?
Assuming this is the terraform resource you're using, it doesn't appear to currently support alternate chain selection. The author would have to add that feature in order for you to use it.
Looks like you could submit an issue with the feature request in their github repo.
Since this change if we attempt to upload a staging cert to azure via their API (terraform or portal UI) we get back the error "Expired certificate is not allowed". We have confirmed with Microsoft that this is because their validation process is still using the chain with the expired root cert "Doctored Durian". There are a number of comments in this forum advising those who dont want the expired cert should "request for an alternative chain" with root cert "Pretend Pear", can anyone advise how this is requested to allow Microsoft to validate against the non expired root?
@mattduguid, based on @rmbolger's analysis you would have to get the "terraform-provider-acme" developer to add this functionality to that package, or else use a different ACME client that can already do this.
Per my post above, you could also mention this in the other thread to make the Let's Encrypt staff aware of some unintended impacts from their changes to the staging environment, although I think there's no likelihood that those changes will be reverted in the short term!
if thats the case then currently staging certificates are broken for us for us in azure and we can only use production which brings in issues with rates limits, in terms of raising this as a RFE for the acme provider is there any technical guidance on how to select the alternative chain I can supply them
It's akin to the
--preferred-chain option in the Certbot and acme.sh clients:
The ACME protocol mechanism for this is called the "alternate" link relation:
Client implementers who want more information about this can also come to this forum and post a question in the "Client Dev" category, and they'll get a lot of help and advice!
As noted up-thread the expired root is intentional and production will eventually offer a similiar default chain. Now is the best time to learn this and work with your client developers to get alternate chains working so you can select a different one in Production when Let's Encrypt changes the default.
We should probably start a new thread to discuss the use case for uploading Staging certs. It's not recommended to use them in your trust store:
Important: Do not add the staging root or intermediate to a trust store that you use for ordinary browsing or other activities, since they are not audited or held to the same standards as our production roots, and so are not safe to use for anything other than testing.
This statement is incorrect. The staging certificates are not broken, your client is broken and/or limited in functionality.
Although this is annoying, you should be relieved that you are discovering this against the Staging system, long before this is rolled out into the Production system.
In my experience, the vast majority of ACME clients are either poorly written, do not implement the spec correctly, or both. The staging change, while inconvenient, is surfacing many of this issues for people.
If LetsEncrypt were to revert any of these changes, it would not solve any problems, it will exacerbate them - allowing your team to accrue increasing amounts of Technical Debt.
As others said, you have two correct paths forward:
- Work with your client's author to make their code function correctly.
- Select a new client.
Going beyond what @schoen said, while the ACME spec provides for alternate links, it is up to the client developers what to do with them. A very-safe option is to download/process all the alternate links and save them to disk. The more popular option is to allow clients to specify the preferred chain, and only save that one to disk.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.