(STAGING) Doctored Durian Root CA X3 is expired (breaks test environment)

Pulling a specific problem out of this thread: New issuer for letsencrypt staging

After the migration to the new staging environment certificate hierarchy (Staging Hierarchy Changes), there is a new root CA certificate with the issuer CN Doctored Durian Root CA X3.

We've found that certificate (see New issuer for letsencrypt staging - #6 by jgehrcke) and started adding it to trust stores for testing purposes (we understand that this is insecure per se).

However, canonical chain verification algorithms won't pass because that root CA certificate is set to expire in the past, namely on Jan 30 14:01:15 2021 GMT.

This was first reported by @harishmurali here: New issuer for letsencrypt staging - #5 by harishmurali

In some testing scenarios, one cannot simply set the individual TLS client to 'insecure' mode -- but instead, has to do so implicitly, by adding an insecure trust anchor to the trust database. This is precisely what we do; and what we'd like to be able to continue to do.

Thoughts and quick help would be appreciated. Thanks!

3 Likes

This is intentional. The Doctored Durian Root CA X3 staging root is an analogue for the real DST Root CA X3, which will expire in September of this year. See our previous announcements and discussions (both in this forum and on the blog) about why we are offering a chain with an expired root as the default chain, while offering a chain with a non-expired but slightly-less-widely-trusted root as the alternate chain.

We backdated the expiration of Doctored Durian Root CA X3 precisely so that folks who interact with staging would see these sorts of issues now, rather than when the real DST Root CA X3 expires later this year. If your use case requires that the trust root be not expired, you need to request the alternate chain.

13 Likes

@aarongable Probably a heads up (with actual dates et cetera) would have been nice, even if it's the staging environment :slight_smile: The current API Announcement thread is in the past tense.

Just guessing here, but might it be Let's Encrypt underestimated the use of the staging environment for these kind of testing environments? And the consequences of these recent unannounced actions? This is not the only thread about it.

6 Likes

The Staging environment is just that -- an environment in which we can stage upcoming changes, and get some amount of real-ish traffic to confirm whether or not those changes break anything.

It has no SLAs, and no performance, uptime, or data integrity guarantees. It exists for the explicit purpose of breaking things, so that we don't accidentally break things in prod. Anyone whose infrastructure relies on staging operating exactly like prod is working under false pretenses.

Obviously, there's a balancing act here. If we make staging so unreliable that no one can accomplish anything useful with it, then we'll cease getting any traffic and staging will no longer serve its purpose. So we get paged when staging is down and try to fix it as quickly as we can, we post announcements weeks or months in advance that staging will be changing at some point in the future, and we post announcements again when changes are actually made to staging.

In this particular case, deploying the new certificates on staging was part and parcel with a much larger (but otherwise invisible) change to how staging operates as a whole. We didn't know how long that change would take, and when it was finally ready we made the call to proceed with it ASAP in order to unblock other changes (like beginning ECDSA issuance in staging!).

In engineering, there are always tradeoffs to be made. In light of the unreliable-by-design nature of staging, in this case we decided to move quickly at the expense of having staging announcements up days or weeks in advance. It's perfectly reasonable to believe that that wasn't the right tradeoff to make, and we'll keep an eye on how much community support has to happen (and try to weigh that against how much community support would have had to happen even with announcements), and see if we want to change our calculus in the future.

14 Likes

This is brilliant. Thank you.

6 Likes

Thanks for the elaborate explanation @aarongable ! Note that I'm just playing advocate of the Devil here, personally I don't use the staging environment that much, but I just noticed more than one of these threads within a short amount of time :slight_smile: Also, might I suggest to include this information on the staging environment documentation page? It now contains a small paragraph about the staging environment not being suited for CI. Perhaps a few lines from your current explanation would be helpful in that paragraph too.

8 Likes

@aarongable I'm a little confused. Is the staging environment currently broken, or working as intended? We need to use it to avoid throttling limits from your prod environment while testing. We are using cert-manager, and by default it is getting the already expired root, though it is unclear if that is a cert-manager bug or a Let's Encrypt bug. What root of trust is supposed to be used for staging? Your link doesn't make it clear. We found "(STAGING) Pretend Pear X1" but it is not self-signed and thus incompatible with certain libraries.

4 Likes

Lovely, looks like this is the right technical solution that I wasn't aware of. Thanks for this wonderful discussion here everyone!

5 Likes

The API Announcement is so confues.

On 18 Feb

(STAGING) Artificial Apricot R3 and chain to our new Staging root (STAGING) Doctored Durian Root CA X3

But on 20 Feb

Addtionally, we noticed our test hierarchy has the wrong AIA CA Issuers URI and CRL Distribution Point URI on the certificate: (STAGING) Artificial Apricot R3 signed by (STAGING) Doctored Durian Root CA X3 . That fix will be done in the next few days too.

Apparently, both mentioning (STAGING) Doctored Durian Root CA X3, so is the expired root cert will be corrected in the next few days?

Or a more simple question waiting to be answered, is the staging env are able or will be able to get a fake cert with the root cert is not expired?

3 Likes

I can see the issue is now has been fixed, (STAGING) Doctored Durian Root CA X3 is not the root cert anymore, the new root cert (STAGING) Pretend Pear X1 is come with a valid date.

So much thanks!

4 Likes

There are two root certs, my friend. :slightly_smiling_face: (Three if you want to be really technical.)

This was the root for the primary chain of issuance:
(STAGING) Doctored Durian Root CA X3 self-signed

This is now the root for the primary chain of issuance:
(STAGING) Pretend Pear X1 signed by
(STAGING) Doctored Durian Root CA X3 self-signed

This was/is the root for the alternate chain of issuance:
(STAGING) Pretend Pear X1 self-signed

5 Likes

I think a key thing here was that the expired root in the chain is intentional to test out the planned production strategy for handling Android etc. The alternate chain is for those who don't want/need the expired root in their chain, I believe so anyway?

3 Likes

the expired "Doctored Durian Root CA X3 self-signed" is not yet removed from the chain. It is still there. Those who don't want the expired cert should request for an alternate chain that has (STAGING) Pretend Pear X1 self-signed?

4 Likes

I see that the text in Staging Environment - Let's Encrypt was updated. Thanks for that!

Quoting, for posterity:

The staging environment intermediate certificate ("(STAGING) Artificial Apricot R3") is issued by a root certificate not present in browser/client trust stores. If you wish to modify a test-only client to trust the staging environment for testing purposes you can do so by adding the "(STAGING) Pretend Pear X1" certificate to your testing trust store. Important: Do not add the staging root or intermediate to a trust store that you use for ordinary browsing or other activities, since they are not audited or held to the same standards as our production roots, and so are not safe to use for anything other than testing.

Thanks to the entire Let's Encrypt team for the discussion and support in this thread. The technical solution to simply trust Pretend Pear X1 instead of Doctored Durian Root CA X3 is of course super nice and clean. I have used the opportunity to read up on the cross certification concept used by the Let's Encrypt cert chains (for the record: super nicely documented here: Chain of Trust - Let's Encrypt). As far as I am concerned, this 'issue' is resolved to my satisfaction, and it's also fair to say that it never really was a technical issue -- but just a lack of understanding, at least on my side. I feel like I've learned a lot about LE in the process of thinking this here through! Thanks again.

5 Likes

That's correct. The alternate chain should not be changing for awhile to my knowledge. Thus, both the primary and alternate R3 intermediate certificates (used for signing user leaf certificates) will soon be signed by an X1 root certificate. The only difference between the two chains will be that the X1 root certificate for the primary chain will be signed by Root CA X3 while the X1 root certificate for the alternate chain will be self-signed.

Please note that either root X1 certificate can be used to authenticate the R3 intermediate certificate served by either chain. That's because both root X1 certificates use the same private key. The very definition of cross-signing is having multiple certificates for the same subject using the same private key that are signed by different certificates (Root CA X3 and Root X1 in this case).

4 Likes

agree we use staging to not hit rate limits for production, maybe there should be another level of environment for breaking changes? at the moment this has broken our environment builds using terraform/azure as the certificate is not allowed to upload to azure due to the expiration of the root?

1 Like

@mattduguid, there was a recent thread where other people (including me) proposed something similar to what you suggest here.

This led to this statement from Let's Encrypt staff:

You can post your experience and opinions in the former thread, but following the latter thread, it seems like Let's Encrypt's current intentions are not to guarantee that staging and production will maintain a particular level of equivalence or parallelism with each other such that you can reliably condition a production issuance decision on a successful staging issuance.

3 Likes

we use terraform resource "acme_certificate" to request the certificate from "https://acme-staging-v02.api.letsencrypt.org/directory"

a number of comments advise "Those who don't want the expired cert should request for an alternate chain" is there any further information on how to achieve that?

1 Like

Assuming this is the terraform resource you're using, it doesn't appear to currently support alternate chain selection. The author would have to add that feature in order for you to use it.

Looks like you could submit an issue with the feature request in their github repo.

7 Likes

Since this change if we attempt to upload a staging cert to azure via their API (terraform or portal UI) we get back the error "Expired certificate is not allowed". We have confirmed with Microsoft that this is because their validation process is still using the chain with the expired root cert "Doctored Durian". There are a number of comments in this forum advising those who dont want the expired cert should "request for an alternative chain" with root cert "Pretend Pear", can anyone advise how this is requested to allow Microsoft to validate against the non expired root?

1 Like