Discovering/Identifying Roots

Client dev
1

I ran into an issue while working on some tests as I prepare for longer chains - there doesn't seem to be a way to identify the roots. (essentially the same situation as in How to get hold of root certificate? · Issue #152 · letsencrypt/pebble · GitHub, but I'm not limited to pebble)

The fullchain file does not contain roots, only the intermediates, as the root is not needed by end-users (website admins and visitors) as it is in a browser/os/app trust store. Pebble now has a feature to expose the root(s). Boulder does not support this, and most people seem to be copying from computer/browser trust stores.

Does anyone have good strategies for doing the following detecting / identifying roots ? it seems like the only real option is to code the production "expected" roots in my client, and interrogate pebble for them during tests. This is less than ideal, because it requires a code update if a new root is deployed, as I wold essentially be operating a Trust Store. I would prefer not to do that, I just want to validate the certificate chain.

Some use-cases:

  • Upon downloading a certificate, I need to analyze the chain and make sure everything validates. (This is as much to ensure against our own bugs than distrusting the ACME server.) Most libraries and utilities require the root to be identified for this.

  • We need this info to make informed decisions on chains.

Related:

3 Likes
2

For development against boulder running in docker, I found it easier to mount the location in the container that holds the certificates and read them from there. Similar for pebble.

This rules out end to end testing against the dev server though, production at least you can follow the chain itself and grab the root cert from your system cert store (depending on programming language though this can be easier said than done).

2 Likes
3

I may be misremembering but I think pebble used to have an endpoint for certificates to get the current root, I'm not sure if it still does.

3 Likes
4

Maybe /root?

3 Likes
5

Ha, I just realised you already mentioned all that pebble stuff. I completely agree ACME should include a standard endpoint for root certs because they can be hard to find for some CAs, how people then use those would be up to them.

3 Likes
6

Yes, Pebble has endpoints for that, also for downloading all intermediate certificates and private keys: GitHub - letsencrypt/pebble: A miniature version of Boulder, Pebble is a small RFC 8555 ACME test server not suited for a production certificate authority. Let's Encrypt is hiring! Work on Pebble with us.

For production, this is a valid problem. I guess the best way is to look at the OSes trust store and try to find the issuing certificate of the last intermediate in the chain in there.

I remember that I started a thread on the ACME protocol mailing list about this (Archive - the mechanism was inspired by An optional MIME parameter for application/pem-certificate-chain? · Issue #435 · ietf-wg-acme/acme · GitHub, and I agree that it wasn't a very good idea; probably having a Link header for this would be better, or as others suggested in the discussion, a flag during order creation or something else) - I guess someone has to bring up a proposal for an extension to the specs, and then this can be discussed and hopefully turned into an RFC eventually.

4 Likes
7

There are so many things I'd love to see in the RFC. Sigh. :frowning:

2 Likes
8

You can discover the Issuer of the furthest certificate in the chain, you should be able to get both a name for it and an Authority Key ID. The full name ought to be unique in PKIX, in terms of what it refers to, but there may be more than one certificate with that name (there may be cross signatures) and the Authority Key ID designates a specific public key, in a sense this key "is" the root rather the self-signed certificate with that key.

I believe either approach would be a sensible way for someone to figure out which root a chain "leads" to (but notice that TLS 1.3 specifically says that's not how this works, and in practice for earlier TLS versions likewise good clients do not treat the "chain" as anything more than some optional ingredients for a recipe to pick a trust path with their own logic)

But for your first bullet I don't think either of these things solve your problem. In fact I think the problem is philosophical, you desire to "validate" that something is trusted, and yet, you desire not to operate a trust store and thus you don't want to decide what's trustworthy. It's OK not to want to make trust decisions, but from what I can tell here your goal is in fact to make trust decisions, so there's no way around that.

You might plausibly consider consuming an existing trust store (most likely Mozilla's) and "make sure everything validates" with that, as this would catch "our own bugs". Or you might consider using an OS installed trust store (if you use a Free Unix it's probably essentially the same trust store as Mozilla's but on Windows or MacOS it's pretty different). So you could punt the trust question to somebody else, but you can't just sidestep it because that won't end up meaning anything.

For your second bullet point I think the information in Issuer and Authority Key ID ought to help you.

2 Likes
9

First, thanks for your reply.

I guess I am not clear about my need. I am not trying to validate "Trust" nor assume any amount of "Trust". The concept of "Trust" is irrelevant to me. My client could be used within the realms of "Public Computing" (default browser and trust stores) and "Private Computing" (internal/privately managed stores). The public trust stores also change frequently. Within the realms of CI tests, servers like Boulder and Pebble will generate ad-hoc Root certificates. A Trusted certificate could also be revoked and untrusted, eventually removed from a "Trust Store". So there is no guarantee or assumption that a utilized root will appear in the browser/os Trust Store. Sorry to harp on these points, but I want to make this clear: dealing with the concept of "Trust" is completely irrelevant to my concerns and needs. I am not looking to develop code that is concerned with "Trust".

Within the concerns of my usage, a conventional "Trust Store" does not exist to provide "Trust", but functions as a black box that may or may-not contain a terminal certificate. That causes a lot of problems for my usage, so I am actively looking to remove it from the equation.

What I do wish to accomplish, is validating a downloaded certificate against it's downloaded chain. Let's call this the "self-referential integrity of a certificate chain", as I can't think of a better name for it right now. The vast majority of SSL libraries and toolkits do not easily support this without being provided with the terminal root(s) (more than one root is indeed possible) involved -- and that terminal root must be self-signed. (If only we could treat the last intermediate as the root...)

I am not alone in this -- mailing lists, github, stack overflow, and discussion forums are filled with engineers and computer scientists going through extreme workarounds trying to do the same thing (in just about every situation) for CI/Unit Tests. In these cases, no one cares if the end certificate is "Trusted" or not - that is not our call to make, what matters is identifying and obtaining the terminal certificate(s), so the entire chain can be verified.

I would be fine doing this entirely based on signatures, as all I really care about is "Did the first intermediate sign the cert?", "Did the upchain intermediate sign the downchain intermediate?" - but that started getting absurdly complex and is hard to do with the basic OpenSSL commands. To keep installation requirements minimal for the emergency debugging features, we support them as a fallback if python's crypto tools are not installed.

So my problem is better stated that OpenSSL/etc requires a self-signed Root to validate the referential integrity of a certificate chain, but - as @felixf stated - the ACME spec does not provide for identifying and procuring a root, only the intermediates. Again, to be clear, I am not trying to test if a Chain is trusted - I am trying to test if a Chain is well-formed with it's own self-referential integrity.

My interim solution is probing Pebble for tests, and we already hardcode the ISRG roots, but this is far from ideal and essentially does a lock-in to LetsEncrypt for something that should work against any ACME server.

3 Likes
10

I see. Well, I guess @felixf has the best way forward for you, engage with the process that created and updates ACME, to get this feature into the standard or an adjunct, and then lobby Let's Encrypt (and any other servers you care about) to prioritise implementing the now-standard feature.

2 Likes
11

I believe boulder doesn't even know about the root certs in the dev/production environment because it works with the R3/R4/E1/E2 intermediate certs - has no actual knowledge of the root certs. So programatically, or as part of the RFC, I'm not sure it would make sense to include the root certs as part of the API somewhere.

Verifying the chain for security purposes and blindly trusting the root they tell you to trust would definitely introduce an attack vector. From a security perspective, it also makes sense to have the root trust cert obtained/stored out of band. Isn't that kind of the point of the system trust store anyway? Especially if you can query it, it's not that big of a black box but at that point you're back to hardcoding and verifying.

If for whatever reason you cannot trust the system root store as part of your security model / environment, hardcoding certs (either the full cert or any identifying part of it in terms of fingerprints / ID's) seems like the best way to do it, imo.

3 Likes
12

When talking to a "real" ACME service (e.g. our prod or staging environments), you should be able to use the Authority Information Access url from the intermediate to download the full contents of the root certificate.

In other contexts, such as running an ACME service as part of your CI testing setup, you could add a small amount of additional infrastructure to do the same. Generate your ephemeral roots, load them into a tiny webserver, grab the URLs at which they'll be served, put those into the intermediates, then load the intermediates into the ACME server.

5 Likes
13

My client also tries to verify an obtained certificate up to the root. It does this by following the CA Issuers links found in the certificates for certs not in the chain. I got this from the new staging:

Chain #0 (default):
    Subject: CN=dns.acme-test.labs.vu.nl
    Issuer: CN=(STAGING) Artificial Apricot R3,O=(STAGING) Let's Encrypt,C=US
    CA Issuers - URI:http://stg-r3.i.lencr.org

    Subject: CN=(STAGING) Artificial Apricot R3,O=(STAGING) Let's Encrypt,C=US
    Issuer: CN=(STAGING) Doctored Durian Root CA X3,O=(STAGING) Internet Security Research Group,C=US
    CA Issuers - URI:http://stg-x1.i.lencr.org/

Chain #1 (alternate):
    Subject: CN=dns.acme-test.labs.vu.nl
    Issuer: CN=(STAGING) Artificial Apricot R3,O=(STAGING) Let's Encrypt,C=US
    CA Issuers - URI:http://stg-r3.i.lencr.org

    Subject: CN=(STAGING) Artificial Apricot R3,O=(STAGING) Let's Encrypt,C=US
    Issuer: CN=(STAGING) Pretend Pear X1,O=(STAGING) Internet Security Research Group,C=US
    CA Issuers - URI:http://stg-x1.i.lencr.org/

I think the second CA Issuers link is wrong and should have been http://stg-dst3.i.lencr.org/.

3 Likes
14

Agree with @kjb the intermediate certificate in the default chain has AIA CA Issuers that yields a certificate with Subject not same as Issuer in the intermediate certificate.

Also the alternate chain when downloaded has 1 EE/Leaf certificate and 1 intermediate - the entire chain looks to have four certificates EE->stg-r3.i.lencr.org->stg-x1.i.lencr.org->stg-dst3.i.lencr.org. I was under the impression (from this thread: Questions re: OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates - #4 by joeshaw) that the long chain would provide the EE certificate and the multiple intermediate certificates. Is my impression mistaken, or should the alternate chain download provide 3 certificates in this case?

Thanks!!

1 Like
15

Thanks! Both! This is working well so far!

This is NOT being done. I can't be any more clear about this. No one wants to do this.

The chain is being verified for self-referential integrity, and only that. No trust is being assumed or utilized. The certificate and chain is not being used for anything, other than verifying it is complete.

The use-case here is “I have downloaded a cert and chain from an ACME server; is it complete/correct?”. This is tested on unit tests, and ideally after download. This is not for “integrated tests”, which would care about reconciling with a trust store, or for any actual usage whatsoever. This is simply to ensure the ACME client has properly downloaded the certificates and maintained their structure/formatting.

The only way to accomplish this with standard libraries and command line tools, is to create an ephemeral sandboxed trust store, insert the (unknown) root, iterate the chain backwards to test the presented intermediate verifies against the root(s), and then testing the end entity cert in the same manner. This all happens within the scope of a “ensure_chain_integrity” function, or an OpenSSL "verify" command. There is no security concern or attack vector. No Certificates are being added to a system store; no operations are being performed with a Certificate outside of a sandboxed environment.

Most ACME client installations don’t need to test this, because they are concerned with a single server and just keep the chain/fullchain files as-is; if they can parse a single “—-begin certificate—-“ marker in the text, that is generally good enough. Should that stuff be tested? Sure, but they solve 80% of problems with 20% of work - there are better allocations for internal resources.

Clients designed for scalable networks are different. There is a lot of standardization, data normalization and caching going on, so the integrity of chains needs to be tested when downloaded from an ACME server and after they are assembled from storage.

Again, I can’t stress this enough — the goal here is to test the internal integrity of the chain, not determine if the chain is trusted or use the chain/elements for anything. This is something most clients do not implement, but really should. I've seen multiple clients use a regex or string-split to split a fullchain file into a cert+chain, but (i) lack testing that ensure the total number of certificates are correct and (ii) ensure their cert+chain are structured correctly.

4 Likes
16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.