Standards and Best Practices for Obtaining and Perhaps Validating Chain Certificates with an ACME Client

Client dev
1

Synopsis
How do I download what I expect to download and have the chain files I want to have?
RFC 8555, part 7.4.2 “Downloading the Certificate” has normative standing with indefinite expression of the certificate-chain span of the certification data provided by the ACME2 server. What certificate-chain spans are standardized and what spans are provided in practice by Let's Encrypt and other Certificate Authorities (CAs)? What and how can an ACME2 client specify for certificate chain span?, what span can an ACME2 client reasonably expect?, and what should an ACME2 client do to request and process the certification data provided by the ACME2 server? Can I rely on a big-chain download from the ACME2 server of any compliant CA? How do the alternative certificate chains ‘fork’ from the end-entity certificate?
I am not aware of a reasonably direct and definitive source of information about my certificate-chaining-span concerns. I am trying to cobble together a practical understanding. What are the fundamentals to be grasped and how definitive are they?
As it stands now, I would probably assume, and moreover enforce by trimming, the DV certificate alone and try to download the higher certificates one by into a cache directory until I get the root certificate. Should I validate the chain? Seems daunting; have not studied it seriously yet. Then I would concatenate the PEM encoding formats as most completely appropriate for nginx and Postfix.
Thanks.

My Chain File Use Cases
nginx Web server current in 2025, module ‘ngx_http_ssl_module’, directives ‘ssl_certificate’ and ‘ssl_certificate_key’ require references to files in Privacy-Enhanced Mail (PEM) format. The use of ‘ssl_certificate_key’ to specify the key file can be obviated by putting the private keypair data in a PEM chain file with its certificate one of the files referenced by ‘ssl_certificate’. As I understand it, the provisioning of the root certificate by the Web server is superfluous because a root certificate is not trusted except as determined by the Web client with its cache of trusted root CA certificates.
Postfix email server current in 2025, configuration file ‘main.cf’, parameter ‘smtpd_tls_chain_files’ requires reference to files in PEM format. Each file should have a certificate's private keypair and the corresponding certificate and possibly more up to the root. It won't hurt to include it (as far as I know).
It is my understanding that nginx and Postfix servers start or otherwise ‘configure’ with root privileges and have access to not only the configuration files owned by their dedicated, respective non-root users but also the key and certificate files, wherever they may be located and however they may be owned in the relevant file system. I run my client software without root privileges. The http-01 challenge requires the provisioning of particular resources. I put those resources in to a dedicated directory without root privileges, and I believe that nginx Web server does not have root privileges to access the content of that directory.
How much do I need chain files for nginx Web server and Postfix email server? I feel bad if I don't score at least an A on the SSL Labs test for my Web server configuration, but I'm tired of this ACME client coding and might just skip it. I have been manually building my chain files. That is not practical with 6-day certificates. When will that become mandatory?

Chain-Nonspecific Certificate Data
RFC 8555, part 7.4.2 “Downloading the Certificate” simultaneously supposes the download of a single certificate with singular expression and the download of a nondescript certificate chain with plural expression. The section further refers to a default format ‘application/pem-certificate-chain’ without indication of a standardized way to get any other format, which without further clarification may or may not indicate specification of which chain certificates to include. If a chain is provided, does it include the root certificate? Is chain validity a security risk that the ACME2 client should address with chain validation before bundling into a chain file?
The part of the standard also identifies that ACME2 servers MAY provide ‘alternative certificate chains starting with the same end-entity certificate’. That's not literally possible without some inexplicit critical information. Certificates follow the X.509 (structure and encoding?) standard. I have never seen an end-entity certificate signed twice, which would provide a ‘fork’ for two alternative certificate chains from the very same end-entity Domain Validation (DV) certificate. Perhaps there are instead alternative certificates with the same end-entity domain validation (i.e. ‘certification’)?

Certificate Span in Practice
My experience with my personal Python 3-based client is entirely with Certificate Authority (CA) Let's Encrypt (LE) 2020–present (May 2025) and Buypass 2020–circa February 2025. I have only observed that the single Domain Validation (DV) certificate in PEM format is provided, after all the authorizations have been completed for the domains in question as is usual. However, threads of this forum composed just prior to 2020 indicate that LE supplied a certificate chain of some sort. The forum thread “ACME Account Registration in Python”, ACME Account Registration in Python , 20 March 2025 eventually digresses to the issue of certificate chain span. The author of the initial thread post wrote in a later post that his CA, Sectigo/InCommon, ‘included the whole chain in the response’.

Chain Validation
How important to security is certificate chain validation? Do alternative chains fork from a single certificate (in X.509), and should software code be able to detect the alternatives in a single certificate? If so, what are the ways to do it, and how do they compare? Does certbot validate the chain?

Certificate Processing Alternatives
For some reason, the online record shows that some people or some uses cases find the use of child processes (or subprocesses) with Python to invoke OpenSSL to be unacceptable. Why is that? I can guess industrial speed for high volume processing, but that's only a guess. I don't know if I should be worried about it. I currently use OpenSSL via subprocesses.
Specifically, I compare the output of ‘openssl pkey -text’ to a large regular expression to determine key type and public key values. With certificate chaining using the same brittle (and light, simple) technique I would compare the output of ‘openssl x509 -text’. Can there be two URIs to the next certificate. It seem to me that checking the signature or key information of chained certificates would be a good idea. I am ignorant of these specifics. I am comfortable using OpenSSL manually to generate keys and certificate signing requests and to inspect keys, certificate signing requests, and certificates. As you may know, OpenSSL will only inspect the first (should be of the end-entity) key or certificate of a chain file in PEM encoding format. I know to cut and paste to expose the non-first entities of a chain file to inspection with OpenSSL -text.
I am aware that ‘certbot’ has recently transitioned from 'pyOpenSSL' to 'pyca/cryptography'. My current workload just to get my client code into working condition is daunting without another little mountain to climb. I am not sure how necessary the chain bundling feature is, but the PyPI package 'pyca/cryptography' looks like it should facilitate it easily enough. I can't be sure I can navigate the ‘dinosaurs with lasers’ without a commitment of weeks. I don't see how the library package is useful without plunging into the ‘hazardous material’.

2

Wow. Usually we ask for people to explain more but that is a lot to digest. There is a significant learning curve to writing your own ACME Client. And, in understanding all the nuances of TLS. It's beyond what I'm willing to help you with but I'll address at least this

The certificate URL will return the leaf and its intermediates. I don't know why you are only seeing the leaf. Perhaps there is a problem in your code once you get it? The Let's Encrypt chains are described here: Chains of Trust - Let's Encrypt

When retrieving the certificate URL there may be an alternate link provided. This depends on the CA and perhaps other criteria. LE offers an alternate for an ECDSA cert to chain to its X2 root, for example.

No. A TLS Server should not send the root. No client should trust it anyway. It should only validate against its already known trusted roots. It is useless and wasteful of bandwidth to provide it in your server's reply.

5 Likes
3

Thank you, Mike. I was thinking that it would make sense to just provide the whole chain and be done with it. I think it's weird to not do that, but what do I know? I will be looking to download a certificate chain. Maybe it will be easier than the approach of a file cache I started coding yesterday.

I spell things out when I write because clarity will help someone else (and I sometimes need a reminder). It is a lot of information, perhaps too much. I think you got to the essence of my problem.

I don't know why it would hurt to include the root certificate for Postfix directive smtpd_tls_chain_files. It seems to me that the server should be smart enough to exclude anything unnecessary. I do not understand the DNS-based Authentication of Named Entities (DANE) use case of Postfix email or what future DANE has as a technology. It would be a CYA for me (being ignorant of the why and wherefores). I don't think the bandwidth usage of root certificates would be much in the overall scheme of network traffic. Decades ago I had a friend who quipped the primary purpose of the Internet was the distribution of certain videos. CAs having to remember all those used keys 'forever' seems sorta crazy to me. Wouldn't five years be enough?

Course correction appreciated. You know that feeling when the programming issue is just you because you're doing something wrong that's original? I should have known when the Web oracles did not show me that others have had the same problem I have now. I will be investigating the apparent uniqueness of my crazy code.

4

If my parlance is correct, the body of the 'success' reply from the staging LE server was a JSON 'Order Object':

{'status': 'valid', 'expires': '2025-05-30T01:09:52Z', 'identifiers': [{'type': 'dns', 'value': 'doug.thoal.us'}, {'type': 'dns', 'value': 'salvage.thoal.us'}, {'type': 'dns', 'value': 'thoal.us'}, {'type': 'dns', 'value': 'www.doug.thoal.us'}, {'type': 'dns', 'value': 'www.salvage.thoal.us'}, {'type': 'dns', 'value': 'www.thoal.us'}], 'authorizations': ['https://acme-staging-v02.api.letsencrypt.org/acme/authz/195966004/17533380374', 'https://acme-staging-v02.api.letsencrypt.org/acme/authz/195966004/17533380384', 'https://acme-staging-v02.api.letsencrypt.org/acme/authz/195966004/17533380394', 'https://acme-staging-v02.api.letsencrypt.org/acme/authz/195966004/17533380404', 'https://acme-staging-v02.api.letsencrypt.org/acme/authz/195966004/17533380414', 'https://acme-staging-v02.api.letsencrypt.org/acme/authz/195966004/17533380424'], 'finalize': 'https://acme-staging-v02.api.letsencrypt.org/acme/finalize/195966004/24827091224', 'certificate': 'https://acme-staging-v02.api.letsencrypt.org/acme/cert/2cac455e3374b2cc1d74273f8a20f59f66ff'}

My code uses the 'certificate' 'resource':
# Download the certificate.
body, code, _ = query_acme_server(
order_dict['certificate'],
directory_dict['newNonce'],
payload_obj='',
authen_key_ops=account_key_ops,
nonce=nonce,
kid=kid)

I will now ponder what I might be doing wrong.

5

certificate link has two certificates in it in pem format, try open that file in notepad. Are you using right function for parsing it?

(keep mind on plural 'certificates')

4 Likes
6

I don't know for certain about Postfix but generally TLS servers do not second-guess what you tell it to do. If you tell it to send the certs in a certain file that is what they do.

You are making a pretty big assumption "It seems to me ..." which isn't reasonable

We deal with problems all the time when someone gives a server the wrong file and certs. I can assure you servers don't "fix" things. Now, browsers on the other hand do. They work very hard to deal with poorly constructed web servers while still honoring the TLS trust.

4 Likes
7

Hi, @orangepizza. I can't remember the clicky interface way to insert a mention, hope I did it right. You are are a genius on Windows? I use Debian. I just used cat to see the download.

YOU ARE RIGHT! It is two certificates with one blank line in between. The second certificate is named '(STAGING) Pretend Pear X1'. I don't know why my current production certificates have only one certificate before manually downloading and adding them. Weird.

I have this function to alter the data:

def normalize_newline_encoding(text):
    """
    Replaces all occurrences of '\r\n' and otherwise '\r' with '\n'.

    Caveat: This function does NOT ensure the last character is '\n'.
    """
    alt_lines = []
    from_pos = 0
    match = None
    for match in re.finditer(r'\r\n?', text):
        alt_lines.append(text[from_pos:match.start()])
        from_pos = match.end()
    if match:
        if from_pos < len(text):
            alt_lines.append(text[from_pos:len(text)])
        return '\n'.join(alt_lines)
    return text

Addendum on 24 May 2025:

I am sure that I want to scrub/normalize line break encodings to the internal convention of Python 3 strings (LF/'\n') and output using the 'universal newlines mode' of Python 3's built-in function open(), which does 'the right thing' according to the file system (presumably according to the operating system, afaik). It seems that this has been considered before: " Does the newline implementation matter?"

RFC 7468, dated April 2015 (apparently not obsolete), part 3 specifies that any of the historically normative line break encodings are valid: "eol = CRLF / CR / LF". I think I'll even add a trailing line break if the last line ends in something else.

8

They don't. Let's Encrypt sends the leaf and intermediates. You must be losing something.

You shouldn't modify the cert data anyway. You don't need that function. And, pem doesn't have crlf line endings, just lf. And, that is how LE sends it out.

5 Likes
9

Thanks, @MikeMcQ. I will keep that in mind, but when you don't know, you must guess.

From Postfix TLS Support (not as pretty):

# Postfix ≥ 3.4. Preferred configuration interface. Each file
# starts with the private key, followed by the corresponding
# certificate, and any intermediate issuer certificates. The root CA
# cert may also be needed when published as a DANE trust anchor.
#
smtpd_tls_chain_files =
/etc/postfix/rsa.pem,
/etc/postfix/ecdsa.pem,
/etc/postfix/ed25519.pem,
/etc/postfix/ed448.pem

Like I said, I am not familiar with DANE and likely never will be. I try stuff and see. I used to blindly follow documentation, but that burnt my ass and I learned that 'hello world' testing never stops. I will never get the finer points. I am not that smart. I am not trying to publish a DANE trust anchor. I use the http-01 challenge. I think my client software is deficient without dns-01 challenge, wildcards would be nice, but another topic that I just can't investigate for lack of capacity.

10

Sure, or you can research or ask people more experienced than you. Might save you a lot of trouble before digging a deep rabbit hole like you've done already. Look how much work you've done thinking about chains when you don't have to.

I'm sure this comes across as preachy. It isn't intended as that. More as sage advice :slight_smile:

4 Likes
11

Not too preachy. I do better alone. I don't like it, but it's a fact. I am old enough to be sure. It is great that people who know what's going on provide help here.

I can't understand why I am getting a chain now. That is concerning. It is late for me and I will look at all this tomorrow and try to figure out what I am doing. I am in the middle of an involved update of my client code.

I can guess that I need to count the certs and log it. Not sure after that. Yes, lots of coding work could be scrapped. Are single certificate files even worth saving? Lots for me to ponder. Goodnight, everyone (pretty soon at least).

1 Like
12

No, why?

Probably not.

3 Likes
13

I took a look at "Chains of Trust", 11 June 2024, Chains of Trust - Let's Encrypt as suggested by @MikeMcQ. I have come to these conclusions as my current working ideas, subject to corrections as merited. I hope this helps someone else.

(1) A digital certificate document is in a structural and encoding format called X.509.
(2) An X.509 certificate document is about a subject party and keypair combination and is necessarily signed exactly once, whether by itself or by another key, which is normally certified as the subject of a different X.509 certificate document next higher up in the chain.
(3) A certificate chain is always linear, no forks. Each X.509 document has exactly one issuer to go with the one signature. Each X.509 document has exactly one subject party (implied for domain validation) and keypair.
(4) However, Certificate Authorities (CAs) may publish a primary certification of an abstract certificate identity signed by any number (probably just two) of issuer party and keypair combinations.
(5) Each of these variant certificate variants should have or has a distinct issuer party with a distinct key (as the astronomical times astronomical odds of randomized key generation will have it virtually always, assuming adequate implementation).
(6) Each X.509 certificate that is not self-signed specifies a single URI (URL and/or URN?) for the X.509 certificate of the issuer. I wonder how important that URI is for trust.

Any mistakes so far?

Consider the Let's Encrypt root certificate 'ISRG Root X1', which was root in the sense of highest ranking certificate under control of Let's Encrypt and perhaps in the sense of the aspirations and intentions of Let's Encrypt at the beginning of its production operations before they had a credibility from Web client software people for trusted 'root' certificates.

I am guessing that the first 'ISRG Root X1' X.509 certificate was the one signed by Digital Signature Trust Co. using their abstract certificate identity 'DST Root CA X3'. Because the certificate issuer and subject identities were different, that 'root' X1 certificate listed the URI of the issuer party and key combination as identified by the 'DST Root CA X3' certificate.

Some URI must have been used to refer to 'ISRG Root X1' as the issuer of directly subordinate intermediate Let's Encrypt certificates, and I am guessing that the certificate resource at the URI was the first and initially only version of 'ISRG Root X1'. I can guess (not sure) that Let's Encrypt made a second self-signed version of 'ISRG Root X1' that was not at that or any URI, but was promoted and made available to the software people who control the caches of trusted certificates that come with operating systems or Web browsers. I am guessing that the certificate initially at the URI for issuer certificate 'ISRG Root X1', was replaced with the self-signed variant.

The variant signed by CA Digital Signature Trust Co. was retired at some point per the webpage I just read. Was it considered 'cross-signed' at the beginning or did it become cross-signed when it was retired? I suspect that depended on one's perspective of how much trust was in the certificate 'ISRG Root X1'. Cross-signing seems to mean 'not the primary signing'.

I wonder if the use of trusted certificate caches of Web client software does not consider the URI of the trusted certificates as given in the certificates it or another variant of it has signed. I am still unclear on that.

The designation of cross-signed certificates given by the instructive Let's Encrypt webpage analogously indicates that the perspective and intention of Let's Encrypt is to transition from a hierarchy of their certificates with RSA keys to a hierarchy of their certificates with NIST prime elliptic curve keypairs. That is not surprising given that technology advances and RSA keys are old tech. Interesting management technique, don't ya think? Now about those Edwards keys. lol

I thought it would be nice for someone else to realize that chaining 'forks' are pure abstraction (even by digital standards, it's something like a 'legal fiction' that is the law). Any X.509 certificate has but one chain (even in the degenerate case of self-signing). Maybe I am slow for taking words too literally. I have an uncommon mental 'wavelength'. Also, there is no spoon, pretty sure.

14

A key, identity pair can have multiple certificates issued for the pair, so while E5 might have one key, there are multiple certificates for it (one signed by X1 and one signed by X2) and getting a different chain means getting a different E5 (or whatever intermediate is used) certificate. However as the public key is the same in both, the signature for the end certificate will be the same.

If you look at the different txt representations of certificates available on Chains of Trust - Let's Encrypt the Subject Public Key Info remains the same.

6 Likes
15

As @MaxHearnden suggests there can be more than one "version" of an specific issuer certificate, which in turn can be chained differently to a different root, this can be useful for compatibility.

Some client systems (e.g. Windows) build their own preferred path through the chain based on their own trust store (i.e. if it doesn't trust the served chain, but it trusts a version it can build), but most others take the chain you serve via your webserver etc literally and they either trust the root your last intermediate points to, or they don't.

3 Likes
16

That is incorrect. Here is my correction:

(4) However, Certificate Authorities (CAs) may publish one or more certificates of an abstract certification identity signed by whatever certification identity. A certification identity is an issuer party and keypair. (I don't know how trust stores work, but the public key could be all that matters for trust by an implemention for all I know.) One certificate of the variants may be designated as the 'primary' certificate as a matter of perspective and policy.