Domain ordering not respected, unexpected certificate subject

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: poc.moxehealth.com (note: this was done in --staging but the behavior also exists in production)

I ran this command: certbot certonly -n --agree-tos --email admin@moxehealth.com --dns-route53 -d poc.moxehealth.com -d poc-san-1.moxehealth.com,poc-san-2.moxehealth.com,poc-san-3.moxehealth.com,poc-san-4.moxehealth.com,poc-san-5.moxehealth.com,poc-hello.moxehealth.com --expand --config-dir /tmp/config-dir/ --work-dir /tmp/work-dir/ --logs-dir . --staging

It produced this output: The certificate creation succeeded; however, the Subject (primary domain name) of the certificate is poc-hello.moxehealth.com when I am expecting it to be the first domain name added, poc.moxehealth.com

My web server is (include version): AWS ACM

The operating system my web server runs on is (include version): n/a

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): yes (the certificate is created using an AWS Lambda function)

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.4.0, also experienced on 1.20.0

2

Attaching a copy of the certificate generated by the command. I've also tried having everything as a single -d for the domains (so in my case it would be -d poc.moxehealth.com,poc-san-1.moxehealth.com,poc-san-2.moxehealth.com,poc-san-3.moxehealth.com,poc-san-4.moxehealth.com,poc-san-5.moxehealth.com,poc-hello.moxehealth.com) and I observed the same behavior.
poc.moxehealth.com-cert_and_chain.pem (4.3 KB)

3

My response here might help explain what's going on.

5 Likes
4

Thank you for the reply.

Our organization does weekly certificate renewals, and prior to the certificate renewals today, the first domain name provided was the subject - I can provide certificates generated for the past year indicating this. Your post indicates that this isn't an issue with certbot itself but how Let's Encrypt is determining the Subject - is there any change log of their behavior on this front?

In my case after generating the certificate I use the Subject name to determine which certificates our AWS load balancers use, and this change has broken that functionality.

You mention in your post:

Currently, Certbot does not provide any control over the Subject CN field. We have an issue open to track that (#9543) and you are free to leave your feedback there, but unless there is a sufficiently compelling argument in favor of including a Subject CN on the CSR, we are probably not going to do anything about it.

as suggested by the help which states that the first specified domain is going to be the subject

Our documentation should not say this. I had a look in the User Guide and I couldn't find where this is stated. Are you able to point me to where you read this? Thanks.

However, this text is included when I run certbot -h all with certbot 2.2:

 -d DOMAIN, --domains DOMAIN, --domain DOMAIN
Domain names to apply. For multiple domains you can use multiple -d flags or enter a comma separated list of domains as a parameter. The first domain provided will be the subject CN of the certificate, and all domains will be Subject Alternative Names on the certificate. The first domain will also be used in some software user interfaces and as the file paths for the certificate and related material unless otherwise specified or you already have a certificate with the same name. In the case of a name collision it will append a number like 0001 to the file path name. (default: Ask)

Emphasis on this quote: "The first domain provided will be the subject CN of the certificate" which is no longer accurate.

I put in a vote on that issue you suggested.

2 Likes
5

Thanks for locating the quote! It's definitely inaccurate and we'll get it changed.

Edit: looks like we already did that in Certbot v2.3.0, I just forgot about it: docs: update -d flag copy to be CA-agnostic by alexzorin · Pull Request #9542 · certbot/certbot · GitHub

Is this for the same set of domains, or do the names change on the certificate? If the names change, so will the sorting order.

I'm not sure if anything changed on Let's Encrypt's side. Some of the relevant code has been refactored recently but I can't tell if there's any real side effects.

Linking the previous and latest certificate (where the CN changed) might help pinpoint things.

5 Likes
6

Thanks for the report! @alexzorin has the analysis right. We did recently refactor this code, and made a slight change.

In general, you can request a specific name in the Subject CN but putting it in the Subject CN of your CSR. That's the supported workflow on the Let's Encrypt side, though it's a bit obscure.

If you don't put anything in your CSR's Subject CN, we pick something from the list of SANs. That used to be the first in the list of SANs provided. Now it's the alphabetically first. Sorry for the surprise there!

Short-term, if it's very important to have a specific name in the Subject CN, you can generate your own CSRs and pass them to Certbot.

Long-term, we and the rest of the industry plan to put nothing in the Subject CN. The Subject CN is not used for validation in any modern systems so it's clearer to leave it empty. I'd be curious to hear what your use case is for needing a specific Subject CN. And in general it would be good to plan for Subject CNs to go away in the not-too-distant future (no timeline announced yet).

11 Likes
7

Thanks for the explanation here. As a result we'll be moving away from using certbot in this case.

I'd be curious to hear what your use case is for needing a specific Subject CN.

We generate Let's Encrypt certificates by environment to avoid excessive Let's Encrypt requests. The certificate we get back would then be made available in AWS ACM as well as via a PEM file and a PKCS12 PFX package. To cross-walk between this generation and our deploy software that would create our AWS load balancers, we would use our reliable subject common name to look up the correct certificate. Since that's now not reliable, I'll be moving away from certbot.

2 Likes
8

To which other ACME client?

Better yet: To which other CA?

3 Likes
9

I can generate certificates via Amazon Certificate Manager and will be doing that for the AWS-based endpoints that were using the certbot generated certificates.

3 Likes
10

That sounds like a reasonable approach! And sorry to hear that this change is bringing you more work. If you wanted another approach, one thing I would consider is using the full set of Subject Alternative Names as the unique identifier for certificates in place of the Subject CN.

6 Likes
11

I considered that as well, but (I forgot to let @_az know this) this list is subject to change, and the lookup tool doesn't currently support lookup by SAN. Thank you though for the suggestions. We'll still be using certbot for other certificate generation and we also use cert-manager in our K8s deploys.

12

It would be nice if that CN field could be repurposed...
With an added control of ALPHA/NUMERIC ONLY [no dots nor dashes], it could serve the "labeling" function.

2 Likes
13

That'd definitely not happening any time soon, as there's no way of guaranteeing old implementations don't trust it. It's much safer to simply retire it. It's possible to add new fields in the future, but in general we want to make certs smaller if possible, and every field should be validated by the CA. It's better to store metadata outside of the certificate.

3 Likes
14

Trusting a name with no dots is not much of a risk.
But a risk none-the-less...
So, I see your pint.

2 Likes
15

This explanation is helpful. However, this is causing issues with my current hosting provider as my Let's Encrypt certificates are no longer accepted for any domain names starting with w, x, y, or z because they are alphabetically after "www". I'm creating certs with the www and non-www versions of the domain name, and my hosting provider requires that the common name be the "www" version. This worked fine up until the change, since I can look back at prior certs and verify that the www version used to be first and now it's not.

I suspect this change is going to cause a lot of extra work for people, including me. Was there a purpose for this change? I don't suppose it can be put back the old way if there wasn't a real purpose for it, to provide time for my hosting provider to update? (I can use a CSR but it's a manual process to generate each CSR, and I'm managing thousands of SSLs so that's not a sustainable option.)

16

Why? What is the point of that?

I'm pretty sure that there are more hosting providers that work with LE certs that those that don't.
I think your hosting provider should revisit their "requirements".

2 Likes
17

I am working with them to see if they can change it, however I'm quite entrenched with them so it's not an easy change for me. I didn't realize it was an issue until about a week ago when it stopped accepting certificates.

18

We made this change incidentally as part of a refactoring, and decided not to roll back at that time because it was a behavior that we never intended to guarantee, and switching it up seemed like a reasonable way to flush out problems that will come in the future, since we hope to gradually shift to issuing with no CN at all. But of course that change will come with a preannouncement and a phase in process to be announced.

We're talking about it some more internally - we'll get back to you shortly with a decision.

4 Likes
19

Thanks for the consideration. I know it's challenging balancing forward progress with existing implementations.

As a suggestion, what about keeping the default behavior as-is and adding a command line flag to set the common name (or using the old behavior of the first domain name being the common name)? This flag could immediately throw a deprecation warning. This would serve the benefit of easing into the change (and raising the issue in the community like this has successfully done with me) and providing a way to take the pressure off unintended side effects like this one has caused my situation. I've raised the issue with my hosting provider but they will likely need some time to prepare and make the change to support the removal of common names.

Again, thank you for the consideration.

1 Like
20

We considered/are considering doing that in this issue.

With that said, I'm a little curious about if @jsha has strong opinions, the different compatibility problems, what other ACME clients do, and what bad things would happen if we started setting the CN by default again though. We don't have to look into all those things, but I think things like that could influence us to just close this as extremely niche and wontfix or consider making setting the CN the default (maybe with a flag to not set it).

All that said, if enough users are coming out of the woodwork with reasonable complaints about being unable to set the CN on the CSR, I think we would be open to making this change.

3 Likes