Can not issue X3 chain certificate even with the --preferred-chain "DST Root CA X3" option

meddle0x53 · December 23, 2020, 11:11am

I ran this command:

certbot certonly -n --agree-tos --email inflowsys@inflowmatix.com --dns-route53 -d *.resilient-network.com --preferred-ch
ain "DST Root CA X3" --expand --config-dir config --work-dir work --logs-dir log --force-renewal

It produced this output:

Saving debug log to /home/admin/venv/le/log/letsencrypt.log
Found credentials in shared credentials file: ~/.aws/credentials
Plugins selected: Authenticator dns-route53, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for resilient-network.com
Waiting for verification...
Cleaning up challenges
Non-standard path(s), might not work with crontab installed by your operating system package manager

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /home/admin/venv/le/config/live/resilient-network.com/fullchain.pem
   Your key file has been saved at:
   /home/admin/venv/le/config/live/resilient-network.com/privkey.pem
   Your cert will expire on 2021-03-23. To obtain a new or tweaked
   version of this certificate in the future, simply run
   letsencrypt-auto again. To non-interactively renew *all* of your
   certificates, run "letsencrypt-auto renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

My web server is (include version):

HAProxy Load Balancer 1.7.5-2

The operating system my web server runs on is (include version):

Debian 9.2

My hosting provider, if applicable, is:

AWS

I can login to a root shell on my machine (yes or no, or I don't know):

I can.

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

No, I'm the administrator of the servers.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot --version
certbot 1.8.0

Hi,

I'm trying to renew the certificate with certbot 1.8.0

certbot certonly -n --agree-tos --email inflowsys@inflowmatix.com --dns-route53 -d *.resilient-network.com --preferred-ch
ain "DST Root CA X3" --expand --config-dir config --work-dir work --logs-dir log --force-renewal

I'm installing it myself to the HAProxy server, so that's not an issue, but the certificates themselves show as R3, not X3 (which I specified). I tried different things, I tried with "renew" too:

certbot renew --text --no-self-upgrade --deploy-hook /home/admin/venv/le/push_renewed_cert.sh \
          --config-dir config --work-dir work --logs-dir log --preferred-chain "DST Root CA X3"

But it is always the R3 type of certificates.

Can you help me find out what the issue could be?

JuergenAuer · December 23, 2020, 11:24am

Hi @meddle0x53

to check your config, a subdomain name to test is required. www doesn't work.

What's

the content of that file?

meddle0x53 · December 23, 2020, 11:30am

I was working with a subdomain but hit the limit of renewals so moved to "*.", the subdomain was "gateway.resilient-network.com" before that.

JuergenAuer · December 23, 2020, 11:39am

That's

meddle0x53:

-----BEGIN CERTIFICATE-----
MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
/xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
-----END CERTIFICATE-----

the DST Root signed R3 Intermediate certificate.

So all is ok. You can't get a Not-R3 certificate. That

has worked.

PS: If you use Windows: Save the content in a .crt file, open that file - then you see it.

meddle0x53 · December 23, 2020, 11:45am

Then it is bad, as we are using IOT devices that can work only with Let's Encrypt Authority X3 or Let's Encrypt Authority X4 chains. What can we do to issue such a certificate now?

Also I'm on linux and am using
openssl x509 -in <file_path> -inform PEM -text

I can see that it was issued from the DST Root CA X3, but the certificate itself is unusable for our IOT devices as it shows as R3.

Osiris · December 23, 2020, 12:10pm

If that really is the case, then it was a bad design choice. Intermediate certificates should never be hardcoded or pinned, never.

If your previous certificate is still valid for a few days (which should be the case if you adhered to the recommendation of Let's Encrypt to renew after 60 days), you might be able to restore that certificate, so you have a few days left (until the cert expires) to fix your IOT device.

hebbet · December 23, 2020, 12:12pm

The Let's Encrypt Authority X3 is no longer used for new certs.

Relying on intermediates is a very bad desicion as they can change at any time.

You should use the root ca and maybe include severals and not just LE.

Btw. Let's Encrypt Authority X3 will expire in march 2021

meddle0x53 · December 23, 2020, 12:14pm

Unfortunately because of a recent change the auto-renew logic never worked properly for a few months and I'm noticing that just now, so we don't have a backup certificate.

We thought this will work until March so we didn't update the IOT devices on time, yeah - bad.

My question is - is there a way, or can I contact somebody who can issue an X3 certificate for 30 days or so? I understand that bad management of the devices and the servers (and very bad luck with this auto-renew logic) lead to this, but we can't do much?

JuergenAuer · December 23, 2020, 12:16pm

There is no longer a X3 certificate support.

You have to change your (bad) design.

Only R3 signed certificates are possible.

Or buy a certificate from another CA.

Osiris · December 23, 2020, 12:19pm

Not that I know of. I'm not a Let's Encrypt employee, but I expect that the hardware security modules used to sign certificates aren't even configured for the X3 intermediate any longer (note: I don't know this for a fact at all).

We can ask @lestaff

meddle0x53 · December 23, 2020, 12:31pm

Thanks @Osiris and @JuergenAuer

So @lestaff can you help us. We are really in a bad situation and the mixture of bad luck (the auto-renew script breaking because we added a new domain) and bad design lead to a very bad situation (in a very bad time - holidays). If there is a way to issue an X3 intermediate for gateway.resilient-network.com can you help us?

Or maybe tell me from where we could buy one?

dokie · December 23, 2020, 1:06pm

Hi there

I am the CEO of the Company that @meddle0x53 works for and if we cannot resolve the issue with some kind help from Let's Encrypt, @lestaff we could be in serious danger of losing lots of our business. Even if this was our error or bad choice of design, I want to appeal, business to business for some help please.

Regards
Mike

petercooperjr · December 23, 2020, 1:44pm

What "business" do you think Let's Encrypt would lose? They're a non-profit offering certificates for free. While I'm sure they'll be happy to help if they can, I don't think that there's much they'll be able to do. Since the X3 intermediate (cross-signed with DST Root X3 version) expires in March it'd be tricky at best to issue a 90-day-certificate from it at all a this point, and that's assuming that the key is still available somewhere to do so.

There's some advice in this thread about what certificates to put in an embedded device trust store, including that you definitely want more than one CA in there (possibly one you create yourself) just in case one CA isn't available or has a problem.

But if you only put Let's Encrypt Authority X3 in your trust store, with no plan for when it expired, I'm really not sure what else you were expecting? The suggestion to get one from another CA would be if you also had a root certificate from some other CA in there, you could use that.

dokie · December 23, 2020, 1:46pm

Sorry, I meant We, not LE are in danger. We would only need a cert for 30 or 60 days that our devices can use and we can update them over the air to resolve correctly as you have suggested on the thread.

dokie · December 23, 2020, 2:42pm

Sorry to ask this, but is there any way that our expired certificate can be extended by some number of days to allow us to roll out a fix to our devices?

Thanks in advance
Mike

JuergenAuer · December 23, 2020, 3:09pm

Certificates are always read-only.

Nobody can extend an existing certificate, that would break the whole certificate system.

You have to fix that bug, that's all.

dokie · December 23, 2020, 3:13pm

@JuergenAuer Thanks for the response. The trap we are in is that to fix the bug is code we can write, BUT, to deliver the updated code to the devices we need them to communicate to our servers which is where the expired certificate is and without having to "visit" all the devices (not possible) we cannot solve it that way. Unless I have misunderstood your reply, and if so, apologies.

Mike

JuergenAuer · December 23, 2020, 3:24pm

Then you have created a really fatal bug.

As already written: If the client would accept another CA, buy a certificate.

In theory, you can change the time of server and clients. But I don't know if that would really work.

Or you have to update every client manual with code that ignores expiration errors.

josh · December 23, 2020, 4:24pm

Hi @dokie and @meddle0x53,

We talked this over and we just don't have a way to help here. The issue is that the cross-sign for X3 expires in less than 90 day. That wouldn't technically prevent us from issuing from X3 but most of our subscribers do not want a certificate with a lifetime that outlives the intermediate and we have no way to issue from X3 specifically to you.

Is there any chance that your devices would accept certificates from IdenTrust because they trust the cross-sign? If so, IdenTrust may be able to help.

dokie · December 23, 2020, 4:37pm

Hi @josh

Unfortunately our modem in the device only accepts the Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 intermediate cert. (our bad design we know). We really need a one off special generation of our certificate with this intermediate cert to get us out of this hole. This special request certificate does not have to last until March but as long as we have around 45 days from now we can patch our devices over the air with the fix.

We are helping a large number of global water companies monitor their drinking water networks and with these devices "dark" they will may miss vital events that can be critical to their customers.

We would be willing to make a financial contribution to you guys for this.

I hope you can help us

Regards
Mike
CEO Inflowmatix Limited

I can be reached directly on mike at inflowmatix dot com if that helps.

Topic		Replies	Views
Is the preferred-chain 'DST X3 Chain' option live? Help	5	2098	January 20, 2021
DST Root CA X3 Expiration Help	16	2711	December 12, 2021
Cert-manager+LE giving unwanted DST X3 chain after Feb 8 Issuance Tech	8	817	March 20, 2024
--preferred-chain not taking effect Help	36	10213	December 17, 2021
DST_Root_CA_X3.pem no longer exists Help	4	1044	October 27, 2021

Can not issue X3 chain certificate even with the --preferred-chain "DST Root CA X3" option

Related topics