New server renewal certificate denies connection to server

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: www.feeder-bot.com

I ran this command:
The certificate on the IIS server renewed automatically as it has done for several years every 3 months. This time clients with certificates that are supposed to expire next year can no longer connect over SSL. Updating the client certificate to the new one fixes the problem, but the clients need to connect in to update their certificates. Possibly need a compatible certificate on the server to allow the old one in or some other solution such as allowing all certificate authentication requests to be valid on the server. Any direction to temporarily allow the clients in with the old cert would be most helpful.

Old Client .pem

-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

New Client .pem

-----BEGIN CERTIFICATE-----
MIIFBjCCAu6gAwIBAgIRAIp9PhPWLzDvI4a9KQdrNPgwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjQwMzEzMDAwMDAw
WhcNMjcwMzEyMjM1OTU5WjAzMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDEMMAoGA1UEAxMDUjExMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAuoe8XBsAOcvKCs3UZxD5ATylTqVhyybKUvsVAbe5KPUoHu0nsyQYOWcJ
DAjs4DqwO3cOvfPlOVRBDE6uQdaZdN5R2+97/1i9qLcT9t4x1fJyyXJqC4N0lZxG
AGQUmfOx2SLZzaiSqhwmej/+71gFewiVgdtxD4774zEJuwm+UE1fj5F2PVqdnoPy
6cRms+EGZkNIGIBloDcYmpuEMpexsr3E+BUAnSeI++JjF5ZsmydnS8TbKF5pwnnw
SVzgJFDhxLyhBax7QG0AtMJBP6dYuC/FXJuluwme8f7rsIU5/agK70XEeOtlKsLP
Xzze41xNG/cLJyuqC0J3U095ah2H2QIDAQABo4H4MIH1MA4GA1UdDwEB/wQEAwIB
hjAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwEgYDVR0TAQH/BAgwBgEB
/wIBADAdBgNVHQ4EFgQUxc9GpOr0w8B6bJXELbBeki8m47kwHwYDVR0jBBgwFoAU
ebRZ5nu25eQBc4AIiMgaWPbpm24wMgYIKwYBBQUHAQEEJjAkMCIGCCsGAQUFBzAC
hhZodHRwOi8veDEuaS5sZW5jci5vcmcvMBMGA1UdIAQMMAowCAYGZ4EMAQIBMCcG
A1UdHwQgMB4wHKAaoBiGFmh0dHA6Ly94MS5jLmxlbmNyLm9yZy8wDQYJKoZIhvcN
AQELBQADggIBAE7iiV0KAxyQOND1H/lxXPjDj7I3iHpvsCUf7b632IYGjukJhM1y
v4Hz/MrPU0jtvfZpQtSlET41yBOykh0FX+ou1Nj4ScOt9ZmWnO8m2OG0JAtIIE38
01S0qcYhyOE2G/93ZCkXufBL713qzXnQv5C/viOykNpKqUgxdKlEC+Hi9i2DcaR1
e9KUwQUZRhy5j/PEdEglKg3l9dtD4tuTm7kZtB8v32oOjzHTYw+7KdzdZiw/sBtn
UfhBPORNuay4pJxmY/WrhSMdzFO2q3Gu3MUBcdo27goYKjL9CTF8j/Zz55yctUoV
aneCWs/ajUX+HypkBTA+c8LGDLnWO2NKq0YD/pnARkAnYGPfUDoHR9gVSp/qRx+Z
WghiDLZsMwhN1zjtSC0uBWiugF3vTNzYIEFfaPG7Ws3jDrAMMYebQ95JQ+HIBD/R
PBuHRTBpqKlyDnkSHDHYPiNX3adPoPAcgdF3H2/W0rmoswMWgTlLn1Wu0mrks7/q
pdWfS6PJ1jty80r2VKsM/Dj3YIDfbjXKdaFU5C+8bhfJGqU3taKauuz0wHVGT3eo
6FlWkWYtbt4pgdamlwVeZEW+LM7qZEJEsMNPrfC03APKmZsJgpWCDWOKZvkZcvjV
uYkQ4omYCTX5ohy+knMjdOmdH9c7SpqEWBDC86fiNex+O0XOMEZSa8DA
-----END CERTIFICATE-----

It produced this output:

My web server is (include version): MS IIS 10.0

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): Login to server directly

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

What have client certificates to do with the Let's Encrypt certificate? And why are you using the Let's Encrypt intermediates as "client.pem"? I'm very confused what you're doing.

Usually one uses their own private root and intermediates for client authentication. Or is that not what you mean by "client certificates"?

4 Likes

Maybe I am mixing terminology.

On Sunday the 23rd, the server certificates auto renewed like they do every 3 months.

The clients are devices that are connecting to the webserver using TLS. The old client.pem no longer authenticates although it is not expired.

The devices using the new client.pem connect just fine. This is found using:
openssl s_client -showcerts -connect https://www.feeder-bot.com:443

The devices in the field need to connect into the server using the old client.pem to update to the new client.pem. This was to be scheduled sometime in the next year.

What can I do to temporarily allow the old unexpired client.pem to continue to authenticate?

server rejects client, or client reject server's certificate?

3 Likes

Client receives message "Failed to verify Peer Certificate". I believe the server is rejecting the client.

what will happen if you disable client certificate at all from server side?

3 Likes

Not sure. We can possibly try that later. My thought is that the client will be looking for an authentication response from the server and won't get it, so the client will then be rejecting the server.

that'll be different error: do you have toy env to that that without production impact? looks like you have a client locally able to patch as needed

2 Likes

System is basically down at this point, so can make changes in production environment. Have a client locally that I can patch.

looks like you binded client CA certificate and server certificate's issuer together:
only configs I think you are useing **IIS Client Certificate Mapping authentication
I think it's much more likely clients doesn't trust r11 in their config: do you have list of what ssl lib client uses?

4 Likes

Client is an esp32 using mbed_tls. Not sure about what list of ssl lib you are looking for. Pretty sure problem is not with client as changing the .pem file to new one allows it to connect just fine.

The bindings you are talking about are on the server? That was auto renewed/generated all by itself.

nah now I'm sure that's client's trust store for verify server certificate against, if it were client certificate esp32 must have it's private key, which you wouldn't have for any of LE's intermediates

3 Likes

roll back your server to use old certificate (that signed by R3): and deploy update to use https://letsencrypt.org/certs/isrgrootx1.pem as client.pem (but actually ca-cert)
I would append few more root certificate there, like your own CA certificate as fallback and some other free CAs

4 Likes

Thanks for the help, orangepizza. I will do that. Other changes have been made to disassociate the .pem from the firmware.

What is client.pem? That's not a typical certificate filename I'm familiar with. How is it used on the clients?

I'm a little bit worried you have configured your ESP32 devices to trust the intermediate certificates of Let's Encrypt. THAT IS AN ENORMOUS ERROR!

You should NEVER trust intermediate certificates, as they can and will be replaced regularly or spontaneous. You should rely on the root certificate(s), in case of Let's Encrypt ISRG Root X1 and/or ISRG Root X2.

And always, ALWAYS include a backup CA, as CAs can suddenly go out of business. You can see an overview of free ACME CAs here: ACME CA Comparison - Posh-ACME.

3 Likes

Hi Osirus,
The client.pem is a text file of the certificate in PEM format. It is an easy way to incorporate the certificate onto the ESP32. It can be embedded into the firmware at compile time or as a separate file that can be stored in the file system and updated as needed without re-flashing the firmware.

Thank you for your info and concern. The root certificates will be used in the future and backup CAs included, as well.

-Greg

1 Like

I would have guessed it was a certificate indeed (based on the contents of the file earlier) and not a key or something, but what's its purpose?

2 Likes

The ESP32 doesn't natively download a bunch of trusted CAs, unlike a web browser. You have to provide the certificates manually to keep memory usage down. In this case, the devices connect to only one web server, so only need one certificate housed in the client.pem file to connect to the server through TLS.

Ok, that's clear. So the problem was that someone made the incorrect decision to use an intermediate certificate instead of a root certificate.

I assume mbed_tls can do chain validation?

3 Likes

Yes, mbed_tls can do chain validation.