Validating LE certificate with mbedTLS on an ESP32 device

Hello!

I’ve got some code running on an ESP32 device, that downloads firmware from a server over a secure connection.

When I use the following pair of URL and certificate, all is well:

.url = "https://someserver.us-west-1.amazonaws.com/xxxxx.bin”
.cer_pem = "-----BEGIN CERTIFICATE-----\n"

"MIIDQTCCAimgAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjANBgkqhkiG9w0BAQsF\n"
"ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6\n"
"b24gUm9vdCBDQSAxMB4XDTE1MDUyNjAwMDAwMFoXDTM4MDExNzAwMDAwMFowOTEL\n"
"MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv\n"
"b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj\n"
"ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM\n"
"9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw\n"
"IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6\n"
"VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L\n"
"93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm\n"
"jgSubJrIqg0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMC\n"
"AYYwHQYDVR0OBBYEFIQYzIU07LwMlJQuCFmcx7IQTgoIMA0GCSqGSIb3DQEBCwUA\n"
"A4IBAQCY8jdaQZChGsV2USggNiMOruYou6r4lK5IpDB/G/wkjUu0yKGX9rbxenDI\n"
"U5PMCCjjmCXPI6T53iHTfIUJrU6adTrCC2qJeHZERxhlbI1Bjjt/msv0tadQ1wUs\n"
"N+gDS63pYaACbvXy8MWy7Vu33PqUXHeeE6V/Uq2V8viTO96LXFvKWlJbYK8U90vv\n"
"o/ufQJVtMVT8QtPHRh8jrdkPSHCa2XV4cdFyQzR1bldZwgJcJmApzyMZFo6IQ6XU\n"
"5MsI+yMRQ+hDKXJioaldXgjUkK642M4UwtBV8ob2xJNDd2ZhwLnoQdeXeGADbkpy\n"
"rqXRfboQnoZsG4q5WTP468SQvvG5\n"
"-----END CERTIFICATE-----\n”;

When I switch to this URL and certificate, I get an error':

.url = "https://optimized.realyze.com/xxxx.bin",
.cer_pem = "-----BEGIN CERTIFICATE-----\n"

"MIIFTjCCBDagAwIBAgISA/LYkehAvDsQPYvuHYVuNoaWMA0GCSqGSIb3DQEBCwUA\n"
"MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD\n"
"EwJSMzAeFw0yMjA0MDUxODI0NDdaFw0yMjA3MDQxODI0NDZaMCQxIjAgBgNVBAMT\n"
"GXd3dy5vcHRpbWl6ZWQucmVhbHl6ZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IB\n"
"DwAwggEKAoIBAQDCkNukwqoBPCH8hp4Axx1WoSyk2J/uoCRpE6/71JLUj38w4APE\n"
"dTprv2CD2PS7oH8y1cIjS3ilP7Dezf5bzYkfkcvt6IwaWn77jnsoigO2Oj59xrx4\n"
"s68OtUHpPJf7qzj9g2E8ObHDrKRNv2dFsd1CeJBZrkii7MEswl+VEzb0xLO4TlrS\n"
"qHJvoOBWrfqi661V3eppQRpP8el6EWan5JS68vG4jkeq1kluWpLC674ONxexh1tW\n"
"JxI5vDpB3jpYooedE8DqidbGmxFzx0myXVmeGwpetMSs+WFVHWSr1qccV0VBukAU\n"
"oqcOrwt3zlshI+ZCQI0IYyMQYDkmVXDAKpopAgMBAAGjggJqMIICZjAOBgNVHQ8B\n"
"Af8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB\n"
"/wQCMAAwHQYDVR0OBBYEFF9zKvh0c/JhzofkJ0Y0tkuPqwZSMB8GA1UdIwQYMBaA\n"
"FBQusxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcw\n"
"AYYVaHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMu\n"
"aS5sZW5jci5vcmcvMDsGA1UdEQQ0MDKCFW9wdGltaXplZC5yZWFseXplLmNvbYIZ\n"
"d3d3Lm9wdGltaXplZC5yZWFseXplLmNvbTBMBgNVHSAERTBDMAgGBmeBDAECATA3\n"
"BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNy\n"
"eXB0Lm9yZzCCAQMGCisGAQQB1nkCBAIEgfQEgfEA7wB1AN+lXqtogk8fbK3uuF9O\n"
"PlrqzaISpGpejjsSwCBEXCpzAAABf/svy8UAAAQDAEYwRAIgbd0NtRKP0TvGKdhx\n"
"8fi0Lz8zG82dQW4yPUqbXHq5cfwCIHAHkgCSqbRWBY03ttsmrXmOVSeQNJjkqi3G\n"
"Id9q7zCYAHYAKXm+8J45OSHwVnOfY6V35b5XfZxgCvj5TV0mXCVdx4QAAAF/+y/L\n"
"uAAABAMARzBFAiBXB5ShaoJs+LMONR79kckke1zNYrbpO0bopFOo+ITEiAIhAJJ4\n"
"g/I4jhsdgIUcvM63fs0x0fB7mODUV0FzzrBMjwotMA0GCSqGSIb3DQEBCwUAA4IB\n"
"AQBWupK24uXo2ADlKgTQ0us2U6/5efol2qmHmW8a6mEBLgkgxTvXBHZ1ZWA2+ex7\n"
"IFvP7sIWY33EKqO2T1x3tiblk5ZpQidkXfji6z4HagG1bK/DS9nYm4kYT/lb5AxJ\n"
"HWlB3JPnrEPZxFIsZA6F73pOt+3uQehuAful/ykZr0PvypKz4tRjsvasxDImCEJ5\n"
"NSVqBu3Utqopk9o1+Xjzt80QWq8oKaJ8q6NduU9TdjTy6L0LPHhgvbsFqpIXhoij\n"
"gmzF63gZ7m6BiKt6ujOzWvvpTaXV8c7+Ktf+t30Q2kOVcgpkszuxU2F94un86WqK\n"
"P75UGU5TgbaUVVu99r/yWPFL\n"
"-----END CERTIFICATE-----\n";

The error is:

E (3515) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x2700
E (3516) esp-tls: Failed to open new connection
E (3516) TRANSPORT_BASE: Failed to open a new connection
E (3521) HTTP_CLIENT: Connection failed, sock < 0
E (3524) esp_https_ota: Failed to open HTTP connection: ESP_ERR_HTTP_CONNECT

Googling didn’t find any good clues.

I’ve decoded the certificate and it looks good. I’ve placed that info at the end. One major difference is that I used LetsEncrypt for the latter certificate.
Thanks in advance for any pointers.

Certificate Checks

Check Result
Expiry PASSED - Expires Jul 04 2022 (89 days)
Debian RSA Weak Key PASSED - Does not use a key on our blacklist - this is good
Key Size PASSED (2048 bits)
MD5 PASSED - Not using the MD5 algorithm
SHA1 PASSED - Not using the SHA1 algorithm

Certificate Subject

Common Name (CN) www.optimized.realyze.com

Certificate Properties

Subject CN=www.optimized.realyze.com
Issuer C=US, O=Let's Encrypt, CN=R3
Valid From April 5, 2022, 6:24 p.m.
Valid To July 4, 2022, 6:24 p.m.
Key Size 2048 bits
Key Algorithm RSA

Hi @mzincalli, and welcome to the LE community forum :slight_smile:
[I've moved your post to a separate topic because it really has little to do with that other topic]

The certificate is not the likeliest problem here.
If I had to guess (which I do, since you haven't provided a real functional FQDN) I would have to say that your client lacks TLSv1.2 support.
Being that the failing URL doesn't support TLSv1.0.

2 Likes

The ESP-device might also be lacking the ISRG Root X1 trust anchor.

1 Like

... or you need to add the proper "embedded specific" ciphersuites to your webserver, like CCM for TLS 1.3.

1 Like

This seems to be the key line:

Here's the API definition for that function: ssl.h File Reference - API Documentation - mbed TLS (Previously PolarSSL)

Here's the error code with value -0x2700:

So I suspect this is indeed a certificate problem. @mzincalli can you provide the list of root CAs that you are providing to mbedtls?

4 Likes

I'm only passing that one certificate to this code:

    esp_http_client_config_t config = {
        .url = "https://optimized.realyze.com/xxxx.bin",
        .cert_pem = (char *)optimized_server_certificate,
    };
    esp_err_t ret = esp_https_ota(&config);
    if (ret == ESP_OK) {
        esp_restart();
    } else {
        return ESP_FAIL;
    }

I was under the assumption that since my server domain matches the one in this certificate, that the HTTPS access would succeed.

My, probably incorrect, notion is that when I pass in this certificate, I am basically saying that I trust anything signed from this server, and there is no need for any code on my side to go up the trust chain and try to get to the root CA. If there is malicious actor trying to impersonate that server, then they would have to be able to generate encrypted traffic that I could decrypt with the public key in the cert, which would be unlikely since they don't have the private key that was used to generate the cert that I have.

I know I probably should brush up on SSL, but I knew some of the basic ideas. I'm going to try to figure out where I am wrong.

Where things get more complicated are actually after I get this to work. I expected this to work, as I explained above, but what is unworkable is a certificate that is only valid for 90 days. Since this code is embedded into an IoT device, I'd like it to stay functional for longer than 90 days. And that is where I would need to have a root certificate with a longer validity. However, I first have to overcome this issue.

1 Like

I think that behavior probably varies between TLS client libraries. Some might accept an end-entity (leaf) certificate as a root, meaning "trust this end-entity certificate only." But it's a bit of an incorrect concept, like using a hammer where a wrench is called for. Most client libraries want a list of root certificates. And as you point out, that's important because end-entity certificates expire after 90 days, while root certificates stick around a long time[1].

Can you link to the docs for esp_http_client_config_t, and specifically its cert_pem field?

[1]: Not even root certificates live forever! You should make sure you're thinking about how to deliver root certificate updates to your devices. And don't trust just one root certificate - CAs change roots, go out of business, etc. You should trust a variety of root certificates for flexibility.

4 Likes

Here's the cert_pem:

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/esp_http_client.html?highlight=esp_http_client_config_t#_CPPv4N24esp_http_client_config_t8cert_pemE

And I believe all of this runs on top of Bed TLS:

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/mbedtls.html

One of the reason I brought this up here is that I can easily use the amazon's cert for communication with AWS, but when I try to communicate with my own server, which uses the LE cert, I am having issues. I've also tried the cert provided by this:

openssl s_client -showcerts -connect [optimized.realyze.com:443](http://optimized.realyze.com:443/) < /dev/null

My solution for changing certificates is to provide updates for the device, at least annually. Of course there are users who don't/won't do that, so I may have to have a way to work around that too.

Unfortunately that doesn't really provide enough detail to be useful:

SSL server certification, PEM format as string, if the client requires to verify server

This looks like the documentation you should follow to configure a set of root certificates: ESP x509 Certificate Bundle - ESP32 - — ESP-IDF Programming Guide latest documentation

FYI, it's "mbed TLS." :smiley:

1 Like

So, are you saying that I need a bundle? I have some memory constraints to worry about and I also don't need "90 % coverage according to market share statistics" of all websites; I only want my own server to be covered. How do I do that/what certs do I need to bundle?

Let me be clear:

  • do you think that the certificate I have, like the one I get from the openssl CLI above, should work, and the problem is how I am providing that to mbedtls?
  • or, I need a whole new set of certs (maybe including the one from openssl), because I am lacking enough of the chain of certs?

Sorry for my confusion.

Thank you.

1 Like

There are two sets of certificates you should be thinking about:

  1. The set of end-entity certificates and intermediate certificates served by optimized.realyze.com (the chain)
  2. The set of root certificates you pass to mbedTLS on your devices (the bundle)

(1) looks fine, with a caveat I'll explain below. I think the problem is (2). So yes, you need a bundle of root certificates. It's true that you don't need coverage of all websites, but you do need a backup plan in case Let's Encrypt is hit by a meteor tomorrow and ceases to exist. The most straightforward backup plan is to trust the roots from the Mozilla program, and keep that list up to date. Try that first; if you run into memory constraints you might choose to select a smaller group of roots, but this gets very tricky very fast. There are instructions on how to generate such a bundle on the documentation page I linked.

Here's the caveat for (1): The chain served by optimized.realyze.com includes our cross-sign from the expired DST Root CA X3. That's normal and good and expected. However, some older TLS libraries have issues with it. I think mbedTLS isn't one of them but I'm not sure. This really gets into the nitty-gritty of PKI stuff, and I suspect it isn't your problem, so don't worry about it for now. Just make sure you're running the very latest version of mbedTLS.

2 Likes

I don't think the bundle size should directly affect memory usage.
[it should be a disk storage issue (if any)]

2 Likes

I'm pretty sure ESP devices don't have "disks", but do have flash memory as there storage device :wink:

1 Like

Yes, in this case the memory I was referring to is Flash. And I have to preserve it so that there is enough room leftover to allow for over-the-air updates, etc. The total flash size is 4MB in my case, and the firmware size right now is over 1900KB.

2 Likes

Luckily, a certificate isn't that big, just 1391 bytes for the DER format of ISRG Root X1 and 1939 bytes for the PEM format of ISRG Root X1.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.