MQTT SSL certificate expired

pengle · October 14, 2021, 1:19am

My domain is: broker.avasmartgardens.com

I ran this command:
openssl s_client -connect broker.avasmartgardens.com:8883 -servername broker.avasmartgardens.com

It produced this output:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = broker.avasmartgardens.com
verify return:1
---
Certificate chain
 0 s:CN = broker.avasmartgardens.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFOjCCBCKgAwIBAgISBCu/fGiX2BfPhaHFA7j6V0wsMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEwMTMxODM0NTNaFw0yMjAxMTExODM0NTJaMCUxIzAhBgNVBAMT
GmJyb2tlci5hdmFzbWFydGdhcmRlbnMuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAlvmh+OAB0Xh3gMY+4m4FmsU03o2T2bzv2Pk+k0a3Q3+fbEpD
P/t3S4kjbbE5zardS6RjSMuFVVNgkklRvC1FkPf9C2oOHAWcklTeF9pjcFHVFtog
LEBu8c8j0AWkPhfokZVU93oBcDqGe0mfgx7EUYW22ESiW6n9sO5QSDjd/2UaItCz
nxl95JnFjO0GV67DUy05nBPNTWExiMfcH7Q8RtV0L7+iZ44Q3Z6Vz0sjLOhxLZno
rkBIt1KGAlGEyTCIvl6MdJVV2Egc4qNyc8jQUrqiVG/n0b/T3fmK8/MppaoBd7lO
f+JRiRRmvim7g10mNV0wi2rXrJ1oktrkdfQ+KwIDAQABo4ICVTCCAlEwDgYDVR0P
AQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMB
Af8EAjAAMB0GA1UdDgQWBBSNDX01YbJZnEZp9mgZOJ/Ffd0i7jAfBgNVHSMEGDAW
gBQULrMXt1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUH
MAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3Iz
LmkubGVuY3Iub3JnLzAlBgNVHREEHjAcghpicm9rZXIuYXZhc21hcnRnYXJkZW5z
LmNvbTBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsG
AQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkC
BAIEgfUEgfIA8AB2AN+lXqtogk8fbK3uuF9OPlrqzaISpGpejjsSwCBEXCpzAAAB
fHsmidkAAAQDAEcwRQIgcYfsnIbrxNMnVvMw4PhFQlMQA0qz7zZseKFqDCBgH1cC
IQDbvUGv0pwOTjA+9yXvYCDvPSSYLJGgh8askm2NLGgxwAB2AEalVet1+pEgMLWi
iWn0830RLEF0vv1JuIWr8vxw/m1HAAABfHsmif8AAAQDAEcwRQIgIgR4UyQjSuLY
OxAZcS04XoIV7oYxetHS6s0pwMrne7UCIQCQfZs5g3yu3k0RFuYrQ9fv6SOxqZwR
8joy+cHqQX4P0TANBgkqhkiG9w0BAQsFAAOCAQEAAI0oMy7wOJ5x58Qol/XUqzig
R9DjAreIch9vdLc7bCoLPwItR8O1tcigHJJre2dMdvyoQ1hXoyCjM/DpURmgd0e0
0vHoyu0Z4QYPEold8ZbnaLFWpenFHvliWp2AjRFytzXKLrp0NvwoJq99WILXWo1X
rlIFXeWMeccY5+P0SCIpEvY9Sadiln+sBfkdo7bBty9/2gggCkMVccw2PRjqs8Wg
X9N1RwBI7Yyw4ESVsrA5SyvwY9miU33h2yBstCN15gKZ1i18qmQ3zopFWnVhBcZn
CLwDrXr32W19JbnRRwqTgu3Hx3whvg1Jg4LBGjd3PEleQn3axs3CUkKdedm0vA==
-----END CERTIFICATE-----
subject=CN = broker.avasmartgardens.com

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4720 bytes and written 444 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 59D8F8827A405D5520283F2F92A6B05CCB295BD15E7CFC1F74F2C128B03C551A
    Session-ID-ctx:
    Master-Key: 2F477317B348CC123E20E35A9809A3A2C54AF54053C5B54A7AE6EBDDF58DA60A957E934EEB0E7491437AD28979CEFBD1
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 00 40 87 85 cd 61 87 fa-14 81 2a 87 7d 79 d6 58   .@...a....*.}y.X
    0010 - 55 2c b8 67 f7 7a 52 89-bf 6e 3b 69 97 72 10 c3   U,.g.zR..n;i.r..
    0020 - ca 16 bd 78 69 6a 17 49-53 e2 35 d1 cd 33 0e 2b   ...xij.IS.5..3.+
    0030 - db 39 42 15 23 32 48 23-27 d0 48 03 26 ee eb 15   .9B.#2H#'.H.&...
    0040 - 24 cd ec 83 e6 cb 5f f4-8a 0e ba c9 61 bc f6 c2   $....._.....a...
    0050 - b0 2c 72 fb 18 be 29 60-79 22 68 44 93 0d b8 49   .,r...)`y"hD...I
    0060 - c9 68 60 d9 e8 18 ec 5b-99 ee b3 7f 7b 03 da c5   .h`....[....{...
    0070 - 07 d6 73 65 b3 bf 32 ae-01 23 56 44 04 95 d5 15   ..se..2..#VD....
    0080 - 56 68 78 50 98 17 d8 08-36 cb 76 74 c3 b1 d9 b4   VhxP....6.vt....
    0090 - 27 5b 0d 89 b9 1c 23 5e-d4 39 c2 93 e0 4b ca 70   '[....#^.9...K.p
    00a0 - 0b 9d ce c5 48 56 9b 72-6b 0b 97 06 f3 5b 8a fc   ....HV.rk....[..
    00b0 - ce 2b 3d 0f 42 9a ab 68-1d 39 61 47 e6 c8 24 44   .+=.B..h.9aG..$D

    Start Time: 1634171952
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
closed

The operating system my web server runs on is (include version): Ubuntu 16.04

My hosting provider, if applicable, is: AWS EC2

I can login to a root shell on my machine: yes

I'm using a control panel to manage my site: no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.33

Hello, we are running an mosquitto MQTT broker service at broker.avasmartgardens.com running on port 8883 that was setup by a dev that is no longer with us. It was recently brought to our attention that communication went down and it was due to the DST Root CA X3 cert expiring. I've looked in our mosquitto config file and have found the .pem files that it is using for tls. I've attempted to renew the certificates using certbot, but am still getting the same errors regarding the certificate being expired.

I've attempted to remove the last certificate in the fullchain.pem file that is referenced by the mosquitto conf per this: post but when running the same above command I get an unknown ca error.

The openssl version running on the server is 1.0.2g-1ubuntu4.20

Any ideas of what I can try to do to fix this issue?

Thank you

Error logs from the mosquitto service:

1634172930: New connection from 73.53.67.25 on port 8883.
1634172930: OpenSSL Error: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
1634172930: OpenSSL Error: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure

Resources:

rg305 · October 14, 2021, 2:19am

Hi @pengle and welcome to the LE community forum

As a "quick fix" (/"test"), you could find the fullchain.pem (or chain.pem) file used and remove the last cert from within it.
If that does the trick, you may have to upgrade certbot to version 1.12 (or higher) [OR switch to another ACME client that supports the --perferred-chain parameter] in order to have that edit survive a renewal.
OR play some human tricks to makeshift a shorter chain (which will eventually break - when the chain is changed).

Unrelated but also related (LOL)...
You might want to update the ca-certificates and openssl.
See: Old Let’s Encrypt Root Certificate Expiration and OpenSSL 1.0.2 - OpenSSL Blog

pengle · October 14, 2021, 4:37am

Hi Rudy, thank you for your reply, I tried removing the last cert from fullchain.pem again but get (in the mosquitto logs):

1634186027: Socket error on client <unknown>, disconnecting.
1634186027: New connection from xxx.xxx.xxx.xxx on port 8883.
1634186027: OpenSSL Error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
1634186027: OpenSSL Error: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure

and when I try using: openssl s_client -connect broker.avasmartgardens.com:8883 -servername broker.avasmartgardens.com I get:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = broker.avasmartgardens.com
verify return:1
---
Certificate chain
 0 s:CN = broker.avasmartgardens.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFOjCCBCKgAwIBAgISBCu/fGiX2BfPhaHFA7j6V0wsMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEwMTMxODM0NTNaFw0yMjAxMTExODM0NTJaMCUxIzAhBgNVBAMT
GmJyb2tlci5hdmFzbWFydGdhcmRlbnMuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAlvmh+OAB0Xh3gMY+4m4FmsU03o2T2bzv2Pk+k0a3Q3+fbEpD
P/t3S4kjbbE5zardS6RjSMuFVVNgkklRvC1FkPf9C2oOHAWcklTeF9pjcFHVFtog
LEBu8c8j0AWkPhfokZVU93oBcDqGe0mfgx7EUYW22ESiW6n9sO5QSDjd/2UaItCz
nxl95JnFjO0GV67DUy05nBPNTWExiMfcH7Q8RtV0L7+iZ44Q3Z6Vz0sjLOhxLZno
rkBIt1KGAlGEyTCIvl6MdJVV2Egc4qNyc8jQUrqiVG/n0b/T3fmK8/MppaoBd7lO
f+JRiRRmvim7g10mNV0wi2rXrJ1oktrkdfQ+KwIDAQABo4ICVTCCAlEwDgYDVR0P
AQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMB
Af8EAjAAMB0GA1UdDgQWBBSNDX01YbJZnEZp9mgZOJ/Ffd0i7jAfBgNVHSMEGDAW
gBQULrMXt1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUH
MAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3Iz
LmkubGVuY3Iub3JnLzAlBgNVHREEHjAcghpicm9rZXIuYXZhc21hcnRnYXJkZW5z
LmNvbTBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsG
AQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkC
BAIEgfUEgfIA8AB2AN+lXqtogk8fbK3uuF9OPlrqzaISpGpejjsSwCBEXCpzAAAB
fHsmidkAAAQDAEcwRQIgcYfsnIbrxNMnVvMw4PhFQlMQA0qz7zZseKFqDCBgH1cC
IQDbvUGv0pwOTjA+9yXvYCDvPSSYLJGgh8askm2NLGgxwAB2AEalVet1+pEgMLWi
iWn0830RLEF0vv1JuIWr8vxw/m1HAAABfHsmif8AAAQDAEcwRQIgIgR4UyQjSuLY
OxAZcS04XoIV7oYxetHS6s0pwMrne7UCIQCQfZs5g3yu3k0RFuYrQ9fv6SOxqZwR
8joy+cHqQX4P0TANBgkqhkiG9w0BAQsFAAOCAQEAAI0oMy7wOJ5x58Qol/XUqzig
R9DjAreIch9vdLc7bCoLPwItR8O1tcigHJJre2dMdvyoQ1hXoyCjM/DpURmgd0e0
0vHoyu0Z4QYPEold8ZbnaLFWpenFHvliWp2AjRFytzXKLrp0NvwoJq99WILXWo1X
rlIFXeWMeccY5+P0SCIpEvY9Sadiln+sBfkdo7bBty9/2gggCkMVccw2PRjqs8Wg
X9N1RwBI7Yyw4ESVsrA5SyvwY9miU33h2yBstCN15gKZ1i18qmQ3zopFWnVhBcZn
CLwDrXr32W19JbnRRwqTgu3Hx3whvg1Jg4LBGjd3PEleQn3axs3CUkKdedm0vA==
-----END CERTIFICATE-----
subject=CN = broker.avasmartgardens.com

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3337 bytes and written 444 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: FC4E7FAECEDFAE81B32CFD93F36C9A91157E082EEE941D7ED3F6EF7446CB30E5
    Session-ID-ctx:
    Master-Key: 22737277F3C2971C78227C226A508675029AE97AAA8878989332702F2E9610BAFDCB198C16078848E48595044064F2FE
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - b8 ee ec 0c 55 44 a0 74-f2 b0 55 b3 5c e6 db b5   ....UD.t..U.\...
    0010 - 48 8f dc ab af 4f 0d 01-a0 da f5 4a 04 df c9 52   H....O.....J...R
    0020 - 85 8c 45 10 3a ed 00 00-3f 64 7a 83 02 78 ef 42   ..E.:...?dz..x.B
    0030 - 38 e6 f2 78 18 4e 81 61-04 ad 3c 89 e7 df f0 b3   8..x.N.a..<.....
    0040 - 84 85 db ff 63 41 92 30-b2 a2 70 fe c4 39 87 90   ....cA.0..p..9..
    0050 - 85 88 d3 d4 f0 6b fb bb-61 f6 81 65 63 85 81 94   .....k..a..ec...
    0060 - b8 7d 9c cb 44 36 eb df-e8 54 68 6e 4e 01 7a 8e   .}..D6...ThnN.z.
    0070 - dc 68 68 fd ea 94 a9 c0-ef b6 59 f2 89 01 f2 cf   .hh.......Y.....
    0080 - b6 14 d7 c3 b3 a8 a4 7d-68 46 88 22 a7 a5 c1 53   .......}hF."...S
    0090 - f5 8f df 62 25 9c 7e 42-18 d5 71 aa 32 46 2f 55   ...b%.~B..q.2F/U
    00a0 - bc 36 9b b8 8f 30 a4 4c-6f c0 95 0c f8 2f d8 26   .6...0.Lo..../.&
    00b0 - b5 a5 b1 f9 b3 c1 a0 8c-94 b9 69 bd 31 ac 73 2e   ..........i.1.s.

    Start Time: 1634186022
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
closed

I've upgrade certbot to version 1.20.0. Unfortunately it looks like I have been rate limited for the time being as I've tried to renew the certificates too many times

In regards to your other point, I've updated openssl to the latest version supported for Ubuntu 16.04 which is 1.02g-1ubuntu4.20

rg305 · October 14, 2021, 4:55am

Mosquitto doesn't like the cert:

OpenSSL has no problems with "ISRG Root X1":

You need to find out where/how Mosquitto validates the cert.

Please show the file:
mosquitto.conf

rg305 · October 14, 2021, 4:59am

You don't need to renew the cert.
You need to update the trusted root store (that Mosquitto uses).
[and likely add the "ISRG Root X1" cert into it]

pengle · October 14, 2021, 5:30am

# mosquitto.conf

# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

# conf.d/default.conf

allow_anonymous false
password_file /etc/mosquitto/passwd

listener 1883 localhost

listener 8883
certfile /etc/letsencrypt/live/broker.avasmartgardens.com/cert.pem
cafile /etc/letsencrypt/live/broker.avasmartgardens.com/chain.pem
keyfile /etc/letsencrypt/live/broker.avasmartgardens.com/privkey.pem

listener 8083
protocol websockets
certfile /etc/letsencrypt/live/broker.avasmartgardens.com/cert.pem
cafile /etc/letsencrypt/live/broker.avasmartgardens.com/chain.pem
keyfile /etc/letsencrypt/live/broker.avasmartgardens.com/privkey.pem

I've tried changing chain.pem to fullchain.pem with no luck

rg305 · October 14, 2021, 5:46am

We need to change that...
Try:

certfile /etc/letsencrypt/live/broker.avasmartgardens.com/fullchain.pem
cafile   /etc/ssl/certs/ca-certificates.crt

[thisshoulddothetrick]

pengle · October 14, 2021, 5:57am

First, I would like to say thank you for your time in helping me with this issue

Unfortunately, I'm still seeing the same in the mosquitto logs after changing the certfile and cafile entries in the default.conf file:

1634190821: New connection from xxx.xxx.xxx.xxx on port 8883.
1634190822: OpenSSL Error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
1634190822: OpenSSL Error: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure

I think for mosquitto the cafile has to be PEM encoded, should I try changing the ca-certificates.crt file to .pem?

Check the Certificate based SSL/TLS support section to see what I am talking about:

rg305 · October 14, 2021, 6:01am

No.
If you "look" at that file you should see that all the certs in it are already in PEM format.
head -n 44 /etc/ssl/certs/ca-certificates.crt

rg305 · October 14, 2021, 6:06am

Add after:
cafile /etc/ssl/certs/ca-certificates.crt
This line:
capath /etc/ssl/certs/

pengle · October 14, 2021, 6:46am

Same thing:

1634193715: New connection from xxx.xxx.xxx.xxx on port 8883.
1634193715: OpenSSL Error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
1634193715: OpenSSL Error: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure

pengle · October 14, 2021, 8:37am

Workaround 3 (on the servers that 1.0.2 clients connect to)

Configure the server to use the alternative certificate chain which can be requested from Let’s Encrypt with most up-to-date ACME protocol clients. This chain does not contain the ISRG Root X1 cross-signed by the soon to be expired DST Root CA X3 and thus any OpenSSL 1.0.2 clients will not be misled by this expired path.

The downside is that the servers will be seen as using an untrusted root certificate by some older Android clients because these clients do not contain the self-signed ISRG Root X1 certificate in their trust stores.

From the article you linked. I think this would be the next step since nothing else seems to have worked. I would have to request an alternate certificate chain using the updated version of certbot and the preferred-chain="ISRG Root X1" option to get a new certificate, correct? I would obviously have to wait for my timeout to end.

rg305 · October 14, 2021, 12:03pm

That is worth the effort.

pengle · October 20, 2021, 8:02pm

Found that the issue was not with the server but with the clients themselves, I've gone ahead and updated the trust store on the client to include ISRG Root X1 certificate and now the devices can successfully communicate with the server.

Thank you for the help @rg305

system · November 19, 2021, 8:02pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.