Certbot generated certificates work on IP but not on domain

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mqttbtest.caredpal.com

I ran this command:

sudo certbot certonly --manual --preferred-challenges dns –d mqttbtest.caredpal.com

It produced this output:

It produced a TXT record and passed the validation and finally showed the following:
IMPORTANT NOTES:

  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/mqttbtest.caredpal.com/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/mqttbtest.caredpal.com/privkey.pem
    Your certificate will expire on 2024-12-15. To obtain a new or
    tweaked version of this certificate in the future, simply run
    certbot again. To non-interactively renew all of your
    certificates, run "certbot renew"

  • If you like Certbot, please consider supporting our work by:

    Donating to ISRG / Let's Encrypt: Additional Donation Information - Let's Encrypt
    Donating to EFF: Support EFF's Work on Let's Encrypt | Electronic Frontier Foundation

My web server is (include version):

Mosquitto version 1.6.10, an MQTT v3.1.1 broker.

Certbot generated SSL certificates are used in mosquitto.conf:

listener 8883
certfile /etc/letsencrypt/live/mqttbtest.caredpal.com/cert.pem
cafile /etc/letsencrypt/live/mqttbtest.caredpal.com/chain.pem
keyfile /etc/letsencrypt/live/mqttbtest.caredpal.com/privkey.pem

Problem:

I can ping mqttbtest.caredpal.com without any problem and get the ip associated with the domain is 8.154.38.204.

I can use ssl://8.154.38.204:8883 to connect to the mosquitto server without any problem.

But when I useed ssl://mqttbtest.caredpal.com:8883 to connect to the mosquitto I encountered the following issue:

[2024-09-17 11:52:01] [ERROR] Connection for MQTTBTest_SSL failed, MQTT.js onError trigger, Error: Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
[2024-09-17 11:52:01] [WARN] MQTTX force closed the connection MQTTBTest_SSL (Client ID: mqttx_93e9e033)
[2024-09-17 11:52:01] [INFO] Connection for MQTTBTest_SSL closed, MQTT.js onClose trigger
[2024-09-17 12:00:28] [INFO] Assigned ID 65948846-3751-44e2-b025-5cd9ee8b7e1d to MQTTX client
[2024-09-17 12:00:28] [INFO] Client MQTTBTest_SSL connected using MQTT/SSL connection at mqtts://mqttbtest.caredpal.com:8883
[2024-09-17 12:00:29] [ERROR] Connection for MQTTBTest_SSL failed, MQTT.js onError trigger, Error: Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
[2024-09-17 12:00:29] [WARN] MQTTX force closed the connection MQTTBTest_SSL (Client ID: mqttx_93e9e033)

The operating system my web server runs on is (include version):

CentOS Linux release 7.9.2009 (Core).

My hosting provider, if applicable, is:

Tencent cloud

I can login to a root shell on my machine (yes or no, or I don't know):

Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

no.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 1.11.0

2

Hi @rlin1788, and welcome to the LE community forum :slight_smile:

What listens on port 8883?
How is that configured to use an SSL certificate?

3 Likes
3

I set up the port and certificates in mosquitto.conf:

listener 8883
certfile /etc/letsencrypt/live/mqttbtest.caredpal.com/cert.pem
cafile /etc/letsencrypt/live/mqttbtest.caredpal.com/chain.pem
keyfile /etc/letsencrypt/live/mqttbtest.caredpal.com/privkey.pem

I use the command to check the ssl certificate and it looks fine:
openssl s_client -connect mqttbtest.caredpal.com:8883

But when I do this in the Java code:
ssl://mqttbtest.caredpal.com:8883

It gave me in the log:
Connecting to ssl://mqttbtest.caredpal.com:8883 with client ID CloudMqttSubscriber028f9301-b1ac-4f11-b184-78c2509d90d6
connect failedMqttException (0) - java.net.SocketException: Software caused connection abort: recv failed

4

Is that from your local network? Because I tried from a couple public locations and get failed connection. One such test site was this: https://decoder.link/sslchecker/mqttbtest.caredpal.com/8883

I get the same failure from my own test server too

3 Likes
5

I ran the command in a different server in Tencent: openssl s_client -connect mqttbtest.caredpal.com:8883 and return the certificates.

I did try your link and encountered the same failure. But if I try: https://decoder.link/sslchecker/8.154.38.204/8883, it seemed to find the certificates. Something wrong with DNS? But I can ping the domain.

6

btw I see ICP filling error page if I access it from web browser

3 Likes
7

What version of openssl are you using?

Which ciphers and algorithms are supported?

Using version "OpenSSL 3.0.2" I get:

openssl s_client -connect mqttbtest.caredpal.com:8883
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 324 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Using version "OpenSSL 1.0.1j" I get:

CONNECTED(000002B4)
depth=1 C = US, O = Let's Encrypt, CN = R10
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/CN=mqttbtest.caredpal.com
   i:/C=US/O=Let's Encrypt/CN=R10
 1 s:/C=US/O=Let's Encrypt/CN=R10
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE/DCCA+SgAwIBAgISBBn5IpY3ZhyxTdHh2tMuslM1MA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTAwHhcNMjQwOTE2MTM0MjIzWhcNMjQxMjE1MTM0MjIyWjAhMR8wHQYDVQQD
ExZtcXR0YnRlc3QuY2FyZWRwYWwuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
MIIBCgKCAQEAp8/Steq/q7INzWauauP2uvbUXvJ+p175pheVpVfIXgqBqCPLlMQI
kX2kfiatg5o8Lz25181E0mBNsizaSRWkGeixdXj02LKjn8DcVhI46F3qmkI58CfY
PB41+pJLis3XjbE0CVxh/2z+tYqfzCksmz5xvxxkimRtDKs/UhE+bcFb3LkosI+w
SPSgl4oX67NUz0R5jEEc0dcMt9pnhjz5zraEcHigbIOZvtN55XFaOF0D+O8QVbgX
v0arI2KgBxAQq1RacGST6ROLE3h/KRXhUGcw0NIwyv5+KxmErV8/MVYxivduL+Gf
c4cYnX4HQHsQprulhPBQS+XhIZ13iXqfcQIDAQABo4ICGjCCAhYwDgYDVR0PAQH/
BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8E
AjAAMB0GA1UdDgQWBBQfucU4tv3yRaL2NBnl3cptwLRoDDAfBgNVHSMEGDAWgBS7
vMNHpeS8qcbDpHIMEI2iNeHI6DBXBggrBgEFBQcBAQRLMEkwIgYIKwYBBQUHMAGG
Fmh0dHA6Ly9yMTAuby5sZW5jci5vcmcwIwYIKwYBBQUHMAKGF2h0dHA6Ly9yMTAu
aS5sZW5jci5vcmcvMCEGA1UdEQQaMBiCFm1xdHRidGVzdC5jYXJlZHBhbC5jb20w
EwYDVR0gBAwwCjAIBgZngQwBAgEwggEEBgorBgEEAdZ5AgQCBIH1BIHyAPAAdwA/
F0tP1yJHWJQdZRyEvg0S7ZA3fx+FauvBvyiF7PhkbgAAAZH7R4XEAAAEAwBIMEYC
IQD2AeBY5soXQagT6n6lC89sOtEMOHcacdXwWW1DF2W9QAIhAMczOHx1nhtKTIVc
YVrMogqyUA42lSD2d/9GLWSrvR0hAHUAdv+IPwq2+5VRwmHM9Ye6NLSkzbsp3GhC
Cp/mZ0xaOnQAAAGR+0eF8gAABAMARjBEAiA1LBSTXZA6lBMiWwwHe5aeIVK6phEC
BBnXmKDHVgZduAIgfDoahYqucL1Dc6Zd0hW+U0uQl2OLKDEEcIDDnftJtJowDQYJ
KoZIhvcNAQELBQADggEBAK0P1uorDD8aaGxoADpUMVMmusQuD7lf6HGrC5mkA0Xn
juylyJ9bxUnIY9Tt9fldwcJKh/zLE6l/dM3xiY4AmjZAJgaC09PUugeeH4HFr5eH
M8GoypFAeEBukaJ9l7LqRV24feW8yqNqPxAnulQzsypha3WkbPe4Fy8ozu1C+92M
kvLQRPZ476jB3+nFmYzDmjTEqScdQBiFg7MjEPRhRJVTRspASOge8oDRl7TdB1if
YjxZ8Y7BHlaXVH0bHo3DWRh0Dr9lc3UuX2GyloUqy0N1ICsRA7iERbhZwmA+B4xe
dn6j6g/kf7hCE15AkoxPmhCV3F6tvt6jf/kpIMjmk8Q=
-----END CERTIFICATE-----
subject=/CN=mqttbtest.caredpal.com
issuer=/C=US/O=Let's Encrypt/CN=R10
---
No client certificate CA names sent
---
SSL handshake has read 3247 bytes and written 433 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: AF34BD2E80A9DCAEBC2DFD96B893B103EFB288AF3D4FCB7E58D7002546E2D4BC
    Session-ID-ctx:
    Master-Key: 4A0465EF616EA700321355F2976AB9F69CC6EC92EB3460E879FBD2EB92B04310B337A7AA75DA0A1CF6D2ABE9D842760C
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 42 cd f1 8c c7 a7 8b 05-d1 ae ca 60 a1 90 20 bc   B..........`.. .
    0010 - 2d 70 46 53 e4 02 e7 94-9d be b4 a7 20 b4 00 b9   -pFS........ ...
    0020 - 7b 8d d0 ee 30 7f 53 52-54 25 a9 b6 bd 2e 8f ce   {...0.SRT%......
    0030 - 30 29 03 f9 2a bf 83 20-52 b5 5d 00 ab 5f c9 2d   0)..*.. R.].._.-
    0040 - 47 f7 ec 39 ec 7e d8 58-50 5d 74 8e 8a 30 07 a6   G..9.~.XP]t..0..
    0050 - d6 2a b5 d3 e6 a0 d1 29-24 b5 4c ab 14 25 70 22   .*.....)$.L..%p"
    0060 - a7 bd 99 df c5 b3 05 0e-45 0e c6 53 f3 10 54 2f   ........E..S..T/
    0070 - 78 5c f6 89 85 b3 74 b4-6a 5d cc 7d 75 59 de b1   x\....t.j].}uY..
    0080 - 65 70 d5 20 54 22 cf ee-22 0d 65 fd fd 14 01 55   ep. T"..".e....U
    0090 - 46 18 69 9c 19 57 15 07-36 59 93 14 9e dd a6 d1   F.i..W..6Y......
    00a0 - e5 04 18 5a 34 f5 1a ad-5e 3c b0 53 02 5e ba eb   ...Z4...^<.S.^..

    Start Time: 1726639294
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
3 Likes
8

As best I can tell, these ciphers are supported via TLSv1.2:

ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES256-SHA384
ECDHE-RSA-AES128-SHA256
ECDHE-RSA-AES256-SHA
ECDHE-RSA-AES128-SHA
AES256-GCM-SHA384
AES128-GCM-SHA256
AES256-SHA256
AES128-SHA256
AES256-SHA
AES128-SHA

The problem is that OpenSSL version 3 can't connect using any of them.
As an example of that:

openssl s_client -connect mqttbtest.caredpal.com:8883 \
-tls1_2 -cipher=ECDHE-RSA-AES256-GCM-SHA384

CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 167 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1726640979
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

I believe there may be an issue with the underlying cryptographic libraries in use.
Short of going down that [presumably long] path and upgrading it to a "working" version...
I would first consider just putting this site behind an nginx (reverse) proxy.

3 Likes
9

There is no a web site for the domain. It is a mosquitto broker services.

10

Good info. Thank you!

Does that mean the certbot version (1.11.0) use an old cryptographic libraries that do not support the latest ssl, openssl, or mqtts connection?

How to upgrade to the "working" version of the certbot or to specify the latest cryptographic libraries?

12

No; Certbot is not involved in this problem - it is not the one listening on port 8883.
It is either mqtts OR your operating system that provides the cryptographic libraries.

Hava a look at these reverse proxy instructions I found online:
MQTT Bridge with Mosquitto and nginx | akeil.net.

3 Likes
13

I looked at the info link you provided and did the similar thing to put my mosquitto server behind nigix. Instead of using the bridge, I re-route the traffic back to the same server but on different port. Here is the nginx setting:

server {
    listen 8883   ssl ;
    server_name  mqttbtest.caredpal.com;
    ssl_certificate /etc/letsencrypt/live/mqttbtest.caredpal.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mqttbtest.caredpal.com/privkey.pem;
        ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
        ssl_session_timeout 5m;
    ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
        ssl_prefer_server_ciphers on;
  location /{
        proxy_pass http://127.0.0.1:1883;
        }
    access_log  /data/logs/nginx/logs/caredpal.com.log/caredpalcom.access.log access;
}

===============================
Here is my mosquitto.conf setting:

listener 1883 127.0.0.1

#listener 1883 0.0.0.0/0

#listener 8883
#certfile /etc/letsencrypt/live/mqttbtest.caredpal.com/cert.pem
#cafile /etc/letsencrypt/live/mqttbtest.caredpal.com/chain.pem
#keyfile /etc/letsencrypt/live/mqttbtest.caredpal.com/privkey.pem

==================================

The result is still what it used to be, not working. I found that when I used the domain name to perform the ssl checker, i.e., https://decoder.link/sslchecker/mqttbtest.caredpal.com/8883, it did not find the certificates. But if I used the ip to find the ssl checker, i.e., SSL Checker, it did find all the certificates with only one error "Doesn't match Common Name or/and SANs".

What may be wrong?

14

Very strange. Is there any network device between that nginx server and the internet that handles packet routing?

I ask because if I do an openssl request to your IP with a servername (SNI host) of your domain it fails. But, if I openssl to your domain name with servername of your IP it works. So, it looks like something is inspecting http traffic and directing to the wrong place when it sees that domain.

And, these requests look wrong too. Do you know what this "Beaver" server is? Because an HTTP (not https) request gets rejected differently depending on whether I use the domain name or the IP.

The expected failure sending HTTP to an HTTPS enabled port in nginx is this

curl -i http://8.154.38.204:8883
HTTP/1.1 400 Bad Request
Server: nginx

But, I see this very different response using your domain name

curl -i http://mqttbtest.caredpal.com:8883
HTTP/1.1 403 Forbidden
Server: Beaver

In case anyone is wondering, the IP used is the one in the public DNS

mqttbtest.caredpal.com.	0	IN	A	8.154.38.204

Here is the openssl test I referred to although I think the above showing "Beaver" is more interesting

openssl s_client -connect mqttbtest.caredpal.com:8883 --servername 8.154.38.204 | head
---
Certificate chain
 0 s:CN = mqttbtest.caredpal.com
   i:C = US, O = Let's Encrypt, CN = R10
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep 16 13:42:23 2024 GMT; NotAfter: Dec 15 13:42:22 2024 GMT
 1 s:C = US, O = Let's Encrypt, CN = R10
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256

Swapping the domain name and the IP in the openssl command fails. So, the SNI value is what matters to reproduce this failure.

2 Likes
15

We just found a solution by accident. The root cause is that we bought the domain name and registered with Tencent cloud but the service deployed server is on Alicloud that intercepts/blocks the domain traffic if not registered with ICP. It is very weird! After we moved the service to Tencent cloud, everything work! We are trying to resolve the intercept issue with Alicloud.

Thank you for all your help. Hope that I do not waste your time since this is weird situation unique in Alicloud without our knowledge of the domain interception.

3 Likes
16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.