Renewed certificate working from browsers but not the command line

Help
#1

I used certbot with DNS challenge to create a LetsEncrypt wildcard certificate for my domain a few months ago. It's been working fine. I tried to automate renewal this week but although browsers (Chrome, Firefox) seem happy with the renewed certificate, terminal commands (e.g. curl, openssl) are unable to use it.

I'm very new to SSL/TLS stuff so it's entirely possible there's an important step that I took last time but have since forgotten about. I've trawled around looking for posts about similar problems but haven't found any that seem a close match. My sites are private to my LAN and not exposed to the internet, so presumably online diagnostic resources won't help, and I don't yet have the experience to know how to use the command-line tools that might help. I'd be grateful if someone could point me in the right direction.

My domain is: boxersoft.com

I ran this command: curl -I https://home.boxersoft.com/

It produced this output:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I ran this command: openssl s_client -connect home.boxersoft.com:443

It produced this output:

CONNECTED(00000003)
depth=0 CN = *.boxersoft.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = *.boxersoft.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:CN = *.boxersoft.com
   i:C = US, O = Let's Encrypt, CN = R3
 <<snip>>

My web server is (include version): nginx/1.14.2

The operating system my web server runs on is (include version): Debian GNU/Linux 10 (buster)

My hosting provider, if applicable, is: N/A

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.31.0

Note: I have truncated the above output from openssl because I'm not sure whether any of it should be treated as private. Happy to post the full output from that or other commands if it would help, but would appreciate advice on what if anything should be redacted before posting. Thanks.

1 Like
#2

does this end here? It should look more like:

% openssl s_client -connect letsencrypt.org:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = lencr.org
verify return:1
---
Certificate chain
 0 s:CN = lencr.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
<<snip>>

what does your ssl_certificate_file directive say? it should point to fullchain.pem, not cert.pem

3 Likes
#3

Yes, that's all there is before the first certificate block:

$ openssl s_client -connect omv.boxersoft.com:443
CONNECTED(00000003)
depth=0 CN = *.boxersoft.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = *.boxersoft.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:CN = *.boxersoft.com
   i:C = US, O = Let's Encrypt, CN = R3
---
Server certificate
-----BEGIN CERTIFICATE---

Output prior to certificate renewal did indeed have more in the certificate chain:

$ openssl s_client -connect microserver.boxersoft.com:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.boxersoft.com
verify return:1
---
Certificate chain
 0 s:CN = *.boxersoft.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
1 Like
#4

Check this indeed:

3 Likes
#5

Bingo! (I think).

Assuming you're talking about the nginx .conf file (said I'm new to this), it has a ssl_certificate directive (without the _file, I don't know if that's significant). The configuration is managed by OpenMediaVault which seems to copy certificates into a location under its control, renaming them as a GUID. Having checked that, it does indeed seem to be a copy of cert.pem. I'll take a closer look at the routines I called to trigger OMV's certificate management routines (I lifted them from a blog post), presumably they're they're taking the contents of cert.pem instead of fullchain.pem.

Thanks very much - I'll report back later, hopefully to confirm success.

I'm surprised that browsers are coping when terminal commands aren't. Do browsers perhaps detect that they've just been given a single certificate and do more work behind the scenes to get the rest of the required details or something?

2 Likes
#6

Your system looks offline right now.

I recently had the same problem on some Macs - web browsing was fine, but all the commandline utils failed with OpenSSL errors on some domains. Unfortunately, all those domains were ones that I needed commandline tools for!

If it is not, I am not sure if this will work for you, but the error and fix in my situation was:

  • The servers used the default (Android support) chain that was linked to the expired DST Root. I did not control the servers, so switching roots was not an option.
  • The version of OSX I had locally did not have a more recent OpenSSL lib that builds its own trust path. Rebuilding OpenSSL wasn't an option, because I'd have to rebuilt all the commandline toos (git, curl, etc).
  • I remembered there was a possible fix by altering the trust store. Thankfully that worked. I just had to delete the expired "DST Root X3" certificate that was in openssl's trust store. ISRG Root X1 should already be in the trust store, you just need to delete the expired DST certificate.

IIRC, if you add the verbose command to openssl or curl, it should give you the location of the file.

This may not be your problem solution, but it looks relatively similar to mine.

Updating the openssl library might be an option for your OS and is preferable, but it might not be. if it's an option, try that first. if not, you can try removing the dst root.

2 Likes
#7

yes, browsers have the intermediates in cache and can download them if not.

3 Likes
#8

Ah, saw the above exchange. Most browsers build their own trust path and use a cache of certs they've seen - they probably detected it was signed by a LetsEncrypt cert they already knew about. Commandline tools typically don't use a cache.

3 Likes
#9

It's internal only, not exposed to the internet. I did mention that in the original post, but it was a bit buried in my general ramblings. Now sorted anyway.

Got it - thanks for confirming.

2 Likes
#10

OK, checked and confirmed - the code I lifted was indeed telling OpenMediaVault to use cert.pem instead of fullchain.pem. Modified version has fixed the situation, hopefully the next renewal will go smoothly now.

Many thanks to all who contributed.

5 Likes
#11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.