Acme-client bad comm cert verify failed

I've been running for three years and now I get

140237683025648:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:
140237683025648:error:140E0197:SSL routines:SSL_shutdown:shutdown while in init:ssl/ssl_lib.c:2086:
acme-client: https://acme-v02.api.letsencrypt.org/directory: bad comm

Is this related to the TLS cipher suite issue? My openssl is 1.1.1g.
Paul

1 Like

No, I don't think this is related to the TLS cipher change in that thread you originally posted on. This error looks like a problem connecting TO the acme server. That change was for the LE auth centers contacting your server (so, the other way around).

I've included the form you would have been shown for a new Help topic below. Please answer as much as you can. Thanks

==================================

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

4 Likes

Hi MikeMcQ,
Thank you for quick response:

  1. trifox.com
  2. /usr/local/bin/acme-client -f /home/pade/acme-client.conf mail.trifox.com
    acme-client.conf is
    authority letsencrypt {
    api url "https://acme-v02.api.letsencrypt.org/directory"
    account key "/tmp/acme/letsencrypt-privkey.key" rsa
    }

domain mail.trifox.com {
domain name "mail.trifox.com"
domain key "/tmp/acme/mail.trifox.com.key" rsa
domain full chain certificate "/tmp/acme/mail.trifox.com.pem"
sign with "letsencrypt"
challengedir "/usr2/www/htdocs/.well-known/acme-challenge"
}

  1. acme-client: /tmp/acme/mail.trifox.com.pem: certificate renewable: 28 days left
    acme-client: https://acme-v02.api.letsencrypt.org/directory: directories
    acme-client: acme-v02.api.letsencrypt.org: DNS: 172.65.32.248
    140465168869104:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:
    140465168869104:error:140E0197:SSL routines:SSL_shutdown:shutdown while in init:ssl/ssl_lib.c:2086:
    acme-client: https://acme-v02.api.letsencrypt.org/directory: bad comm
  2. Server version: Apache/2.2.10 (Linux/SUSE)
    Server built: Feb 23 2009 21:08:14
  3. SUSE Linux Enterprise Server 11 (x86_64)
    VERSION = 11
    PATCHLEVEL = 0
  4. Self-hosted
  5. I can log on as root
  6. No control panel, direct logon
  7. I cannot get a version from acme-client. I built it from source with this history:
    HISTORY
    The acme-client utility first appeared in OpenBSD 6.1.
    AUTHORS
    The acme-client utility was written by Kristaps Dzonsons
    kristaps@bsd.lv.
  8. My openssl is 1.1.1g.
  9. My cron runs this on the 1,11,21 every month. The 7/21 run did not return an error, the 8/1 run return the above error.

I appreciate any pointers you can give me.
Regards,
Paul

1 Like

Can your system connect to other servers outbound?

Is there any chance that it, or some router or firewall on its network, somehow thinks that 172.65.32.248 is on a local network? It's "close" to the private 172.16.0.0/12 network, so that's something we've occasionally seen here with systems that are able to connect to most of the Internet but not to Let's Encrypt's servers.

3 Likes

No problem getting out from there. It's been working for years until two days ago. I pinged 172.65.32.248, responded however I just ran Firefox on the server, connected to

https://acme-v02.api.letsencrypt.org/directory and got

acme-v02.api.letsencrypt.org uses an invalid security certificate.

Untitled|517x284

Now this could Firefox' problem running on an ancient SuSe or maybe I somehow have to update the server CA files? Something certainly changed in the last two weeks or so.
Regards,
Paul

1 Like

Can you view the certificate and post that here in the Firefox you are getting that error from? That may be enlightening to help understand what’s going on.

3 Likes

Or, post the result of this:

echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
4 Likes

I'm not sure it will help because the same Firefox trying to get to www.suse.com also gets a (different) cert error. In any case here it is:

Untitled1

Regards,
Paul

Here it is:

depth=1 C = US, O = Let's Encrypt, CN = R10
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = acme-v02.api.letsencrypt.org
verify return:1
CONNECTED(00000004)
---
Certificate chain
 0 s:CN = acme-v02.api.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R10
 1 s:C = US, O = Let's Encrypt, CN = R10
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
DONE

Thanks. Is it possible you added the R3 intermediate cert into your cert store?

Because LE is now using new intermediates R10 and R11. Your last good good cert was issued by R3 so I'm guessing this started failing as soon as the acme-v02.api server got a cert using the new intermediates.

All of these lead to ISRG Root X1 so if you had that in your CA store it should have worked and continued to work.

I don't know how to check the CA Store on SUSE off-hand. But, is this possible?

5 Likes

If you cannot find it with YaST or from the command line, SUSE has zip files with documentation for core components of their obsolete software.

4 Likes

(Before I get into my update I want to thank all of you for your time and help. I did write the SSL and SChannel (Windows) interfaces for our products but the cert management of what goes where has always been a little obscure to me)

I do have ISRG Root X1:

server:/etc/ssl/certs # dir | fgrep isrgrootx1.pem
lrwxrwxrwx 1 root root 14 2021-10-28 09:13 8d33f237.0 -> isrgrootx1.pem
-rw-r--r-- 1 root root 1939 2021-10-28 09:09 isrgrootx1.pem

This is the last thing that has been done in /etc/ssl/certs until the below.

I downloaded the LE CA file using
wget "https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt" --output-document=letsencryptauthorityx3.pem

then set up the hash link in /etc/ssl/certs:

lrwxrwxrwx 1 root root 26 2024-08-03 09:10 4f06f81d.0 -> letsencryptauthorityx3.pem
-rw-r--r-- 1 root root 1984 2024-08-03 09:03 letsencryptauthorityx3.pem

Still no luck. Of course my Yast on this system is missing the ca_mgm module which is what it uses for CA management.

Thank you again for your time and help.

2 Likes

Hmm. The "X3" cert you just downloaded expired almost 3 years ago. So you definitely do not need that. That was an intermediate so you shouldn't have that in your CA root store anyway. Perhaps coincidentally, X3 expired just a few weeks before the timestamp on the isrgrootx1.pem file you show.

Can you post the contents of the isrgrootx1.pem ? The name by itself doesn't mean anything. We want to see what it really is. If it is actually that root it certainly isn't private info. I have a feeling it might actually be the R3 intermediate that leads to X1.

3 Likes

Ok I removed the letsencryptauthorityX3*. I am curious how it can be expired when I just got it from LE? Anyway attached is isrgrootx1.pem.

isrgrootx1.pem (1.9 KB)

It came from the Retired section of certs: Chains of Trust - Let's Encrypt

And, yes, that actually is active ISRG Root X1. So, no luck on an easy answer :slight_smile:

Systems have various ways to manage the root store. And, even different clients can use different stores. So, that must not be used by all your clients (openssl for example). Or, needs to be combined with other .pem files to create a composite root store.

What does this show

curl -v https://acme-v02.api.letsencrypt.org/directory
3 Likes

Linux (50) Yes ? curl -v https://acme-v02.api.letsencrypt.org/directory

  • Trying 172.65.32.248...
  • TCP_NODELAY set
  • Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: none
    CApath: /etc/ssl/certs/
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • NPN, negotiated HTTP1.1
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (OUT), TLS alert, unknown CA (560):
  • SSL certificate problem: unable to get local issuer certificate
  • Closing connection 0
    curl: (60) SSL certificate problem: unable to get local issuer certificate
    More details here: curl - SSL CA Certificates

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Linux (51) Yes ?

What shows?:
curl -k https://acme-v02.api.letsencrypt.org/directory

2 Likes

Well, that matches up. Curl showed the CA path it was using and you've shown ISRG Root X1 in that path. Yet even curl says it can't validate.

What does this show?

curl --cacert /etc/ssl/certs/isrgrootx1.pem https://acme-v02.api.letsencrypt.org/directory
2 Likes

Linux (51) Yes ? curl --cacert /etc/ssl/certs/isrgrootx1.pem https://acme-v02.api.letsencrypt.org/directory
{
"3T3oKLdQ2yw": "Adding random entries to the directory",
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.4-April-3-2024.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"renewalInfo": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-03/renewalInfo",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}Linux (52) Yes ?

Ah, that's helpful. Alas, I don't know how to instruct on fixing it.

That proves it is the right ISRG Root X1 and curl can validate to it when explicitly told to do so.

But, when curl was told just to use the CApath it failed. And, it failed the same way openssl did and your client in the first post.

Taking from the curl manual it says this about CApath. Something related to the highlighted text (about c_rehash) is probably the key to fixing your problem. I just don't know exactly how do to that with your system.

--capath
(TLS) Tells curl to use the specified certificate directory to verify
the peer. Multiple paths can be provided by separating them with ":"
(e.g. "path1:path2:path3"). The certificates must be in PEM format,
and if curl is built against OpenSSL, the directory must have been
> processed using the c_rehash utility supplied with OpenSSL. Using
--capath can allow OpenSSL-powered curl to make SSL-connections much
more efficiently than using --cacert if the --cacert file contains
many CA certificates.

2 Likes