Today was my first attempt to use acme.sh script in 2023: it worked but certs failed to update

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: preciselyparrots.com

I ran this command: acme.sh --issue -d preciselyparrots.com -d www.preciselyparrots.com --server letsencrypt --standalone

It produced this output: Buffer text is no longer available so cannot paste here. However, the session updated with 'success' message (as it always has) but the expired certs remained in place (which has never happened before). Tried multiple times, each one seeming successful as well - cert files showed current timestamps. But the site still showed no cert renewal. Tried deleting the new cert files from the .acme.sh/preciselyparrots.com directory and then running the acme.sh script again. That resulted in it successfully downloading new cert files again... but this time it placed them in a new directory it created called preciselyparrots.com_ecc and left the old non ecc directory empty. Once the cert files were copied to the cert store location and the server was restarted, the same result ensued (ie, old cert with old expiration date - thus no tls connectivity possible). Since my attempts exceeded 5 times, letsencrypt auto-blocked all further attempts (for 2 days). I'm unsure what changed to keep the renewal from reflecting upon connectivity; it had always worked fine previously. I also tried a manual update at gethttpsforfree.com but there was no chance of that working while the block is active. I have a debug output if that's helpful.

My web server is (include version): Apache/2.2.34

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is: CentOS 6.10

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): DA 1.643
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): acme.sh

2

You are getting fresh certs just fine. You can see them in the public logs (link here)

But, your server is mis-configured. It is using a self-signed cert for "localhost". See a SSL Checker site like this one (link here)

You need to review your Apache config. And, Apache 2.2 really?

4 Likes
3

Hi @edgeform, and welcome to the LE community forum :slight_smile:

I see a cert for relay-link.net (as does SSL Labs):
SSL Server Test: preciselyparrots.com (Powered by Qualys SSL Labs)

3 Likes
4

hmm...
I now see:

Certificate chain
 0 s:C = US, ST = Someprovince, L = Sometown, O = none, OU = none, CN = localhost, emailAddress = webmaster@localhost
   i:C = US, ST = Someprovince, L = Sometown, O = none, OU = none, CN = localhost, emailAddress = webmaster@localhost
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
3 Likes
5

Interesting. I see that SSL Labs result. I get a self-signed from openssl on my own server and the SSL Checker sees that too.

Maybe an SNI problem in an old system?

3 Likes
6

You can use:
acme.sh --list
to show the certs being managed.

Not sure what's the CentOS6/Apache2.2 equivalent of:
sudo apachectl -t -D DUMP_VHOSTS
But you definitely need to check (and correct) your Apache configuration.

3 Likes
7

Wow. The Apache config is definitely awful. I went thru the SSL Labs tests and the server responds with 3 different certs. Not a chain and 2 intermediates - 3 different cert chains (well, at least partial chains)

One of which even has a leaf for preciselyparrots issued by Let's Encrypt today. Which is revoked !

I don't have the time to help with this. Just thought the full SSL Labs result worth posting.
https://www.ssllabs.com/ssltest/analyze.html?d=preciselyparrots.com

4 Likes
8

Yes, I also updated relay-link.net via acme.sh earlier today as well. But the site is reporting the old cert, not the new one. Since today's update effort, the cert file shows a date/time stamp of Feb 15 17:28 and the key file shows one of Feb 15 17:29. Yet the certificate that's reported once the website is loaded still shows as having been issued on December 22, 2022 (the previous time it was renewed). So none of the sites on the server will update with the acme.sh script any longer. Nothing was changed other than the new year. Again, acme.sh worked beautifully all through 2022.

preciselyparrots.com "ec-256" www.preciselyparrots.com LetsEncrypt.org 2023-02-16T00:51:56Z 2023-04-16T00:51:56Z
preciselyparrots.com "ec-256" www.preciselyparrots.com LetsEncrypt.org
relay-link.net "2048" www.relay-link.net LetsEncrypt.org 2022-12-22T18:04:31Z 2023-02-19T18:04:31Z
relay-link.net "ec-256" www.relay-link.net LetsEncrypt.org 2023-02-15T23:22:47Z 2023-04-15T23:22:47Z

9

You still need to review the Apache config.
You can match the cert files used within Apache and provided by acme.sh with (something like):
ls -lR /root/.acme.sh/*.cer
and
grep -i certificate /etc/apache2/*
grep -i certificate /etc/httpd/*

2 Likes
10

Okay, so here's the latest. I am seemingly able to renew certs manually via gethttpsforfree.com. I've just updated the relay-link.net certs via that site and the operation went quite smoothly, rendering updated certificates that display the following validity period to clients:

Issued On Thursday, February 16, 2023 at 12:58:13 AM
Expires On Wednesday, May 17, 2023 at 1:58:12 AM

As stated earlier, yesterday afternoon I discovered that while the acme.sh script would indeed create new certificate files - including for relay-link.net - the validation period as seen by the client refused to update. This leads me to believe (or at least hope) that once letsencrypt's block on renewal of the preciselyparrots.com site's certs has been lifted, I may be able to update its certs manually, as I've just done for relay-link.net. If so, all I can say is I am puzzled by whatever it is that's gone wrong with the acme.sh script's functionality since it used to seem quite dependable throughout 2022 (as far as updating all certs on this private server with no apparent issues). Again, I didn't change anything so it seems as if it simply broke on its own. Yes, the issue(s) could be server configuration related. However, it's the exact same configuration that worked fine for quite some time. I'm not sure what to think yet.

Thanks for the replies.

1 Like
11

Your Apache is sending out 3 certs for the preciselyparrots.com domain. There is something wrong.

Your relay-link.net domain is now working but that domain is your default VirtualHost. I know this because when I send a non-SNI request to your server that is the cert I see. So, what "worked" for this domain probably won't work for parrots.

Apache 2.2 is very old but if you show us output of this we might be able to sort it out

apachectl -S

(that's capital S)

4 Likes
12

Also, you got 5 certs for your parrots domain yesterday. All are revoked with the reason of keyCompromise (pic below)

You will need to wait 168 hours from your most recent cert before getting another one because you used up your limit of 5 / week. See Let's Debug cert search (link here)

Also important is you must use a different key when requesting your next cert as the one you've been using is compromised. I don't know how to do that in acme.sh but perhaps someone else here will or see the github for acme.sh (or its docs)

Screenshot 2023-02-16 111958
Screenshot 2023-02-16 1119581920×1111 160 KB

4 Likes
13

@edgeform, revoking a cert is almost never the thing to do. In particular, it won't help you with rate limits (which is the most common reason people ask here about revoking). You should only revoke a cert if you have reason to believe that your private key has been compromised.

6 Likes
14

First of all, thanks to both of you for your insights. And yes, I did use the revoke command 'acme.sh --revoke' yesterday which was a flub I regret. But I had never encountered this with acme.sh before and I acted before thinking. Also, please know that I changed the private key prior to updating relay-link.net.

I suspect the 'something wrong' with preciselyparrots.com was caused by the combination of acme.sh's newly emerged functionality issue and my own actions. All I've ever had is a single cert and a csr for each domain and of course the key file. The certs are only 'pointed to' one time in the vhost.conf file and it is of course referenced only once in the httpd.conf file. There's nothing else pointing to any of the certs and no other certs that I'm aware of. So if there are more certs hiding somewhere and they are somehow accessible to apache, I've never seen them. And how would they all become renewed with a single renewal effort every 3 months? But who knows... stranger things have turned out to be true.

And actually, I think the manual request method may end up working for preciselyparrots.com once the block has expired. I'll try and explain why.

Another domain hosted on the same box was in the exact same condition yesterday after acme.sh got through with it. That domain is shellprox.net. It had expired on Feb 13 but yesterday acme.sh was reporting that it was NOT due to expire until mid-April. There was thus nothing more I could think of to do with the situation utilizing acme.sh and I decided to put off trying the manual method.

Well, that was until I got a few minutes a short time ago today. About an hour ago, I successfully updated the cert for shellprox.net via the gethttpsforfree website's manual cert renewal method. The session went very smoothly - like it always had before I started using acme.sh (which last year worked wonderfully - and was much faster!) So that's why I'm guessing I may not have any problem updating preciselyparrots.com either.

Here's a question for you pertaining to the message pasted below:
Why does it state to retry after "2023-02-17" if I really have to wait 7 days (ie, 168 hours)? I interpret the message text below to mean only that (5) certs were issued in the last 168 hours and that I can try again after tomorrow (2-17-2023). If they don't mean that then I have to say they worded the message incorrectly.

[Wed Feb 15 19:25:05 CST 2023] Create new order error. Le_OrderFinalize not found. {
"type": "urn:ietf:params:acme:error:rateLimited","detail": "Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: preciselyparrots.com,www.preciselyparrots.com, retry after 2023-02-17T09:27:54Z: see https://letsencrypt.org/docs/duplicate-certificate-
limit/",
"status": 429
}

As for why this acme.sh thing happened... my observations, questions, as well as some conclusions (albeit perhaps not all of those are correct) follow with some text output examples:

[root@eccentric ~ 08:58:21]$ acme.sh --issue -d shellprox.net -d www.shellprox.net --server letsencrypt --standalone
[Thu Feb 16 08:58:24 CST 2023] Domains not changed.
[Thu Feb 16 08:58:24 CST 2023] Skip, Next renewal time is: 2023-04-15T23:23:19Z

As you can see, the acme.sh script/process thinks the cert's validation period is still current.

But web browsers report it as being this:

Issued On Tuesday, November 15, 2022 at 8:23:57 AM
Expires On Monday, February 13, 2023 at 8:23:56 AM

...and of course SSL won't load because it truly has expired!

So since acme.sh thinks the cert is still current, it renders the "Add '--force' to force to renew" message when it's executed.

So this is what happens when I do that:

[root@eccentric ~ 08:58:50]$ acme.sh --issue -d shellprox.net -d www.shellprox.net --server letsencrypt --standalone --force
[Thu Feb 16 08:58:53 CST 2023] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Thu Feb 16 08:58:53 CST 2023] Standalone mode.
[Thu Feb 16 08:58:55 CST 2023] Standalone mode.
[Thu Feb 16 08:58:56 CST 2023] Multi domain='DNS:shellprox.net,DNS:www.shellprox.net'
[Thu Feb 16 08:58:56 CST 2023] Getting domain auth token for each domain
[Thu Feb 16 08:58:58 CST 2023] Getting webroot for domain='shellprox.net'
[Thu Feb 16 08:58:58 CST 2023] Getting webroot for domain='www.shellprox.net'
[Thu Feb 16 08:58:58 CST 2023] shellprox.net is already verified, skip http-01.
[Thu Feb 16 08:58:58 CST 2023] www.shellprox.net is already verified, skip http-01.
[Thu Feb 16 08:58:58 CST 2023] Verify finished, start to sign.
[Thu Feb 16 08:58:58 CST 2023] Lets finalize the order.
[Thu Feb 16 08:58:58 CST 2023] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/533092826/165140011936'
[Thu Feb 16 08:58:58 CST 2023] Downloading cert.
[Thu Feb 16 08:58:58 CST 2023] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/0382463d5f24fa80a963d96d06906c5bf96e'
[Thu Feb 16 08:58:59 CST 2023] Cert success.

Note the following lines:

shellprox.net is already verified, skip http-01.
www.shellprox.net is already verified, skip http-01.

For comparison, here's what it should do and what it used to do after a cert had expired:

[root@eccentric ~ 13:31:11]$ acme.sh --issue -d shellprox.net -d www.shellprox.net --server letsencrypt --standalone
[Sat May 7 13:31:14 CDT 2022] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat May 7 13:31:14 CDT 2022] Standalone mode.
[Sat May 7 13:31:15 CDT 2022] Standalone mode.
[Sat May 7 13:31:15 CDT 2022] Multi domain='DNS:shellprox.net,DNS:www.shellprox.net'
[Sat May 7 13:31:15 CDT 2022] Getting domain auth token for each domain
[Sat May 7 13:31:17 CDT 2022] Getting webroot for domain='shellprox.net'
[Sat May 7 13:31:17 CDT 2022] Getting webroot for domain='www.shellprox.net'
[Sat May 7 13:31:17 CDT 2022] Verifying: shellprox.net
[Sat May 7 13:31:17 CDT 2022] Standalone mode server
[Sat May 7 13:31:18 CDT 2022] Pending, The CA is processing your order, please just wait. (1/30)
[Sat May 7 13:31:22 CDT 2022] Success
[Sat May 7 13:31:22 CDT 2022] Verifying: www.shellprox.net
[Sat May 7 13:31:22 CDT 2022] Standalone mode server
[Sat May 7 13:31:23 CDT 2022] Pending, The CA is processing your order, please just wait. (1/30)
[Sat May 7 13:31:26 CDT 2022] Success
[Sat May 7 13:31:26 CDT 2022] Verify finished, start to sign.
[Sat May 7 13:31:26 CDT 2022] Lets finalize the order.

So this new flawed detection and 'skipping' behavior from acme.sh seems to indicate that now for whatever reason it is creating new files, but they contain the same old data as that in the expired certs - which it's convinced are current for some reason.

Once again, I didn't change anything on the server and acme.sh never demonstrated this problem before yesterday. Thankfully, though, the gethttpsforfree manual request page realizes (or perhaps doesn't check) that the certs have actually expired so it renders new ones properly (and guess what? They work!) So We'll see.

1 Like
15

Yes, that message seems odd. And, I mis-stated earlier. The 5/week limit is a rolling limit so once the oldest of those 5 rolls off you can get one to replace it. And, so on. I earlier said is 168H after the latest which is not correct.

The best explanation of this rate limit is below which also describes a work-around.

I do not follow all your steps with acme.sh. I think you would be better off asking on the github for acme.sh. The developer is there and other experts who may understand it better. There are dozens of ACME clients and we cannot know every feature / oddity of every one.

PS: Don't forget that SSL Labs is very confused by your parrots domain. Something more than just the cert is a problem.

4 Likes
16

Thanks for the heads up. I'll try and investigate what's going on with the parrots site when I have some more time. And, of course, that'll be once I get the certs straightened out and it's accessible again. That site has been running on various servers since 2013 and it undoubtedly needs some tweaks here and there.

Thanks again!

2 Likes
17

Testing and debugging are best done using the Staging Environment as the Rate Limits are much higher.

4 Likes
18

Update:
The parrot domain cert has been successfully renewed via the manual method (as expected). All domain certs on the server have been renewing properly and smoothly through the manual method. My conclusion is that something related to the acme.sh script mechanism broke (or was changed) at some point after November, thus causing the issue. As recommended, I will reach out to other resources if the problem persists. At least the manual renewal method has been quite dependable as a workaround.

Thanks to all!

1 Like
19

Your parrots domain is failing cert validation as you are not sending the intermediate chain. You only send the leaf.

See SSL Checker (link here)

or SSL Labs. Note it now only shows 2 certs which is improvement. The second is connection without SNI. As long as you don't need to support non-SNI clients (and you probably don't) you can ignore that.

https://www.ssllabs.com/ssltest/analyze.html?d=preciselyparrots.com&hideResults=on&latest

3 Likes
20

Well, when the check is executed on the www subdomain hostname the chain shows to be intact with no intermediate certificate chain issues at all (except as noted below). Plus, it gives the website an 'A' rating.

https://www.ssllabs.com/ssltest/analyze.html?d=www.preciselyparrots.com

There's a problem, though, on both sites with path 2 (the DST Root CA X3 certificate). It shows the following every time:
Path #2: Not trusted (invalid certificate [Fingerprint SHA256: 0687260331a72403d909f105e69bcf0d32e1bd2493ffc6d9206d11bcd6770739])

But I still get no errors and the 'A' rating when including the www subdomain extension on the parrot site.