My fullchain certificate seems to be breaking over time. [acme.sh] [macOS]

Help
#1

Hi all,

I am using the DNS-01 challenge with the acme.sh client on a macOS computer running 4D 16.4.0. My hosting provider is DreamHost, and acme.sh uses the DreamHost DNS API to automate the process. It works great.

This 4D server is an internal database that we've made accessible from the web to XHR read/write from our actual DreamHost website using various RestAPI's such as Gravity Forms on WordPress.

The problem I seem to be having is that every month or so, before the certificate expires, the chain breaks with no other changes to the certificate files or webserver.

If I run various certificate checkers for fixing the issue, I get errors about intermediates missing and the full chain certificate not being present.

All it takes to fix this is for me to re-run my Terminal command, which is:
./acme.sh --issue --accountemail "info@bel.com" --dns dns_dreamhost -d simon4d.bel.com --fullchain-file "/WebServerPath/cert.pem" --key-file "/WebServerPath/key.pem" --force

The script does it's thing and after restarting the webserver, the chain is fixed and it passes the certificate checkers.

Why would this break over time?

1 Like
[Part 2] My fullchain certificate seems to be breaking over time. [acme.sh] [macOS]
[Part 2] My fullchain certificate seems to be breaking over time. [acme.sh] [macOS]
#2

I'm having trouble rationalizing how anything like this can even happen.
It would be very good to look at the problem while it is happening.
Of course, that might mean that your production system is being impacted.
But I can't even think of any reason how/why this can be.
So, unless you do get a response that clearly explains it and remedy the situation (:crossed_fingers: ), when this happens again, please let us know immediately and maybe we can get a good look at it in action.

In the meantime...
Are there any other scripts that might be run on a similar schedule (monthly) that have anything to do with your cert?
And, how often does the script that fixed it run? Can we see that script?
[ I know you showed the command above - but please show the entire script that calls it (if any)]

1 Like
#3

Hi @muzicman82

sounds there is a second script / another ACME-client that renews the certificate and installs the wrong cert.pem (instead of fullchain.pem).

That would explain such a break.

3 Likes
#4

I've attached the acme.sh script.

The acme.sh client makes a cron job that I believe runs daily, but it issues a --cron flag which I assume just checks and updates if required...
52 0 * * * "/Users/simon/.acme.sh"/acme.sh --cron --home "/Users/simon/.acme.sh" > /dev/null

I've had to manually run my Terminal string so often I don't think the cron job ever actually has to update anything. I've no idea where to find any cron logs. The date on the .pem files was that of my last manual run, not the day it broke things.

Is it possible to run the exact string I do at the Terminal window as the Cron command? I'm not familiar with how to change the command or anything.

acme.sh.txt (192.1 KB)

1 Like
#5

Your terminal string is using --force
That would be bad to do in a script.

1 Like
#6

This part of your command is leading me to believe there are other things involved here:

Why are you using that path for your files instead of directly from where acme.sh stores them?
I suppose this synchronization is NOT happening on each automated renewal.
So the updated files are in another folder and this folder becomes out-of-sync.

1 Like
#7

I've been using the --force so that the script regenerates the files regardless of whether they are expired or need to be updated. I've tried without that before and it checks the files and decides no update is required.

1 Like
#8

That sounds like using a sledgehammer to drive in a thumbtack.
You might only need to resynchronize the files.
Every time you use --force a new certificate is being issued.

That's because not update is required.

1 Like
#9

I abbreviated the path for this post, but using those flags tells acme.sh to put the files in the folder of the 4D Webserver which is where that webserver wants them. If I leave it default, then I have to manually copy the files over.

I can't remember what the script makes by default, but it wasn't the cert/key PEM files that 4D server works with.

1 Like
#10

Sounds like all you need is a script (to override some other script that removes/overwrites) to resynchronize the files.
OR
You could look for the reason they get un-synchronized.

ACME.sh stores files (normally) in /root/.acme.sh/{domain.name[_ecc]}/

1 Like
#11

Next time it fails, try:
cp /User/simon/.acme.sh/simon4d.bel.com/fullcain.cer /WebServerPath/cert.pem
cp /User/simon/.acme.sh/simon4d.bel.com/simon4d.bel.com.key /WebServerPath/key.pem

1 Like
#12

I believe I originally did that (manually copied the files over) before I came up with the single command.

1 Like
#13

Your single command creates a new cert every time you run it.
[which is completely unnecessary]

1 Like
#14

Other than wasting resources, there ia a real danger here of hitting the rate limits.

Renewals are treated specially: they don’t count against your Certificates per Registered Domain limit, but they are subject to a Duplicate Certificate limit of 5 per week. Note: renewals used to count against your Certificate per Registered Domain limit until March 2019, but they don’t anymore. Exceeding the Duplicate Certificate limit is reported with the error message too many certificates already issued for exact set of domains .

A certificate is considered a renewal (or a duplicate) of an earlier certificate if it contains the exact same set of hostnames, ignoring capitalization and ordering of hostnames. For instance, if you requested a certificate for the names [ www.example.com , example.com ], you could request four more certificates for [ www.example.com , example.com ] during the week. If you changed the set of hostnames by adding [ blog.example.com ], you would be able to request additional certificates.

Renewal handling ignores the public key and extensions requested. A certificate issuance can be considered a renewal even if you are using a new key.

Current Certificate History

Screenshot_20201112-131659_Samsung Internet
Screenshot_20201112-131659_Samsung Internet1216×2118 1.51 MB

1 Like
#15

You mentioned you ran certificate checkers.

Have you tried inspecting the broken certificate and keys with OpenSSL to see what is wrong? Perhaps a script invocation failed midway. Perhaps the script corrupted one of the files or deleted/emptied another. There are so many ways a shell script could fail and break a setup like this.

2 Likes
#16

please enable log first:

acme.sh  --cron --log --log-level 2

Then every day it will write log to ~/.acme.sh/acme.sh.log.

Next time you can check this log.

3 Likes
#17

Hi guys,

The problem has occurred again, and this time I haven't fixed it yet.

I made a test website that uses the certificate here.

I used WinMerge to compare the current certificate to the files created the last time I fixed the problem and they are byte for byte identical.

1 Like
#18

Currently that site is not serving the intermediate certificate at all (as if it's using cert rather than fullchain). Can you look at your configuration to try to see why the intermediate certificate isn't being served?

2 Likes
#19

Worth remembering that the Let's Encrypt intermediate has just changed from X3 to R3, so if you use an old pem file as the intermediate the new cert won't work because X3 is the wrong issuer.

2 Likes
#20

Ok, so I see that it the intermediates aren't showing up in a SSL checker but the files are identical to when they did pass the same checkers back in November.

I need this to work so I went ahead and did a force command the same as described in my OP with the PEM paths in the command line, and everything passes once again. I made copies of the files for reference for when they break again.. but I really have no idea what is going on... It seems like the SSL checkers are not validating against the same files at some point in the future. Could this be a problem external to my web server?

Would it help anyone if I posted the old and current PEM files?