Please show:
ls -l /etc/letsencrypt/live/mail.jewettfarm.com/
ls -l /etc/letsencrypt/archive/mail.jewettfarm.com/
3 Likes
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/mail.jewettfarm.com-0001.conf produced an unexpected error: expected /etc/letsencrypt/live/mail.jewettfarm.com-0001/fullchain.pem to be a symlink. Skipping.
Renewal configuration file /etc/letsencrypt/renewal/mail.jewettfarm.com-0002.conf produced an unexpected error: expected /etc/letsencrypt/live/mail.jewettfarm.com-0002/fullchain.pem to be a symlink. Skipping.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
Certificate Name: mail.jewettfarm.com
Serial Number: 41f3874c2aeb88af271fd3a2db6e26881c4
Key Type: RSA
Domains: mail.jewettfarm.com mail.jewettracing.com mail.mail-lab.us
Expiry Date: 2023-04-11 19:07:27+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/mail.jewettfarm.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/mail.jewettfarm.com/privkey.pem
The following renewal configurations were invalid:
/etc/letsencrypt/renewal/mail.jewettfarm.com-0001.conf
/etc/letsencrypt/renewal/mail.jewettfarm.com-0002.conf
When I went to renew this morning, it just shows the same expired certificate.
Thinking I may need to remove the certbot snap and start over? Somehow I hosed the config apparently.
FWIW: I did not get the "too many certificates" error.
I am also wondering since I run two letsencrypt clients (the proxy node runs a similar process using letsencrypt, that one is working fine... knocking on wood.
ls -l /etc/letsencrypt/live/mail.jewettfarm.com/
-rw-r--r-- 1 zimbra zimbra 692 Jan 11 18:42 README
lrwxrwxrwx 1 root root 45 Apr 15 07:18 cert.pem -> ../../archive/mail.jewettracing.com/cert5.pem
lrwxrwxrwx 1 root root 46 Apr 15 07:18 chain.pem -> ../../archive/mail.jewettracing.com/chain5.pem
lrwxrwxrwx 1 root root 50 Apr 15 07:18 fullchain.pem -> ../../archive/mail.jewettracing.com/fullchain5.pem
lrwxrwxrwx 1 root root 48 Apr 15 07:18 privkey.pem -> ../../archive/mail.jewettracing.com/privkey5.pem
ls -l /etc/letsencrypt/archive/mail.jewettfarm.com/
-rw-r--r-- 1 zimbra zimbra 1862 Apr 14 11:08 cert1.pem
-rw-r--r-- 1 zimbra zimbra 1883 Apr 14 11:08 cert2.pem
-rw-r--r-- 1 zimbra zimbra 1879 Apr 14 11:08 cert3.pem
-rw-r--r-- 1 zimbra zimbra 1911 Apr 14 11:08 cert4.pem
-rw-r--r-- 1 zimbra zimbra 1911 Apr 14 11:08 cert5.pem
-rw-r--r-- 1 root root 1911 Apr 15 07:18 cert6.pem
-rw-r--r-- 1 zimbra zimbra 1826 Apr 14 11:08 chain1.pem
-rw-r--r-- 1 zimbra zimbra 1826 Apr 14 11:08 chain2.pem
-rw-r--r-- 1 zimbra zimbra 1826 Apr 14 11:08 chain3.pem
-rw-r--r-- 1 zimbra zimbra 1826 Apr 14 11:08 chain4.pem
-rw-r--r-- 1 zimbra zimbra 1826 Apr 14 11:08 chain5.pem
-rw-r--r-- 1 root root 1826 Apr 15 07:18 chain6.pem
-rw-r--r-- 1 zimbra zimbra 3688 Apr 14 11:08 fullchain1.pem
-rw-r--r-- 1 zimbra zimbra 3709 Apr 14 11:08 fullchain2.pem
-rw-r--r-- 1 zimbra zimbra 3705 Apr 14 11:08 fullchain3.pem
-rw-r--r-- 1 zimbra zimbra 3737 Apr 14 11:08 fullchain4.pem
-rw-r--r-- 1 zimbra zimbra 3737 Apr 14 11:08 fullchain5.pem
-rw-r--r-- 1 root root 3737 Apr 15 07:18 fullchain6.pem
-rw------- 1 zimbra zimbra 1704 Apr 14 11:08 privkey1.pem
-rw------- 1 zimbra zimbra 1704 Apr 14 11:08 privkey2.pem
-rw------- 1 zimbra zimbra 1704 Apr 14 11:08 privkey3.pem
-rw------- 1 zimbra zimbra 1704 Apr 14 11:08 privkey4.pem
-rw------- 1 zimbra zimbra 1704 Apr 14 11:08 privkey5.pem
-rw------- 1 root zimbra 1704 Apr 15 07:18 privkey6.pem
Going to do a bit more troublshooting, since the email app did not like the private key, I think this is a clue that it's using some other key that is no longer valid.
Thanks
1 Like
Ignorance perhaps?
The process runs the certbot renewal process (using the lines I shared above), then another process converts it into "zimbra friendly" files, so that the "/opt/zimbra/bin/zmcertmgr deploycrt comm" can consume the files without giving an error.
I will use opensll to do some testing, I think the issue is with the private key it is using. - in other words, not even a certbot/letsencrypt issue.
Maybe I can trim the process down to just a "certbot renew" moving forward, after I figure out what broke it.
Thanks again
1 Like
So another issue seems to be the symlinks are pointing to the wrong directory.
cert.pem -> ../../archive/mail.jewettracing.com/cert5.pem
certbot is dropping the renewed certificates into /etc/letsencrypt/archive/mail.jewettfarm.com, yet the symlinks point to archive/mail.jewettracing.com.
Updating those next, and will test.
I'd expect so. When certbot issues a cert, it stores all that information. When you run (ordinarily daily) certbot renew, it checks any existing certs to see if they're within 30 days (by default) of expiration. If they are, it renews them, using the same domains, authenticator, credentials, etc. as were used initially. It can then, following a successful renewal, fire off whatever else you need to do to convert the certs to be "zimbra friendly", which itself can fire off the command to deploy the cert.
I'm not saying this is directly related to your problem, but it does seem that your process can be greatly streamlined.
5 Likes
/etc/letsencrypt/renewal/mail.jewettfarm.com.conf
Would you show contents of this file.
3 Likes
# renew_before_expiry = 30 days
version = 2.5.0
archive_dir = /etc/letsencrypt/archive/mail.jewettfarm.com
cert = /etc/letsencrypt/live/mail.jewettfarm.com/cert.pem
privkey = /etc/letsencrypt/live/mail.jewettfarm.com/privkey.pem
chain = /etc/letsencrypt/live/mail.jewettfarm.com/chain.pem
fullchain = /etc/letsencrypt/live/mail.jewettfarm.com/fullchain.pem
# Options used in the renewal process
[renewalparams]
account = 285543fbfc7c999684644ff0896c6044
preferred_chain = ISRG Root X1
authenticator = dns-cloudflare
server = https://acme-v02.api.letsencrypt.org/directory
key_type = rsa
dns_cloudflare_credentials = /root/.secrets/certbot/cloudflare.ini
I don't think updating the symbolic links in "/etc/letsencrypt/live/mail.jewettfarm.com" is a wise idea, since certbot handles those. So leaving them alone for now.
I did run
openssl verify -CAfile fullchain6.pem cert6.pem
cert6.pem: OK
and the renewed cert checks out fine. As does the new privkey. So it just seems down to the symlinks pointing to the wrong set of files in /etc/letsencrypt/live/mail.jewettfarm.com - causing false renewal requests, and breaking the scripted process. A "certbot certificates" does not pick up the renewed certificate files.
That looks correct. Notice the archive_dir and cert folders are jewettfarm. If you were using certbot renew the folders used are those so should be correct.
You should check you don't have any symlinks of your own that map one folder to another. An earlier post said you copied one set of archive folders but they got "renamed".
What do these show?
ls -l /etc/letsencrypt/live/
ls -l /etc/letsencrypt/archive/
Later,
It looks like you might be rate limited but you could try:
certbot renew --cert-name mail.jewettfarm.com --dry-run
And, then without --dry-run once you are no longer rate limited
I agree with danb35 you should use renew rather than re-issuing your command each time
3 Likes
/live/
-rw-r--r-- 1 zimbra zimbra 740 Aug 26 2022 README
drwxr-xr-x 2 zimbra zimbra 7 Apr 15 07:39 mail.jewettfarm.com
drwxr-xr-x 2 zimbra zimbra 8 Apr 14 15:53 mail.jewettfarm.com-0001
drwxr-xr-x 2 zimbra zimbra 8 Apr 14 16:05 mail.jewettfarm.com-0002
/archive/
drwxr-xr-x 2 zimbra zimbra 26 Apr 15 07:18 mail.jewettfarm.com
drwxr-xr-x 2 zimbra zimbra 6 Apr 14 15:41 mail.jewettfarm.com-0001
drwxr-xr-x 2 zimbra zimbra 8 Apr 14 16:16 mail.jewettfarm.com-0002
drwxr-xr-x 2 zimbra zimbra 22 Jan 11 15:07 mail.jewettracing.com
No manually created symlinks, but yes I did copy them to mail.jewettfarm.com from the mail.jewettracing.com directory, since I thought that was what it was looking for.
I run the certbot process as root, so I wouldn't think the zimbra user being owner should trip it up.
If I were to remove the invalid symlinks, do we know if certbot will create new ones in /live/mail.jewettfarm.com?
It is worth trying. But, I would just fix the symlinks manually instead. Just from the dates it looks like new certs are being created in jewettfarm. So, it is just the symlinks that are wrong.
We could look at this first though:
ls -l /etc/letsencrypt/archive/mail.jewettracing.com/
3 Likes
Ok, may give that a shot, I don't want it to keep trying to pull new certs everyday, since it thinks the cert is expired. Updating the symlinks should show the new valid cert, and stop the renewal loop.
Here is the output of /etc/letsencrypt/archive/mail.jewettracing.com/
drwxr-xr-x 2 zimbra zimbra 22 Jan 11 15:07 ./
drwx------ 6 zimbra zimbra 6 Apr 14 15:57 ../
-rw-r--r-- 1 zimbra zimbra 1862 Sep 26 2022 cert1.pem
-rw-r--r-- 1 zimbra zimbra 1883 Sep 26 2022 cert2.pem
-rw-r--r-- 1 zimbra zimbra 1879 Sep 26 2022 cert3.pem
-rw-r--r-- 1 zimbra zimbra 1911 Oct 29 19:15 cert4.pem
-rw-r--r-- 1 zimbra zimbra 1911 Jan 11 15:07 cert5.pem
-rw-r--r-- 1 zimbra zimbra 1826 Sep 26 2022 chain1.pem
-rw-r--r-- 1 zimbra zimbra 1826 Sep 26 2022 chain2.pem
-rw-r--r-- 1 zimbra zimbra 1826 Sep 26 2022 chain3.pem
-rw-r--r-- 1 zimbra zimbra 1826 Oct 29 19:15 chain4.pem
-rw-r--r-- 1 zimbra zimbra 1826 Jan 11 15:07 chain5.pem
-rw-r--r-- 1 zimbra zimbra 3688 Sep 26 2022 fullchain1.pem
-rw-r--r-- 1 zimbra zimbra 3709 Sep 26 2022 fullchain2.pem
-rw-r--r-- 1 zimbra zimbra 3705 Sep 26 2022 fullchain3.pem
-rw-r--r-- 1 zimbra zimbra 3737 Oct 29 19:15 fullchain4.pem
-rw-r--r-- 1 zimbra zimbra 3737 Jan 11 15:07 fullchain5.pem
-rw------- 1 zimbra zimbra 1704 Sep 26 2022 privkey1.pem
-rw------- 1 zimbra zimbra 1704 Sep 26 2022 privkey2.pem
-rw------- 1 zimbra zimbra 1704 Sep 26 2022 privkey3.pem
-rw------- 1 zimbra zimbra 1704 Oct 29 19:15 privkey4.pem
-rw------- 1 zimbra zimbra 1704 Jan 11 15:07 privkey5.pem
Ok, I manually created new (correct) symlinks in /live/mail.jewettfarm.com that point to /archive/mail.jewettfarm.com
drwxr-xr-x 2 zimbra zimbra 7 Apr 15 10:08 ./
drwx------ 5 zimbra zimbra 6 Apr 14 15:57 ../
-rw-r--r-- 1 zimbra zimbra 692 Jan 11 18:42 README
lrwxrwxrwx 1 root root 54 Apr 15 10:06 cert.pem -> /etc/letsencrypt/archive/mail.jewettfarm.com/cert6.pem
lrwxrwxrwx 1 root root 55 Apr 15 10:06 chain.pem -> /etc/letsencrypt/archive/mail.jewettfarm.com/chain6.pem
lrwxrwxrwx 1 root root 59 Apr 15 10:07 fullchain.pem -> /etc/letsencrypt/archive/mail.jewettfarm.com/fullchain6.pem
lrwxrwxrwx 1 root root 57 Apr 15 10:08 privkey.pem -> /etc/letsencrypt/archive/mail.jewettfarm.com/privkey6.pem
Output of certbot certificates:
Renewal configuration file /etc/letsencrypt/renewal/mail.jewettfarm.com-0001.conf produced an unexpected error: expected /etc/letsencrypt/live/mail.jewettfarm.com-0001/fullchain.pem to be a symlink. Skipping.
Renewal configuration file /etc/letsencrypt/renewal/mail.jewettfarm.com-0002.conf produced an unexpected error: expected /etc/letsencrypt/live/mail.jewettfarm.com-0002/fullchain.pem to be a symlink. Skipping.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
Certificate Name: mail.jewettfarm.com
Serial Number: 3f3d27ba0c404609f20a16775865155cf3b
Key Type: RSA
Domains: mail.jewettfarm.com mail.jewettracing.com mail.mail-lab.us
Expiry Date: 2023-07-14 10:18:46+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/mail.jewettfarm.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/mail.jewettfarm.com/privkey.pem
The following renewal configurations were invalid:
/etc/letsencrypt/renewal/mail.jewettfarm.com-0001.conf
/etc/letsencrypt/renewal/mail.jewettfarm.com-0002.conf
I am a bit worried the symlinks won't update when it saves future certificate files though. So will need to test against the staging servers.
Given your history of problems I would not test with staging for this specific issue. That will create certs but ones that are not valid for actual use. You then would need to get a fresh cert anyway or start manually modifying the renewal conf file and symlinks which is not a good way to test this situation.
You should also delete the failing configs like:
certbot delete --cert-name mail.jewettfarm.com-0001
certbot delete --cert-name mail.jewettfarm.com-0002
4 Likes
Ok, makes sense. Since the certificate did in fact update, I think that process is fine. Now that it recognized the current cert, it should in theory stop issuing new certs.
I have changed the cron script to just run "certbot renew" at the next scheduled run, added my email address so I get expiry notifications, and removed the unnecessary cert configs. As well as updated the symlinks to point to the correct directory.
I made a note to check the symlinks in /live/mail.jewettfarm.com to make sure those update correctly.
Thanks as always for the guidance and support.
2 Likes
TLDR;
Was this covered already?:
3 Likes
Thanks for the heads up. I will make sure the script changes owner to zimbra before it grabs the archive copy to update the cert.
1 Like
Your best bet is to leave the /etc/letsencrypt folders alone.
Make a copy elsewhere and change permissions in that location.
And, you don't need to grab the archive copy if you meant that literally. Just copy from the symlink in live which always points to latest cert set in archive
5 Likes
If you do that, make sure you dereference the symlink (i.e.: copy the actual file to which the symlink points), as the symlinks are relative links. So if you'd copy just the literal symlink, you'd end up with a broken symlink, as the (relative) destination doesn't exist. I'm not familiar with a feature of cp which would automatically update the target of a relative symlink so it'll point to the destination file, even if the symlink is moved to a different location.
3 Likes