AVD
January 26, 2023, 8:33am
43
ls -lr /etc/letsencrypt/live/
total 12
-rw-r--r-- 1 root root 740 Jul 28 03:35 README
drwxr-xr-x 2 root root 4096 Oct 30 14:26 mail.allanimals.info-0001
drwxr-xr-x 2 root root 4096 Jan 15 09:27 mail.allanimals.info
ls -lr /etc/letsencrypt/archive/
total 8
drwxr-xr-x 2 root root 4096 Jan 25 10:05 mail.allanimals.info-0001
drwxr-xr-x 2 root root 4096 Jan 25 10:04 mail.allanimals.info
rg305
January 26, 2023, 8:34am
44
Sorry, that failed to give me all I wanted...
Try:
ls -lr /etc/letsencrypt/live/*
AND
ls -lr /etc/letsencrypt/archive/*
2 Likes
rg305
January 26, 2023, 8:36am
45
Right there I see a problem.
These two have somewhat close dates:
drwxr-xr-x 2 root root 4096 Jan 15 09:27 mail.allanimals.info
drwxr-xr-x 2 root root 4096 Jan 25 10:04 mail.allanimals.info
These are nowhere near to each other:
drwxr-xr-x 2 root root 4096 Oct 30 14:26 mail.allanimals.info-0001
drwxr-xr-x 2 root root 4096 Jan 25 10:05 mail.allanimals.info-0001
Something went awry during that Jan 25 renewal.
2 Likes
AVD
January 26, 2023, 8:37am
46
ls -lr /etc/letsencrypt/live/*
-rw-r--r-- 1 root root 740 Jul 28 03:35 /etc/letsencrypt/live/README
/etc/letsencrypt/live/mail.allanimals.info-0001:
total 0
/etc/letsencrypt/live/mail.allanimals.info:
total 4
-rw-r--r-- 1 root root 692 Oct 30 14:28 README
lrwxrwxrwx 1 root root 52 Jan 15 09:27 privkey.pem -> ../../archive/mail.allanimals.info-0001/privkey1.pem
lrwxrwxrwx 1 root root 54 Jan 15 09:27 fullchain.pem -> ../../archive/mail.allanimals.info-0001/fullchain1.pem
lrwxrwxrwx 1 root root 50 Jan 15 09:27 chain.pem -> ../../archive/mail.allanimals.info-0001/chain1.pem
lrwxrwxrwx 1 root root 49 Jan 15 09:27 cert.pem -> ../../archive/mail.allanimals.info-0001/cert1.pem
ls -lr /etc/letsencrypt/archive/*
/etc/letsencrypt/archive/mail.allanimals.info-0001:
total 16
-rw------- 1 root root 1704 Oct 30 14:28 privkey1.pem
-rw-r--r-- 1 root root 3684 Oct 30 14:28 fullchain1.pem
-rw-r--r-- 1 root root 3765 Oct 30 14:48 chain1.pem
-rw-r--r-- 1 root root 1858 Oct 30 14:28 cert1.pem
/etc/letsencrypt/archive/mail.allanimals.info:
total 76
-rw------- 1 root root 1704 Oct 29 18:03 privkey4.pem
-rw------- 1 root root 1704 Sep 29 19:36 privkey3.pem
-rw------- 1 root root 1704 Jul 31 16:09 privkey2.pem
-rw------- 1 root root 1704 Jul 28 03:35 privkey1.pem
-rw-r--r-- 1 root root 5607 Oct 29 18:03 fullchain4.pem
-rw-r--r-- 1 root root 3680 Sep 29 19:36 fullchain3.pem
-rw-r--r-- 1 root root 3680 Jul 31 16:09 fullchain2.pem
-rw-r--r-- 1 root root 5608 Jul 28 03:35 fullchain1.pem
-rw-r--r-- 1 root root 5688 Oct 30 13:16 chain4.pem
-rw-r--r-- 1 root root 1826 Sep 29 19:36 chain3.pem
-rw-r--r-- 1 root root 3765 Jul 31 16:17 chain2.pem
-rw-r--r-- 1 root root 3750 Jul 28 03:35 chain1.pem
-rw-r--r-- 1 root root 1858 Oct 29 18:03 cert4.pem
-rw-r--r-- 1 root root 1854 Sep 29 19:36 cert3.pem
-rw-r--r-- 1 root root 1854 Jul 31 16:09 cert2.pem
-rw-r--r-- 1 root root 1858 Jul 28 03:35 cert1.pem
rg305
January 26, 2023, 8:41am
47
OK, the picture has been painted.
I see it clearly now:
/etc/letsencrypt/live/mail.allanimals.info-0001:
total 0
/etc/letsencrypt/live/mail.allanimals.info:
total 4
-rw-r--r-- 1 root root 692 Oct 30 14:28 README
lrwxrwxrwx 1 root root 52 Jan 15 09:27 privkey.pem -> ../../archive/mail.allanimals.info-0001/privkey1.pem
lrwxrwxrwx 1 root root 54 Jan 15 09:27 fullchain.pem -> ../../archive/mail.allanimals.info-0001/fullchain1.pem
lrwxrwxrwx 1 root root 50 Jan 15 09:27 chain.pem -> ../../archive/mail.allanimals.info-0001/chain1.pem
lrwxrwxrwx 1 root root 49 Jan 15 09:27 cert.pem -> ../../archive/mail.allanimals.info-0001/cert1.pem
The -0001 is empty.
And the original one points to the -0001 cert!
A good mess indeed.
OK. let's blow away the symlinks and recreate them.
The files seem to be in tact.
2 Likes
AVD
January 26, 2023, 8:47am
48
Ok. I'll make a copy of the directory first, just in case. Could you provide the command to redirect the sym links to where they should be pointing please.
rg305
January 26, 2023, 8:49am
49
Let's take the slow and steady path:
#1 unlink the incorrect links:
unlink /etc/letsencrypt/live/mail.allanimals.info/privkey.pem
unlink /etc/letsencrypt/live/mail.allanimals.info/fullchain.pem
unlink /etc/letsencrypt/live/mail.allanimals.info/chain.pem
unlink /etc/letsencrypt/live/mail.allanimals.info/cert.pem
Check they have been removed:
ls -l /etc/letsencrypt/live/mail.allanimals.info/
2 Likes
AVD
January 26, 2023, 8:52am
50
unlink /etc/letsencrypt/live/mail.allanimals.info/privkey.pem
unlink /etc/letsencrypt/live/mail.allanimals.info/fullchain.pem
unlink /etc/letsencrypt/live/mail.allanimals.info/chain.pem
unlink /etc/letsencrypt/live/mail.allanimals.info/cert.pem
ls -l /etc/letsencrypt/live/mail.allanimals.info/
total 4
-rw-r--r-- 1 root root 692 Oct 30 14:28 README
rg305
January 26, 2023, 8:54am
51
#2 relink the files:
cd /etc/letsencrypt/live/mail.allanimals.info/
ln -s ../../archive/mail.allanimals.info/privkey4.pem privkey.pem
ln -s ../../archive/mail.allanimals.info/fullchain4.pem fullchain.pem
ln -s ../../archive/mail.allanimals.info/chain4.pem chain.pem
ln -s ../../archive/mail.allanimals.info/cert4.pem cert.pem
cd /etc/letsencrypt/live/mail.allanimals.info-0001/
ln -s ../../archive/mail.allanimals.info-0001/privkey1.pem privkey.pem
ln -s ../../archive/mail.allanimals.info-0001/fullchain1.pem fullchain.pem
ln -s ../../archive/mail.allanimals.info-0001/chain1.pem chain.pem
ln -s ../../archive/mail.allanimals.info-0001/cert1.pem cert.pem
Then make sure they are working:
certbot certificates
2 Likes
rg305
January 26, 2023, 8:56am
52
#3 delete the cert you are not using/needing
certbot delete --cert-name mail.allanimals.info-0001
2 Likes
rg305
January 26, 2023, 8:56am
53
#4 renew the remaining cert [if needed]
certbot renew
2 Likes
rg305
January 26, 2023, 9:11am
55
AVD:
Renewal configuration file /etc/letsencrypt/renewal/mail.allanimals.info-0001.conf produced an unexpected error: fullchain does not match cert + chain for mail.allanimals.info-0001!. Skipping.
Renewal configuration file /etc/letsencrypt/renewal/mail.allanimals.info.conf produced an unexpected error: fullchain does not match cert + chain for mail.allanimals.info!. Skipping.
How does this happen?!?!?!
2 Likes
rg305
January 26, 2023, 9:11am
56
One at a time:
ls -l /etc/letsencrypt/live/mail.allanimals.info/
2 Likes
AVD
January 26, 2023, 9:12am
58
/etc/letsencrypt/live/mail.allanimals.info-0001# ls -l /etc/letsencrypt/live/mail.allanimals.info/
total 4
lrwxrwxrwx 1 root root 44 Jan 26 08:56 cert.pem -> ../../archive/mail.allanimals.info/cert4.pem
lrwxrwxrwx 1 root root 45 Jan 26 08:56 chain.pem -> ../../archive/mail.allanimals.info/chain4.pem
lrwxrwxrwx 1 root root 49 Jan 26 08:56 fullchain.pem -> ../../archive/mail.allanimals.info/fullchain4.pem
lrwxrwxrwx 1 root root 47 Jan 26 08:56 privkey.pem -> ../../archive/mail.allanimals.info/privkey4.pem
-rw-r--r-- 1 root root 692 Oct 30 14:28 README
rg305
January 26, 2023, 9:14am
59
I see a problem we missed before:
That chain is from the following day!
2 Likes
AVD
January 26, 2023, 9:16am
60
Check this config file (there is no 4 after the names):
cat /etc/letsencrypt/renewal/mail.allanimals.info.conf
renew_before_expiry = 30 days
version = 1.32.2
archive_dir = /etc/letsencrypt/archive/mail.allanimals.info
cert = /etc/letsencrypt/live/mail.allanimals.info/cert.pem
privkey = /etc/letsencrypt/live/mail.allanimals.info/privkey.pem
chain = /etc/letsencrypt/live/mail.allanimals.info/chain.pem
fullchain = /etc/letsencrypt/live/mail.allanimals.info/fullchain.pem
Options used in the renewal process
[renewalparams]
account = 994651a718b257d6f21dd3959b6cc9d6
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
key_type = rsa
preferred_chain = ISRG Root X1
rg305
January 26, 2023, 9:16am
61
Time for the surgical gloves the come on...
cat /etc/letsencrypt/archive/mail.allanimals.info/fullchain4.pem
2 Likes
AVD
January 26, 2023, 9:18am
62
cat /etc/letsencrypt/archive/mail.allanimals.info/fullchain4.pem
-----BEGIN CERTIFICATE-----
MIIFLzCCBBegAwIBAgISBKLHd2SRuQFpFyFDxr0JXbIuMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjEwMjkxNzAzMTdaFw0yMzAxMjcxNzAzMTZaMB8xHTAbBgNVBAMT
FG1haWwuYWxsYW5pbWFscy5pbmZvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAw8ouH5QANGqlMwoaqEnH1zbkDjOCVLCqEc5WD1d1kQEbkU8ANZKv5lsL
tVkabEKmhOWHXWqHxY/BHe/D/nEQj/4IMQ3LK9GmwHoRJOGDaO4aJjcdwT3w9Ksq
DJZeCAxn7cvoLn1Ayy4p/Y0uhlH61dWD/dIYjtuhNcxR5BV39dsXdj9Q0GZxr7+o
fmY78hnF1G47Zm5ffp2Tbbzv2lKP0mB2F3PCx8w6mXhkg3n9W6hHK5+aYZVekCfV
I3odj54MwGkuTkkJ+5kXYC07ZJVfZRy7uOSknTu2q3W30sTX9499yM+bSgBxkn7y
UmL+cLcqk8V8Vu4tAkFtZ8R5bYKyWQIDAQABo4ICUDCCAkwwDgYDVR0PAQH/BAQD
AgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAA
MB0GA1UdDgQWBBRThCV5agQSvr16Dj5BWdLGtczxkDAfBgNVHSMEGDAWgBQULrMX
t1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0
dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVu
Y3Iub3JnLzAfBgNVHREEGDAWghRtYWlsLmFsbGFuaW1hbHMuaW5mbzBMBgNVHSAE
RTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRw
Oi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQUGCisGAQQB1nkCBAIEgfYEgfMA8QB2
AHoyjFTYty22IOo44FIe6YQWcDIThU070ivBOlejUutSAAABhCTpkoYAAAQDAEcw
RQIhAOy1tfkXWd190cwhxWXYLZbbc+xsIFcP+VoQfjOjrftfAiBusHjFHP1VAP9+
VyrnSLKyq5Vdbvuo1IfggEWCgSdjtgB3AK33vvp8/xDIi509nB4+GGq0Zyldz7EM
JMqFhjTr3IKKAAABhCTpkpMAAAQDAEgwRgIhAKSamiHvq0QTB8YsdDAD3gllzvsQ
cDGHwUf//KvZ0iILAiEAjwGJ7RECGTLdj6JwKtT6FH9lamDQR0t+9DxROAVjS2ww
DQYJKoZIhvcNAQELBQADggEBAKGObB6OKGIfiGYJQo9T++AhbnN2o3HcG2nK3bJF
rqOYx8wLJQ/ZEq8G0OqXA7vsqfVLounvYae3HLgGnvn65leWLIwfsoHLdIsfLB3P
EC/YyINQhoYAQYmZ1n9KteSO5KwTt0R3nDCgchE1KPfa2R26mRm7WBExbPKdXXDI
clzc8FSqeyG2bqEPAQwLzWtCkg1GzG3vBTpoq34GW36UoKNqvPyeNtaGi2ND/wGJ
Uyf23bAaabGtylprK7eIaTea2P9tEZlCOPzJam7DCRl19VfOXA3sKyLhIVC7g2f3
gos+WsxyWeonZrL2giDAFE5sbv+lUDldLB8kWDMsU1evH04=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----