We got the version from the epel-repo:"
certbot.noarch 2.6.0-1.el9 @repo_epel
python3-certbot.noarch 2.6.0-1.el9 @repo_epel
This file is 2147 bytes in size.
If I check the size of the "fullchain.pem" from:
It's just 1517 bytes..
So why would a cat on a file which should be 2147 bytes result in just 1517 bytes of output?
Hm, 2147 from the ls -l output is exactly the same size as cert20.pem.. That said, the Let's Encrypt ACME server did send the full chain, because otherwise Certbot could not have written 1826 bytes worth of chain20.pem, as the ACME server doesn't send the chain separately.. You can check this in the letsencrypt.log file: it should provide the entire certificate chain in the log.
I agree with above that something must be modifying the files somehow.
5 Likes
Here you have shown the fullchain.pem in the directory of ac2-afname.test-facet.onl .
However, the listing of the files are from the directory ac1-keycloak.test-facet.onl.
And the size is not consistent, as @Osiris discovered.
The difference is visible from the Internet as well:
tumbleweed:~ # openssl s_client -connect ac2-afname.test-facet.onl:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ac2-afname.test-facet.onl
verify return:1
---
Certificate chain
0 s:CN = ac2-afname.test-facet.onl
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
v:NotBefore: Feb 28 10:08:02 2024 GMT; NotAfter: May 28 10:08:01 2024 GMT
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
---
The other one:
tumbleweed:~ # openssl s_client -connect ac1-keycloak.test-facet.onl:443
CONNECTED(00000003)
depth=0 CN = ac1-keycloak.test-facet.onl
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = ac1-keycloak.test-facet.onl
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = ac1-keycloak.test-facet.onl
verify return:1
---
Certificate chain
0 s:CN = ac1-keycloak.test-facet.onl
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
v:NotBefore: Feb 29 13:54:35 2024 GMT; NotAfter: May 29 13:54:34 2024 GMT
---
Please provide information only from the system where the chain is broken and not from the system where everything is perfect. Content of configuration file, listing of directory, verification of possible extra program that may truncate the chain, certbot version, etc...
3 Likes
I would check to see if any of the installation files were modified.
I think this is the release you are running:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2204238
4 Likes
Because we badly needed that site to work for acceptance testing, we fixed ac2-afname.test-facet.onl manually. I agree that's unfortunate.
Can you tell me how I can force a renewal of the ac1-keycloak certificate, and let certbot show me the output of the manual_auth_hook and manual_cleanup_hook?
I used this command, but it didn't show me:
certbot certonly -d ac1-keycloak.test-facet.onl --force-renewal -v
I added cat statements to see if and where in the process the fullchain.pem was still having 2 certificates.
Check-up first, then the action. Could you provide please:
cat /etc/letsencrypt/renewal/ac1-keycloak.test-facet.onl.conf
Let's assume that the directories are correct in the config file, then the following too:
ls -l /etc/letsencrypt/live/ac1-keycloak.test-facet.onl
ls -l /etc/letsencrypt/archive/ac1-keycloak.test-facet.onl
And the content of the /etc/crontab file.
2 Likes
[root@bhrod0004-salt letsencrypt]# cat /etc/letsencrypt/renewal/ac1-keycloak.test-facet.onl.conf
# renew_before_expiry = 30 days
version = 2.6.0
archive_dir = /etc/letsencrypt/archive/ac1-keycloak.test-facet.onl
cert = /etc/letsencrypt/live/ac1-keycloak.test-facet.onl/cert.pem
privkey = /etc/letsencrypt/live/ac1-keycloak.test-facet.onl/privkey.pem
chain = /etc/letsencrypt/live/ac1-keycloak.test-facet.onl/chain.pem
fullchain = /etc/letsencrypt/live/ac1-keycloak.test-facet.onl/fullchain.pem
# Options used in the renewal process
[renewalparams]
authenticator = manual
account = bd9492d7c2d0d5be9c8826f90d386352
manual_auth_hook = /srv/beheer/letsencrypt/certbot-authenticator.sh
server = https://acme-v02.api.letsencrypt.org/directory
manual_cleanup_hook = /srv/beheer/letsencrypt/certbot-removetxt.sh
rsa_key_size = 4096
pref_challs = dns-01,
key_type = rsa
[root@bhrod0004-salt letsencrypt]# ls -l /etc/letsencrypt/live/ac1-keycloak.test-facet.onl
total 8
-rw-r--r--. 1 root root 692 Dec 21 2020 README
lrwxrwxrwx. 1 root root 52 Mar 1 09:42 cert.pem -> ../../archive/ac1-keycloak.test-facet.onl/cert22.pem
lrwxrwxrwx. 1 root root 53 Mar 1 09:42 chain.pem -> ../../archive/ac1-keycloak.test-facet.onl/chain22.pem
lrwxrwxrwx. 1 root root 68 Mar 1 09:42 fullchain.pem -> /etc/letsencrypt/archive/ac1-keycloak.test-facet.onl/fullchain22.pem
lrwxrwxrwx. 1 root root 55 Mar 1 09:42 privkey.pem -> ../../archive/ac1-keycloak.test-facet.onl/privkey22.pem
[root@bhrod0004-salt letsencrypt]# ls -l /etc/letsencrypt/archive/ac1-keycloak.test-facet.onl
total 96
-rw-r--r--. 1 root root 2147 Aug 31 2023 cert17.pem
-rw-r--r--. 1 root root 2147 Oct 30 23:11 cert18.pem
-rw-r--r--. 1 root root 2147 Dec 29 23:09 cert19.pem
-rw-r--r--. 1 root root 2147 Feb 27 23:04 cert20.pem
-rw-r--r--. 1 root root 2143 Feb 29 15:54 cert21.pem
-rw-r--r--. 1 root root 2147 Mar 1 09:42 cert22.pem
-rw-r--r--. 1 root root 3749 Aug 31 2023 chain17.pem
-rw-r--r--. 1 root root 3749 Oct 30 23:11 chain18.pem
-rw-r--r--. 1 root root 3749 Dec 29 23:09 chain19.pem
-rw-r--r--. 1 root root 1826 Feb 27 23:04 chain20.pem
-rw-r--r--. 1 root root 1826 Feb 29 15:54 chain21.pem
-rw-r--r--. 1 root root 1826 Mar 1 09:42 chain22.pem
-rw-r--r--. 1 root root 3973 Aug 31 2023 fullchain17.pem
-rw-r--r--. 1 root root 3973 Oct 30 23:11 fullchain18.pem
-rw-r--r--. 1 root root 3973 Dec 29 23:09 fullchain19.pem
-rw-r--r--. 1 root root 2147 Feb 27 23:04 fullchain20.pem
-rw-r--r--. 1 root root 2143 Feb 29 15:54 fullchain21.pem
-rw-r--r--. 1 root root 2147 Mar 1 09:42 fullchain22.pem
-rw-------. 1 root root 3272 Aug 31 2023 privkey17.pem
-rw-------. 1 root root 3268 Oct 30 23:11 privkey18.pem
-rw-------. 1 root root 3272 Dec 29 23:09 privkey19.pem
-rw-------. 1 root root 3268 Feb 27 23:04 privkey20.pem
-rw-------. 1 root root 3272 Feb 29 15:54 privkey21.pem
-rw-------. 1 root root 3272 Mar 1 09:42 privkey22.pem
[root@bhrod0004-salt letsencrypt]# cat /etc/cron.d/certbot
0 23 * * * root /bin/certbot renew >/srv/beheer/cron-certbot.out 2>&1
The fullchain is the exact same size as the leaf cert.
I've never seen that happen before!
It must be doing something weird.
This is also unusual:
1 Like
I want to find out if there's a stage in the whole renewal process at which the fullchain still has both certs. But running a forced renewal command doesn't show me the output of the manual_auth_hook and manual_cleanup_hook...
You may want to read the code of manual_cleanup_hook, does it have the string fullchain, or does it call some other program?
You may also want to compare the same programs between the two systems, are they realy the same?
4 Likes
What does the command:
rpm -V certbot fontawesome-fonts python-josepy-doc python3-acme python3-certbot python3-cffi python3-configargparse python3-configobj python3-cryptography python3-josepy python3-parsedatetime python3-ply python3-pyOpenSSL python3-pycparser python3-pyrfc3339 python3-pytz
give as output?
3 Likes
The rpm command gives this output:
S.5....T. c /etc/letsencrypt/cli.ini
Our manual-cleanup-hook didn't contain the string "fullchain" initially, but I added a
cat /etc/letsencrypt/live/$CERTBOT_DOMAIN/fullchain.pem
to find out if the fullchain does contain both certificates when the cleanup-hook is run.
The only program the hook calls is nsupdate, to delete the TXT record.
You mention two systems. I think there's a misunderstanding. We run certbot from only 1 system. We have 3 categories of certificates, maybe that's what's confusing:
- certificates which haven't been renewed yet after the issue first occurred
- certificates that were manually fixed by us
- certificates that were renewed and are now broken (like ac1-keycloak.test-facet.onl)
You did not fill up the original form, so some information is still missing. What kind of web server are you using?
Could you show as well the configuration of the web server concerning the referrence to the certificates, pelase?
3 Likes
We run:
nginx version: nginx/1.24.0
This is the part where the certificate is configured:
ssl_certificate /etc/nginx/ssl-cert/ac1-keycloak_test-facet_onl.pem;
ssl_certificate_key /etc/nginx/ssl-cert/ac1-keycloak_test-facet_onl.key;
Where can I find the form you are referring to?
When you open the issue originally. If you did not open in the help category but in a different one, then you could not see it.
Can you execute please the following commands in a sequence and provide me the output:
ls -ld /etc/nginx/ssl-cert
ls -l /etc/nginx/ssl-cert/ac1-keycloak_test-facet_onl.pem
ls -lL /etc/nginx/ssl-cert/ac1-keycloak_test-facet_onl.pem
cat -te /etc/nginx/ssl-cert/ac1-keycloak_test-facet_onl.pem
3 Likes
Unfortunately, we had to manually fix the certificate for ac1-keycloak because an important chain test relies on it.
I will give you the requested information for a certificate which suffers from the same issue:
[root@ac1od0004 nginx]# ls -ld /etc/nginx/ssl-cert
drwxr-xr-x. 2 root root 4096 Feb 27 23:08 /etc/nginx/ssl-cert
[root@ac1od0004 nginx]# ls -l /etc/nginx/ssl-cert/ac1-opgavenetalage_test-facet_onl.pem
-rw-r--r--. 1 root root 2163 Feb 27 23:08 /etc/nginx/ssl-cert/ac1-opgavenetalage_test-facet_onl.pem
[root@ac1od0004 nginx]# ls -lL /etc/nginx/ssl-cert/ac1-opgavenetalage_test-facet_onl.pem
-rw-r--r--. 1 root root 2163 Feb 27 23:08 /etc/nginx/ssl-cert/ac1-opgavenetalage_test-facet_onl.pem
[root@ac1od0004 nginx]# cat -te /etc/nginx/ssl-cert/ac1-opgavenetalage_test-facet_onl.pem
-----BEGIN CERTIFICATE-----$
MIIGEDCCBPigAwIBAgISA3CjGhDMOkTEtaCp/Hu/7SmiMA0GCSqGSIb3DQEBCwUA$
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD$
EwJSMzAeFw0yNDAyMjcyMTA3MzdaFw0yNDA1MjcyMTA3MzZaMCwxKjAoBgNVBAMT$
IWFjMS1vcGdhdmVuZXRhbGFnZS50ZXN0LWZhY2V0Lm9ubDCCAiIwDQYJKoZIhvcN$
AQEBBQADggIPADCCAgoCggIBAK19xT5gP7QKI7Jil3R2i6sb/CKWvyTkZkS03EhN$
/kUeQSXfQJOWz+AEnjD9lhWILHKcX/OQAmYJBhPJFvBJ31TyruodABaPyFkWu5Lv$
XsBj0ovQDMx0wy+Qv18D7+ZvV7AKWSOBruGuVDMCObF5LI9smrscdJvstzBnKqij$
LNxXiWE1dJbQDb/xhJnXVh192P8i9vI0RLAPuo9xDGo+2F+zfqtouXyg4jLPI9zI$
1AY+KLmQuLMmtdZ04PE/+LWeiEqMXLItuRBmyptV3Q0nVBYBDhd7d/3Rkr78OT/M$
1S2cz9xSvLt5qn6B25/ZQ6OqTFy4Rf1jvLFE0K8vgLF047icInKfQXAlsfxGBtIk$
QzTAI2nYrGuTD5h9fAjtvbqc5DLuzwBBomZwlaYuxjnaKJ5C6E2N8RB+2nP2R5x7$
3KCRWcjVcvseU+o2VOQJyCoE9hj0iGk8Rs+mEylhkLT9KtmMBXBLZ/G2SYKGbaEI$
g3EN4xkrr2HjxPRT7m6kdpx+9FWJuvPxj1bTYNj2KPJGzNY737D32moCsoXawB0X$
tzwDZ3xaA0LFBxpXQ1i+VkaAMDJt+lDIo2QVvL1ww4V5jISH3YjPaaTXfRUPR8H0$
00jrzxaBIBleMLB+EaQcO+Gkieu229RMi16P1pa+5BxzxBeiOgJqmL8QYSoKe3TP$
aUJ9AgMBAAGjggIkMIICIDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYB$
BQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFMy6l/h4TzNC$
STYAZ2DhWKOSzdIsMB8GA1UdIwQYMBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUG$
CCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3Jn$
MCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5vcmcvMCwGA1UdEQQlMCOC$
IWFjMS1vcGdhdmVuZXRhbGFnZS50ZXN0LWZhY2V0Lm9ubDATBgNVHSAEDDAKMAgG$
BmeBDAECATCCAQUGCisGAQQB1nkCBAIEgfYEgfMA8QB3ADtTd3U+LbmAToswWwb+$
QDtn2E/D9Me9AA0tcm/h+tQXAAABjeyb6b8AAAQDAEgwRgIhAK1N7ie/V/TT6fz7$
r1q5163IxaIe35XMXTsSzpBn0T69AiEAlnpOCpxV3sHjtBm6zZKFWvwy745iMKmZ$
CNi5mLukx10AdgBIsONr2qZHNA/lagL6nTDrHFIBy1bdLIHZu7+rOdiEcwAAAY3s$
m+m5AAAEAwBHMEUCIBvha5zhfECpTi7xO3nx94SDvbXGJXK7q6pKRXsT3lLhAiEA$
uB0Uihbo5n3rdboPBNIAWiZOIJfb1txeO9r8DE0BdV0wDQYJKoZIhvcNAQELBQAD$
ggEBALZlJOY6Dt5nrI4iwBSSxLDuVvW5hkQci6NwYO4Wc7FIw+CAYtpPKD9rivZK$
34gwerHw9C8ea7LnpqY1P7Q9uA7PUYjDqNWupHfol4uhiOrhSAgh6xaNWYdQyAL/$
oiKfdpazUoXB3xiOfQOxZzPn3rrL7RdBn/FesjpMvnyvn8lOcFs7KUCH3IFKrN03$
G4KOGHw762AEVGk1UzAHZGz5trVDO1mPk7k4pEct2r+IeXR1X1ZSTr5SWuNPWc09$
izg+dz2Pk0c9mdUJBMeoY6tqeW+Eiql1CcO4LptK41n9MgcXVVqu5tNu1pLRnTFj$
llDXo6/nWI2MwxvluYmkZKFyxlA=$
-----END CERTIFICATE-----$
[root@ac1od0004 nginx]#
This is the directory listing for the live and archive folder of that certificate:
[root@bhrod0004-salt ac1-opgavenetalage.test-facet.onl]# ls -la
total 28
drwxr-xr-x. 2 root root 4096 Feb 27 23:07 .
drwx------. 105 root root 12288 Feb 28 10:22 ..
-rw-r--r--. 1 root root 543 Apr 25 2018 README
lrwxrwxrwx. 1 root root 58 Feb 27 23:07 cert.pem -> ../../archive/ac1-opgavenetalage.test-facet.onl/cert36.pem
lrwxrwxrwx. 1 root root 59 Feb 27 23:07 chain.pem -> ../../archive/ac1-opgavenetalage.test-facet.onl/chain36.pem
lrwxrwxrwx. 1 root root 74 Feb 27 23:07 fullchain.pem -> /etc/letsencrypt/archive/ac1-opgavenetalage.test-facet.onl/fullchain36.pem
lrwxrwxrwx. 1 root root 61 Feb 27 23:07 privkey.pem -> ../../archive/ac1-opgavenetalage.test-facet.onl/privkey36.pem
[root@bhrod0004-salt ac1-opgavenetalage.test-facet.onl]# ls -la /etc/letsencrypt/archive/ac1-opgavenetalage.test-facet.onl
total 112
drwxr-xr-x. 2 root root 4096 Feb 27 23:07 .
drwx------. 105 root root 12288 Jan 22 08:59 ..
-rw-r--r--. 1 root root 2240 Apr 12 2023 cert31.pem
-rw-r--r--. 1 root root 2240 Jun 11 2023 cert32.pem
-rw-r--r--. 1 root root 2163 Aug 31 2023 cert33.pem
-rw-r--r--. 1 root root 2163 Oct 30 23:15 cert34.pem
-rw-r--r--. 1 root root 2159 Dec 29 23:13 cert35.pem
-rw-r--r--. 1 root root 2163 Feb 27 23:07 cert36.pem
-rw-r--r--. 1 root root 3749 Apr 12 2023 chain31.pem
-rw-r--r--. 1 root root 3749 Jun 11 2023 chain32.pem
-rw-r--r--. 1 root root 3749 Aug 31 2023 chain33.pem
-rw-r--r--. 1 root root 3749 Oct 30 23:15 chain34.pem
-rw-r--r--. 1 root root 3749 Dec 29 23:13 chain35.pem
-rw-r--r--. 1 root root 1826 Feb 27 23:07 chain36.pem
-rw-r--r--. 1 root root 4066 Apr 12 2023 fullchain31.pem
-rw-r--r--. 1 root root 4066 Jun 11 2023 fullchain32.pem
-rw-r--r--. 1 root root 3989 Aug 31 2023 fullchain33.pem
-rw-r--r--. 1 root root 3989 Oct 30 23:15 fullchain34.pem
-rw-r--r--. 1 root root 3985 Dec 29 23:13 fullchain35.pem
-rw-r--r--. 1 root root 2163 Feb 27 23:07 fullchain36.pem
-rw-r--r--. 1 root root 3272 Apr 12 2023 privkey31.pem
-rw-r--r--. 1 root root 3272 Jun 11 2023 privkey32.pem
-rw-r--r--. 1 root root 3268 Aug 31 2023 privkey33.pem
-rw-r--r--. 1 root root 3272 Oct 30 23:15 privkey34.pem
-rw-r--r--. 1 root root 3268 Dec 29 23:13 privkey35.pem
-rw-r--r--. 1 root root 3268 Feb 27 23:07 privkey36.pem
Please let me know if you need more information for this certificate.
That is fine if we continue the investigation with the ac1-opgavenetalage.test-facet.onl domain.
I am wondering what copied the pem file 1 minute after its creation from /etc/letsencrypt/... to the /etc/nginx/ssl-cert directory? I see no reference to nginx in the certbot's config file. Do you know the command flow, by chance?
3 Likes
I found the cause of all this:
# Make sure the certificate and private key files are
# never world readable, even just for an instant while
# we're copying them into daemon_cert_root.
# Before copying, we have to remove the expired root certificate DST Root CA X3
# Note: the LECERTFILE is a symbolic link to the latest archive file
LINE=`grep -n BEGIN $LECERTFILE | tail -1 | cut -d: -f1`
LINE=$((LINE-1))
head -n $LINE $LECERTFILE > $LECERTFILE.tmp
mv $LECERTFILE.tmp $LECERT_ARCHIVEFILE
umask 077
cp $LECERTFILE $GITCERTFILE
cp $LEKEYFILE $GITKEYFILE
I found this code snippet in the script certbot-pushanddeploycerts.sh. It's in the /etc/letsencrypt/renewal-hooks/deploy/ directory.
I think this code worked when there were 3 certificates in the fullchain.pem file. But in the new chain situation, it removes a certificate whereas it shouldn't.
I changed the script to only perform the umask + 2 cp commands and now we have a working situation again! 
4 Likes