Since January 28th of this year we are experiencing strange behavior after a certificate renewal:
the chain.pem file that is created only contains the intermediate certificate (C = US, O = Let's Encrypt, CN = R3). This used to be the intermediate certificate + the root CA certificate.
the fullchain.pem only contains the renewed server certificate. This used to be: renewed server cert + intermediate + root.
Because of this modified behavior we are facing issues in letting our webservers use the renewed certificates.
We use certbot version 2.6.0
Our cli.ini:
rsa-key-size = 4096
email = **************
agree-tos = Yes
authenticator = manual
preferred-challenges = dns
manual-auth-hook = /srv/beheer/letsencrypt/certbot-authenticator.sh
manual-cleanup-hook = /srv/beheer/letsencrypt/certbot-removetxt.sh
account = bd9492d7c2d0d5be9c8826f90d386352
there never been root certificate in chain, but DST X3 signed cross signed certificate but it expire soonShortening the Let's Encrypt Chain of Trust
6 Likes
Osiris
February 28, 2024, 12:11pm
3
The chain has been changed recently. Please see the API announcement category.
Please show the fullchain.pem, as I highly doubt this.
3 Likes
@ErwinFACET welcome to the community!
Apart of the root not being to a real root but a cross-signe intermidate to another CA's root, the intermediate by your naming supposed to be still there with the subject "R3".
3 Likes
Thanks for the quick responses. This is the content of the fullchain.pem of one of the renewed certificates:
# cat fullchain.pem
-----BEGIN CERTIFICATE-----
MIIF/zCCBOegAwIBAgISBHob90Ekacz5nfQymHF+B5ioMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yNDAyMjcyMTIwMjFaFw0yNDA1MjcyMTIwMjBaMCUxIzAhBgNVBAMT
GmFjMi1wb3J0YWFsLnRlc3QtZmFjZXQub25sMIICIjANBgkqhkiG9w0BAQEFAAOC
<cut>
<cut>
WlgXNhzfKJUtZE7cPfyKXZ9AFxdKyr51UPGQyw6kvRoEGUpu4uPAi6KKYH9oW3mc
7+5qhLWKN5Jc/USLenDQ0Deluptw+1nf7DqpD9il+CFfkrPtMli6qkKszGZL914m
RPvprQp6kDvRnSK4ZHE1AypaeCF93/1nMlr+Yf2/V3A/Km127KTecoulZ/fdPQTi
5TwU
-----END CERTIFICATE-----
We see in the letsencrypt.log that the letsencrypt API does in fact return both the server certificate and the intermediate certificate:
2024-02-27 23:04:00,035:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/cert/0445ccdcfe10be9e00843da836f545dd7b7f HTTP/1.1" 200 3974
2024-02-27 23:04:00,035:DEBUG:acme.client:Received response:
HTTP 200
<cut>
<server certificate>
<intermediate certificate>
but somehow they're not both being written to the fullchain.pem.
1 Like
Nekit
February 28, 2024, 4:06pm
6
Are you certain your fullchain doesn't have multiples of BEGIN/END CERTIFICATE? Could you post the entirety of this file? It doesn't contain any private information in it (in fact those contents could be looked up in the public CT logs)
3 Likes
Please verify that neither of these scripts modify the fullchain.pem file. We've seen users that used scripts to remove the last certificate in this file - these broke after the chain change.
9 Likes
This is one of the full chain files that were 'affected':
[root@bhrod0004-salt ac2-afname.test-facet.onl]# cat fullchain.pem
-----BEGIN CERTIFICATE-----
MIIEMzCCAxugAwIBAgISA9NXC2zPQpA+Rzb/IZ1poDhoMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yNDAyMjgxMDA4MDJaFw0yNDA1MjgxMDA4MDFaMCQxIjAgBgNVBAMT
GWFjMi1hZm5hbWUudGVzdC1mYWNldC5vbmwwWTATBgcqhkjOPQIBBggqhkjOPQMB
BwNCAATdjfZ/Eidq0jP0Eia3dsxu/YnzJjVLs8Ss+UKn2vKXsy0AuIklGGvo4Y6S
OOp5hOQAaQf6cgvA0Fb27zZt90YEo4ICGjCCAhYwDgYDVR0PAQH/BAQDAgeAMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1Ud
DgQWBBQIbEuEfOx20+GODmHLTZ4UXB0ZRzAfBgNVHSMEGDAWgBQULrMXt1hWy65Q
CUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6Ly9y
My5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub3Jn
LzAkBgNVHREEHTAbghlhYzItYWZuYW1lLnRlc3QtZmFjZXQub25sMBMGA1UdIAQM
MAowCAYGZ4EMAQIBMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHYASLDja9qmRzQP
5WoC+p0w6xxSActW3SyB2bu/qznYhHMAAAGN72ZnuAAABAMARzBFAiEAqO5tFDRS
k0uzjnP3gD2pNoaMTbjBUUmVwUqw/8lAh/MCIE8OOF7pXySBe3foVuzROGZGYXK6
7FGRoJasDrVgKVYcAHUA7s3QZNXbGs7FXLedtM0TojKHRny87N7DUUhZRnEftZsA
AAGN72ZnvAAABAMARjBEAiARoz2I3Tww/pDRqki+NFEkOabdq2wYLObu0NjK16OG
HQIgXnkzVA2rm07l04tW+GWkCRYsxLCNiGZs0TbtSrdwlrIwDQYJKoZIhvcNAQEL
BQADggEBAAOMgzXsg5eoBMHIzmGbyWVSYa/ERKrAeF7a9pfPYFku/5jFyhzDN3vh
uQjmwvbJcOEw5E7dK2OoBBi0Ueu5sSJu1v83y21dX2J3Fx1CRBdGsh0JIoWEOI1U
2FQi9aXh7rZzz+Fx+MmFvo2iUTNEVcxGz4lOCzdDtf+HVb0BUvrvnvdpAsiz4/Gw
rwWMd+BlA7NGArJgZY62y1qgzOkxk3d15StDMh55X4gqpeIP1kCroLJPOgcoKRco
Z1L5fC9r+aZR8ffWg9dFFEULka7/Zc3wZw0sX80bYiC2pGRvuZOj6j/EYkrtvGRi
gw2GxsE0o/oVbmupoDy5UYLvcZxZ8+M=
-----END CERTIFICATE-----
[root@bhrod0004-salt ac2-afname.test-facet.onl]#
1 Like
Nekit
February 29, 2024, 11:02am
9
Seems like the file was incorrectly modified by something . Follow Nummer378's advice.
6 Likes
I checked. These scripts don't use any file at all. They just create and remove the required TXT record in the DNS.
MikeMcQ
February 29, 2024, 11:30am
11
Would you show the contents of a renewal config file for an affected certificate? It is found in folder
/etc/letsencrypt/renewal
3 Likes
Sure
[root@bhrod0004-salt renewal]# cat ac1-keycloak.test-facet.onl.conf
# renew_before_expiry = 30 days
version = 2.6.0
archive_dir = /etc/letsencrypt/archive/ac1-keycloak.test-facet.onl
cert = /etc/letsencrypt/live/ac1-keycloak.test-facet.onl/cert.pem
privkey = /etc/letsencrypt/live/ac1-keycloak.test-facet.onl/privkey.pem
chain = /etc/letsencrypt/live/ac1-keycloak.test-facet.onl/chain.pem
fullchain = /etc/letsencrypt/live/ac1-keycloak.test-facet.onl/fullchain.pem
# Options used in the renewal process
[renewalparams]
authenticator = manual
account = bd9492d7c2d0d5be9c8826f90d386352
manual_auth_hook = /srv/beheer/letsencrypt/certbot-authenticator.sh
server = https://acme-v02.api.letsencrypt.org/directory
manual_cleanup_hook = /srv/beheer/letsencrypt/certbot-removetxt.sh
rsa_key_size = 4096
pref_challs = dns-01,
key_type = rsa
1 Like
Could you show the output of the
ls -l /etc/letsencrypt/archive/ac1-keycloak.test-facet.onl
command, please?
3 Likes
[root@bhrod0004-salt renewal]# ls -l /etc/letsencrypt/archive/ac1-keycloak.test-facet.onl
total 96
-rw-r--r--. 1 root root 2224 Apr 13 2023 cert15.pem
-rw-r--r--. 1 root root 2224 Jun 12 2023 cert16.pem
-rw-r--r--. 1 root root 2147 Aug 31 13:28 cert17.pem
-rw-r--r--. 1 root root 2147 Oct 30 23:11 cert18.pem
-rw-r--r--. 1 root root 2147 Dec 29 23:09 cert19.pem
-rw-r--r--. 1 root root 2147 Feb 27 23:04 cert20.pem
-rw-r--r--. 1 root root 3749 Apr 13 2023 chain15.pem
-rw-r--r--. 1 root root 3749 Jun 12 2023 chain16.pem
-rw-r--r--. 1 root root 3749 Aug 31 13:28 chain17.pem
-rw-r--r--. 1 root root 3749 Oct 30 23:11 chain18.pem
-rw-r--r--. 1 root root 3749 Dec 29 23:09 chain19.pem
-rw-r--r--. 1 root root 1826 Feb 27 23:04 chain20.pem
-rw-r--r--. 1 root root 4050 Apr 13 2023 fullchain15.pem
-rw-r--r--. 1 root root 4050 Jun 12 2023 fullchain16.pem
-rw-r--r--. 1 root root 3973 Aug 31 13:28 fullchain17.pem
-rw-r--r--. 1 root root 3973 Oct 30 23:11 fullchain18.pem
-rw-r--r--. 1 root root 3973 Dec 29 23:09 fullchain19.pem
-rw-r--r--. 1 root root 2147 Feb 27 23:04 fullchain20.pem
-rw-------. 1 root root 3272 Apr 13 2023 privkey15.pem
-rw-------. 1 root root 3268 Jun 12 2023 privkey16.pem
-rw-------. 1 root root 3272 Aug 31 13:28 privkey17.pem
-rw-------. 1 root root 3268 Oct 30 23:11 privkey18.pem
-rw-------. 1 root root 3272 Dec 29 23:09 privkey19.pem
-rw-------. 1 root root 3268 Feb 27 23:04 privkey20.pem
1 Like
Thanks. And now, I would like to ask the output of the same command with the live directory:
ls -l /etc/letsencrypt/live/ac1-keycloak.test-facet.onl
2 Likes
[root@bhrod0004-salt renewal]# ls -l /etc/letsencrypt/live/ac1-keycloak.test-facet.onl
total 8
-rw-r--r--. 1 root root 692 Dec 21 2020 README
lrwxrwxrwx. 1 root root 52 Feb 27 23:04 cert.pem -> ../../archive/ac1-keycloak.test-facet.onl/cert20.pem
lrwxrwxrwx. 1 root root 53 Feb 27 23:04 chain.pem -> ../../archive/ac1-keycloak.test-facet.onl/chain20.pem
lrwxrwxrwx. 1 root root 68 Feb 27 23:04 fullchain.pem -> /etc/letsencrypt/archive/ac1-keycloak.test-facet.onl/fullchain20.pem
lrwxrwxrwx. 1 root root 55 Feb 27 23:04 privkey.pem -> ../../archive/ac1-keycloak.test-facet.onl/privkey20.pem
That's weird... why does the fullchain.pem have a absolute path while the other files use a relative path?
Good question, I do not know the answer. It looks ugly. May someone else know the answer?
The fullchain.pem is supposed to be the concatenation of the cert.pem and the chain.pem.certbot renew command?
4 Likes
Thanks for your analysis. I'll check.
1 Like
We only run this cron command:
0 23 * * * root /bin/certbot renew >/srv/beheer/cron-certbot.out 2>&1
And I see the last modified time of the file is the same time that's logged:
2024-02-27 23:04:00,038:DEBUG:certbot._internal.storage:Writing full chain to /etc/letsencrypt/archive/ac1-keycloak.test-facet.onl/fullchain20.pem.
Is the /bin/certbot a patched version? The 2.6.0 seems to be quiet recent.
3 Likes