Can somebody help me to solve the issue with Letsencrypt certificate for Zimbra?

My domain is: aproservice-x.md

I ran this command:

certbot renew --standalone
cp /etc/letsencrpyt/live/mail.aproservice-x.md/* /opt/zimbra/ssl/letsencrypt/
chown zimbra:zimbra /opt/zimbra/ssl/letsencrypt/*
echo "-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----" >> /opt/zimbra/ssl/letsencrypt/chain.pem
cp /opt/zimbra/ssl/letsencrypt/privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
runuser -l zimbra -c 'cd /opt/zimbra/ssl/letsencrypt/ && /opt/zimbra/bin/zmcertmgr deploycrt comm cert.pem chain.pem'
runuser -l zimbra -c 'zmcontrol restart'

It produced this output:

ERROR: Unable to validate certificate chain: C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error cert.pem: verification failed

My zimbra server is:
Release 8.8.15_GA_3953.RHEL8_64_20200629025823 RHEL8_64 FOSS edition, Patch 8.8.15_P25

The operating system is:
CentOS Linux release 8.5.2111

I can login to a root shell on my machine.

I'm not using a control panel to manage my site.

The version of my client is:
certbot 1.22.0

Since I have changed the certificates in the zimbra's repository, its services fail to start, thus the webserver is down as well.
Is there a chance to get these back to life? I'm running the company's mail server on it and now it is broken definitely.

1 Like

Before you deploycrt, you must verifycrt.
Have a look at their recommendation:
Installing a LetsEncrypt SSL Certificate - Zimbra :: Tech Center

1 Like

OK, you're definitely right! But things happened already. :frowning:

1 Like

Understood.
Follow the doc and see if you can fix things.

1 Like

Fine! What do you mean follow the doc? Which doc?
(I really mean it! For over 10 hours trying to get a solution. Still "Unable to validate certificate chain: C = US, O = Internet Security Research Group, CN = ISRG Root X1")

1 Like
2 Likes

Thanks for suggestion, tried again, still not solved.
Actually there are 3 steps in this document prior to verification:

  1. getting the certificates: certbot certonly
  2. copying the privkey to commercial: cp $letsencrypt-home/privkey.key $zimbra/ssl/zimbra/commercial/commercial.key
  3. adding the ISRG-X1.pem to chain.pem

Doing this does not help my case. The same Unable to validate certificate chain.
Any other ideas, please?

1 Like

Ah! Shall I run certbot with the keys --force-renewal --preferred-chain "ISRG Root X1"?

1 Like

Well, I have found a workaround here.
It was you who had warned the author of the post that this is a shortcut and not a solution. But right now I just have to start my server.
Ideas?

1 Like

After some investigation, I found that the chain.pem that is received from the Letsencrypt contains two certificates, one of them is the certificate for the R3 chain, issued by the ISRG Root X1. The second certificate is for the ISRG Root X1, issued by DST Root CA X3. But I don't have a valid certificate for DST Root CA X3 in the system. This is why I get error unable to validate chain ISRG Root X1.

Ideas?

     Version: 3 (0x2)
    Serial Number:
        40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
    Signature Algorithm: sha256WithRSAEncryption
    Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
    Validity
        Not Before: Jan 20 19:14:03 2021 GMT
        Not After : Sep 30 18:14:03 2024 GMT
    Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
            RSA Public-Key: (4096 bit)
1 Like

Zimbra is super anal.
It requires the entire chain before it can be verified.
The fullchain.pem would normally contain all that is needed [but NOT for Zimbra].

Here is a script file I made and use to update the cert:

#!/bin/bash

cp /etc/letsencrypt/live/MY-DOMAIN/*.pem /opt/zimbra/ssl/letsencrypt/
su - zimbra -c "/opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/letsencrypt/privkey.pem /opt/zimbra/ssl/letsencrypt/cert.pem"

cp /opt/zimbra/ssl/letsencrypt/privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
su - zimbra -c "/opt/zimbra/bin/zmcertmgr deploycrt comm /opt/zimbra/ssl/letsencrypt/cert.pem /opt/zimbra/ssl/letsencrypt/shortchain.pem"
1 Like

Here is the secret sauce to that script:

cat /opt/zimbra/ssl/letsencrypt/shortchain.pem

-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

It won't last forever but we'll be here to update you when that starts to fail :wink:

1 Like

Thank you a lot for the script and the solution.
Well, it seems that the script won't use the chain.pem file, but rather the file shortchain.pem, which consists of the R3 certificate issued by ISRG Root X1 and the self-signed certificate for ISRG Root X1. This is the .pem that was proposed by denos in the thread that I have mentioned above.
So, why this solution is not so good in long time perspective? It could even be automated. The issue with it would be limited time for use - only 2 years instead of 13. :slight_smile:
Regards

1 Like

R3 can be immediately replaced for any reason (even ISRG Root X1) and will eventually expire.

1 Like

Then we have to put an eye on R3.
Is there a place where R3 can be downloaded from automatically?
Updated.
Ok, nevermind, R3 can be downloaded from the link https://letsencrypt.org/certs/lets-encrypt-r3.pem. X1 can be downloaded using the link https://letsencrypt.org/certs/isrgrootx1.pem.
BTW, why in the scripts is used the file with .txt extension for downloading, because of antiviruses?

So, we need to download 2 files and put them one by one into the shortchain.pem.
I assume that all the certificate renewing procedure could be done using the following script. Please, be welcome to correct the statements if something is wrong.

#!/bin/bash

MYDOMAIN="www.example.com"  # change www.example.com with your domain
ZimbraSSL="/opt/zimbra/ssl/letsencrypt"

certbot certonly --standalone

cd $ZimbraSSL/
rm ./*.pem
cp /etc/letsencrypt/live/$MYDOMAIN/*.pem ./
wget https://letsencrypt.org/certs/lets-encrypt-r3.pem https://letsencrypt.org/certs/isrgrootx1.pem -O ./shortchain.pem
chown zimbra:zimbra ./*

su - zimbra -c "cd $ZimbraSSL/ && /opt/zimbra/bin/zmcertmgr verifycrt comm privkey.pem cert.pem shortchain.pem"
if [ $? -ne 0 ]; then
    echo FAIL to verify the certificates, so no changes produced
    exit 1
fi

\cp ./privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
su - zimbra -c "cd $ZimbraSSL/ && /opt/zimbra/bin/zmcertmgr deploycrt comm cert.pem shortchain.pem"

I'd like to make the script running automatically by a cron job, but I don't know how to do it in case of certbot, as I have seen some info that it generates shedules upon installation. But I didn't find a cron job regard certbot.

certbot should already be cron enabled [by default installation].
It should run certbot renew twice a day.
What you need is the addition of a --deploy-hook ("deployment hook"), that can run the added script to ingest the new cert into Zimbra.
OR
You can simply run that ingest script once a week (every week - always ingesting the latest available cert).

As for R3, it can be replaced with R4 or R5 or whatever without any notice.
Intermediate certs are never expected to be "hard-coded" anywhere.
The ACME protocol ensures the delivery of the current chain.
We are trying to work outside that protocol...
So, whatever we do will not be permanent.
OR will it?

Try using these steps to create that shortchain.pem file programmatically.

cat "/opt/zimbra/ssl/letsencrypt/chain.pem" > "/var/tmp/shortchain.pem"
cat "/etc/ssl/certs/$(openssl x509 -in /etc/letsencrypt/live/EXAMPLE.com/chain.pem -noout -issuer_hash).0" >> "/var/tmp/shortchain.pem"
3 Likes

One added piece to the puzzle...
You have to use the short chain in the certbot request:
--preferred-chain "ISRG Root X1"

[then you can build on to it with the self-signed "ISRG Root X1" cert]
[otherwise you'll get: R3+cross-signedX1+self-signedX1]

3 Likes

I have build a script in order to test the pieces you suggested, but first a question to the first statement: Why do we have to use "/opt/zimbra/ssl/letsencrypt/chain.pem" while that folder exists there only when we create it via script or manually - this folder (and the files inside) doesn't exist by default?
Nevertheless, I used the statements proposed by you.

RootTemp=/opt/zimbra/ssl/temp
MyDomain=mail.example.com
...
cat "/etc/ssl/certs/$(openssl x509 -in /etc/letsencrypt/live/$MyDomain/chain.pem -noout -issuer_hash).0" >> "$RootTemp/shortchain.pem"
...

This statement returns error "cat: /etc/ssl/certs/4042bcee.0: No such file or directory"
I guess that this statement should copy first certificate from the chain, but it is wrongly built. My "openssl version" returns "OpenSSL 1.1.1k FIPS 25 Mar 2021"