Zimbra renewal - Problems with R3

Help
43

Thanks for your support in this matter, @rg305

You are totally right, manually replacing the chain.pem won't do in a fully automatized environment. Therefore, a colleague of mine used the snap version of certbot on centos7 and receives a different chain.pem:

# openssl crl2pkcs7 -nocrl -certfile chain.pem | openssl pkcs7 -noout -print_certs
subject=/C=US/O=Let's Encrypt/CN=R3
issuer=/C=US/O=Internet Security Research Group/CN=ISRG Root X1

subject=/C=US/O=Internet Security Research Group/CN=ISRG Root X1
issuer=/C=US/O=Internet Security Research Group/CN=ISRG Root X1

When I request a certificate using the --preferred-chain "ISRG Root X1" switch on centos7 and the certbot installed via epel repo I get the following chain.pem:

# openssl crl2pkcs7 -nocrl -certfile 0000_chain.pem | openssl pkcs7 -noout -print_certs
subject=/C=US/O=Let's Encrypt/CN=R3
issuer=/C=US/O=Internet Security Research Group/CN=ISRG Root X1

subject=/C=US/O=Internet Security Research Group/CN=ISRG Root X1
issuer=/O=Digital Signature Trust Co./CN=DST Root CA X3

The exact command used was:

sudo certbot certonly  --standalone --non-interactive --agree-tos --preferred-chain "ISRG Root X1"  --email valid-email@example.com --csr request.pem

In my naive, ideal world I would expect the same result of the snap and epel installed certbot client?! Did I miss something along the way?

Thanks!

2 Likes
44

Hi @sphinx-nh welcome to the LE community forum :slight_smile:

certbot has been developed and matured over time.
There have been:

  • letsencrypt
  • letsencrypt-auto
  • certbot
  • certbot-auto
  • certbot (via snap)

You need to check the version and ensure it is 1.12 (or higher)
[which is when the --preferred-chain was added]

2 Likes
45

Ah, that's a known pattern: the usual process of software development. :wink:

Thanks for the hint regarding the version and the feature availability in certbot >=1.12. Of course I assumed that the documentation of the option in the help message indicates the availability. Sadly, the help text of certbot -h certonly describes the option --preferred-chain PREFERRED_CHAIN although the certbot version is 1.11 (installed via epel on CentOS 7). Therefore, the current CentOS 7 epel package does not support the aforementioned option based on your information.

To overcome this issue I will use the snap version of certbot and will document on the outcome afterwards.

The reason why I did not want to use the snap version of certbot is simply side effects on the Zimbra disk space monitoring when using snap on Zimbra installations: Disable Disk Space Monitoring for Loop Devices - Zimbra :: Tech Center

We may end up disabling the Zimbra df-monitoring as our installations are monitored by a separate agent anyway.

2 Likes
46

You can also try using an alternate ACME client (like: acme.sh).
And you don't have to uninstall one to install the other - they can coexist peacefully :slight_smile:
Mind you, they will each store certs independent of each other - but if either works, I'd use that one.

2 Likes
47

Thanks for mentioning acme.sh - you are right, this would resolve our problem and is an option for small deployments. The decision for certbot are in regards to operation and organization.

For example: we value the structure of certbot and how certificates are organized in the filesystem (/etc/letsencrypt). Moreover, we are experienced in the deployment and operation using certbot.

Thanks again for your help, it is highly appreciated!

2 Likes
48

Using the snap certbot in combination with the --preferred-chain 'ISRG Root X1' option (as suggested by @rg305) fixed our issue. In order to automatize the renew process - for hopefully another five years - we use the Github project letsencrypt-zimbra (GitHub - VojtechMyslivec/letsencrypt-zimbra: Files to automate the deploy of letsencrypt certificates to Zimbra). Maybe the repo owner will update the script in order to adopt to the certificate chain changes - maybe not. Here is what we did:

  1. install snap (see Installing snapd | Snapcraft documentation)
  2. uninstall CentOS 7 EPEL certbot package
  3. install certbot with snap
  4. manually download the ISRG-X1.pem (see Zimbra renewal - Problems with R3 - #5 by georged) and copied it to the root_certs directory of the letsencrypt-zimbra repository.
  5. change the path to point to the ISRG-X1.pem: root_CA_file="${letsencrypt_zimbra_dir}/root_certs/ISRG-X1.pem"
  6. add the option --preferred-chain 'ISRG Root X1' to the "$letsencrypt" certonly command in the bash script obtain-and-deploy-letsencrypt-cert.sh

The requirements of Zimbra in regards to the certificate chain is a challenge. In order to fulfill those the script appends the ISRG X1 certificate to the intermediate certificate chain. This step is static and will fail if the root certificate changes again.

br,
nh

2 Likes
49

Hi there,

I ran into the same issues and the problem was in the Zimbra instructions which I ran over a couple of years now (based from JDunphy-Letsencrypt - Zimbra :: Tech Center). My instructions were cat root.cert chain.pem >> fullchain.pem

The issue is that the root certificate is appended to fullchain. If this file has been created a long time ago, it includes multiple root certificates, i.e. multiple old (now expired) certificate info plus the new valid certificate. All the old expired certificates were the reason for verifycrt to fail.

Using openssl crl2pkcs7 -nocrl -certfile fullchain.pem | openssl pkcs7 -noout -print_certs (in the case of the instructions georged refers to, chain.pem will have to be validated) helped me to identify and resolve the issue.

I now overwrite fullchain.pem instead of appending the root cert every time :slight_smile:

1 Like
50

be aware of the issue mentioned by @rg305 in Zimbra renewal - Problems with R3 - #42 by rg305

2 Likes
51

Hi Everyone,
I use Ubuntu 16.04.7 LTS
OpenSSL 1.1.0h
Zimbra 8.8.15_GA
removed all old archives /etc/letsencrypt/archive/mydomain ; /etc/letsencrypt/live/mydomain

Downloaded latest version certbot-1.19.0/
./letsencrypt-auto certonly --standalone --non-interactive --agree-tos --preferred-chain "ISRG Root X1" -d mydomain1.com -dmydomain2.com
after created files:
/etc/letsencrypt/live/mydomain/fullchain.pem, privkey.pem, chain.pem, cert.pem

copy all files with /etc/letsencrypt/live/mydomain/* to /opt/zimbra/ssl/letsencrypt/mydomain/

cp /etc/letsencrypt/live/mydomain/* /opt/zimbra/ssl/letsencrypt/mydomain/
cd /opt/zimbra/ssl/letsencrypt/mydomain/
wget -O /tmp/ISRG-X1.pem https://letsencrypt.org/certs/isrgrootx1.pem.txt
cat /tmp/ISRG-X1.pem >> /opt/zimbra/ssl/letsencrypt/mydomain/chain.pem
cp /opt/zimbra/ssl/letsencrypt/mydomain/privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
chown zimbra:zimbra /opt/zimbra/ssl/zimbra/commercial/commercial.key

after open zimbra user (su - zimbra)

cd ~
/opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.key /opt/zimbra/ssl/letsencrypt/mydomain/cert.pem /opt/zimbra/ssl/letsencrypt/mydomain/chain.pem

** Verifying 'cert.pem' against 'privkey.pem'
Certificate 'cert.pem' and private key 'privkey.pem' match.
** Verifying 'cert.pem' against 'chain.pem'
ERROR: Unable to validate certificate chain: C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error cert.pem: verification failed

check
$ openssl crl2pkcs7 -nocrl -certfile chain.pem | openssl pkcs7 -noout -print_certs
subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

Please help solve this problem...
does it make sense to upgrade to version Ubuntu 18 LTS and OpenSSL
Thanks all

1 Like
52

Looks like "ISRG Root X1" is not in your trusted root store.
Try:
sudo apt update
sudo apt-get update
sudo install ca-certificates

3 Likes
53

you might also try the chain.pem posted in Zimbra renewal - Problems with R3 - #39 by denos - if your /opt/zimbra/bin/zmcertmgr verifycrt command returns OK with this chain.pem then your chain is not as Zimbra expects it. I got this issue before.

The second certificate in your chain (DST Root CA X3) indicates that the option --preferred-chain might be ignored or is not supported by your client. Therefore, you may want to give the certbot (via snap) a try as mentioned by @rg305 in Zimbra renewal - Problems with R3 - #44 by rg305

br,
nh

2 Likes
54

Thanks
I updated my OS to Ubuntu 18
delete old certbot
install new version (certbot 1.20.0)
OpenSSL 1.1.1

snap install core; snap refresh core
snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot

run command with post 51

after run command

certbot certonly --standalone --non-interactive --agree-tos --preferred-chain "ISRG Root X1" -d $mail_server_url --hsts --expand
2 Likes
55

Thank you @denos . This is working.

2 Likes
56

Thanks! Its worked for me.

2 Likes
57

Thank you. worked for me.
you saved my day.

3 Likes
58

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.