Migrating certs between VPCs with renew-ability

pkiula · January 3, 2021, 11:23pm

My domain is:
karlaporter.com

I ran this command:
certbot certonly -a webroot --expand --webroot-path=/home/kp -d karlaporter.com -d www.karlaporter.com

It produced this output:
Installed certbox

My web server is (include version):
Nginx 1.14.1

The operating system my web server runs on is (include version):
CentOS 8

My hosting provider, if applicable, is:
Digital Ocean

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
NO

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
1.10.1

I love LE and have used it to install certbot in my previous liux box (centos 7). Now we've upgraded to a CentOS 8 box. All our domains have moved over with a simple rsync, including the /etc/letsencrypt folders, and the /etc/security/ where we had generated the SSL stuff.

Question: is this a good enough way to "move" the certs, or will we have to regenerate the certs on this new server all afresh? Whenn I issue certbot -renew it gives me errors. Hence the question.

Thanks!

sahsanu · January 3, 2021, 11:34pm

Hello @pkiula,

If you rsync all the directories with the same perms, same symlinks (this is important) etc. you shouldn't have issues.

I suppose the dash in command certbot -renew is a typo, is it right?

What are the errors you are getting when using certbot renew?

Cheers,
sahsanu

griffin · January 4, 2021, 12:37am

Welcome Back to the Let's Encrypt Community, Phoenix

In my experience with helping with certbot "migrations", I can say that things can be much messier than one might expect due to the tendrils that certbot creates. I highly recommend running certbot update_symlinks on the target system at the very least.

rg305 · January 4, 2021, 12:39am

Please show the errors.

pkiula · January 4, 2021, 12:51am

The command we run in our crontab:

certbot renew --renew-hook "service nginx reload"

The result of this command:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/DOMAIN1.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 70, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/storage.py", line 470, in __init__
    self._check_symlinks()
  File "/usr/lib/python3.6/site-packages/certbot/_internal/storage.py", line 537, in _check_symlinks
    "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/DOMAIN1.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/DOMAIN1.com.conf is broken. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/DOMAIN2.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 70, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/storage.py", line 470, in __init__
    self._check_symlinks()
  File "/usr/lib/python3.6/site-packages/certbot/_internal/storage.py", line 537, in _check_symlinks
    "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/DOMAIN2.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/DOMAIN2.com.conf is broken. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

No renewals were attempted.
No hooks were run.

Additionally, the following renewal configurations were invalid: 
  /etc/letsencrypt/renewal/DOMAIN1.com.conf (parsefail)
  /etc/letsencrypt/renewal/DOMAIN2.com.conf (parsefail)
  .......
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
0 renew failure(s), 12 parse failure(s)

Didn't realize we needed to create symlinks. Running that yields issues with the very first domain in the list:

# certbot update_symlinks

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Expected /etc/letsencrypt/live/DOMAIN0.com/cert.pem to be a symlink

What should I be symlinking and to what? Thanks for any pointers.

griffin · January 4, 2021, 1:18am

--deploy-hook perhaps?

sahsanu · January 4, 2021, 1:19am

Maybe those messages are related to the symlinks issues but it is something that you should check once the symlinks are recreated.

Yeah, symlinks are important

All the files in /etc/letsencrypt/live/DOMAIN0.com/ are symlinks that points to the last files in ../../archive/DOMAIN0.com/

For example, if you have the following files in /etc/letsencrypt/archive/DOMAIN0.com/:

cert1.pem
cert2.pem
chain1.pem
chain2.pem
fullchain1.pem
fullchain2.pem
privkey1.pem
privkey2.pem

You should create symlinks in /etc/letsencrypt/live/DOMAIN0.com/ to the last files in archive, in this case to *2.pem files.

cd /etc/letsencrypt/live/DOMAIN0.com/
ln -sf ../../archive/DOMAIN0.com/cert2.pem cert.pem
ln -sf ../../archive/DOMAIN0.com/chain2.pem chain.pem
ln -sf ../../archive/DOMAIN0.com/fullchain2.pem fullchain.pem
ln -sf ../../archive/DOMAIN0.com/privkey2.pem privkey.pem

WARNING: Before doing any change, backup all the files and dirs in /etc/letsencrypt/. Command ln -sf will overwrite any file in live/DOMAIN0.com dir so you should be pretty sure that the last files in archive/DOMAIN0.com dir are the right ones you should be using.

Good luck.

Cheers,
sahsanu

griffin · January 4, 2021, 1:34am

@sahsanu

I gave @pkiula what I believe to be the easiest way to fix the symlinks above:

@pkiula

This is a better command to straighten things out:

certbot certonly --cert-name karlaporter.com --webroot -w /home/kp -d "karlaporter.com,www.karlaporter.com" --deploy-hook "nginx -s reload" --force-renewal

Then you only need the following in your crontab:

certbot renew

sahsanu · January 4, 2021, 1:23am

As far as I know that command doesn't create the symlinks, only updates existing symlinks to the last files in archive dir.

griffin · January 4, 2021, 1:28am

@sahsanu

I've had mixed results. Consider the definition in the user guide:

update_symlinks

Recreate symlinks in your /etc/letsencrypt/live/ directory

sahsanu · January 4, 2021, 1:32am

I haven't used certbot in a long time so let me check it.

griffin · January 4, 2021, 1:33am

@sahsanu

Thanks for taking the time to verify, brother. It is greatly appreciated.

sahsanu · January 4, 2021, 1:42am

@griffin, I did 4 tests:

1.- Files in live/domain/ are regular files and the command does nothing.
2.- Files in live/domain/ are symlinks pointing to the before the last files in archive/domain dir and the command does nothing.
3.- I removed the files in live/domain and the command does nothing.
4.- I removed the dir live/domain and the command does nothing.

So, I've no idea what update_symlinks parameter does

griffin · January 4, 2021, 1:44am

Uh... your certbot is broken.

rg305 · January 4, 2021, 2:10am

So we are left with... ???

certbot delete --cert-name karlaporter.com

certbot certonly --cert-name karlaporter.com --webroot -w /home/kp -d "karlaporter.com,www.karlaporter.com" --deploy-hook "nginx -s reload"

rg305 · January 4, 2021, 1:49am

You needed to have used the excessively-complex rsync - LOL

sahsanu · January 4, 2021, 1:48am

Recreating the symlinks manually should work but yes, sometimes it is better to delete and start fresh

pkiula · January 4, 2021, 2:10am

Thank you. I'll simply do this, thank you!

rg305 · January 4, 2021, 2:16am

@pkiula Glad we could help!

Cheers from Miami