Renew certificate automatically without exposing http port 80

My domain is: gschmidt.duckdns.org

My web server is (include version): Domoticz version 4.10430 (beta)

The operating system my web server runs on is (include version): Raspbian Stretch (Linux 4.14.79-v7+)

I can login to a root shell on my machine (yes or no, or I don’t know): yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.31.0

Hi,

Letsecrypt send me a “Let’s Encrypt certificate expiration notice” e-mail to renew my certificate.
I created the certificate a few months ago using the “–preferred-challenges dns” option because i did not want to open port 80 on my main router.

The command I used was:
/etc/letsencrypt/letsencrypt-auto certonly --manual --preferred-challenges dns --manual-auth-hook /home/pi/duckdns/auth.sh --manual-cleanup-hook /home/pi/duckdns/cleanup.sh

How do I automatically renew (let’s say every 8 weeks) the certificate using the --preferred-challenges dns option?

Greetzzz,

Gerben

You should notice a line starting with pref_challs = in your /etc/letsencrypt/renewal/{domain}.conf file. That's all you need.

(And of course, a cron entry to run `letsencrypt-auto every 12 hours or so):

0 0,12 * * * python -c 'import random; import time; time.sleep(random.random() * 3600)' && /path/to/letsencrypt-auto renew 

To test that it works (it should remember automatically your auth hooks and DNS preference):

letsencrypt-auto renew --dry-run

Hi,

You were right!
However when I run/etc/letsencrypt/letsencrypt-auto renew --dry-run
I get the following error:

Encountered exception during recovery: 
Traceback (most recent call last):
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/error_handler.py", line 108, in _call_registered
    self.funcs[-1]()
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 323, in _cleanup_challenges
    self.auth.cleanup(achalls)
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/plugins/manual.py", line 242, in cleanup
    env = self.env.pop(achall)
KeyError: KeyAuthorizationAnnotatedChallenge(challb=ChallengeBody(chall=DNS01(token='\x9b+\xe3\x1d\x0c~7\xd9\x08\x9ap\x1f\x03\x9d\x86%\xe8\xc73-\xac\xf7fu\x1ev\xc9dJ\xde|8'), status=Status(pending), uri=u'https://acme-staging-v02.api.letsencrypt.org/acme/challenge/fp47MV2DqRJJmR0MTkkYAA0M8F_9AJWORUPVLUjOsJE/254604685', validated=None, _url=u'https://acme-staging-v02.api.letsencrypt.org/acme/challenge/fp47MV2DqRJJmR0MTkkYAA0M8F_9AJWORUPVLUjOsJE/254604685', error=None), domain=u'gschmidt.duckdns.org', account_key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x76718d30>)>))
Attempting to renew cert (gschmidt.duckdns.org) from /etc/letsencrypt/renewal/gschmidt.duckdns.org.conf produced an unexpected error: Missing command line flag or config entry for this setting:
NOTE: The IP of this machine will be publicly logged as having requested this certificate. If you're running certbot in manual mode on a machine that is not your server, please ensure you're okay with that.

Are you OK with your IP being logged?

(You can set this with the --manual-public-ip-logging-ok flag). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gschmidt.duckdns.org/fullchain.pem (failure)

I’m not sure how to interpret that backtrace.

Would you be able to post the /var/log/letsencrypt/letsencrypt.log from that run to somewhere like dpaste.de?

Alternatively, maybe try running your original certonly command from scratch and see if that makes a difference to the success.

Yeah strange because my manual certonly command did work:
/etc/letsencrypt/letsencrypt-auto certonly --manual --preferred-challenges dns --manual-auth-hook /home/pi/duckdns/auth.sh --manual-cleanup-hook /home/pi/duckdns/cleanup.sh

Here is the log: https://dpaste.de/3FjU
You also can see I updated manually because the certbot version is updated to: 0.31.0

Ah, I think that log might make more sense. The way I read it, the earlier issue (MissingCommandlineFlag) may be causing the latter error (KeyError: KeyAuthorizationAnnotatedChallenge).

Does a dry run with --manual-public-ip-logging-ok make it succeed?

1 Like

Sir, you are a great help!

https://dpaste.de/cUPV

So I guess the crontab should look like

0 0,12 * * * python -c 'import random; import time; time.sleep(random.random() * 3600)' && /etc/letsencrypt/letsencrypt-auto renew --manual-public-ip-logging-ok

However since I am running Domoticz on the RPi, I also need to add/update the certificate in the directory:

/home/pi/domoticz/server_cert.pem

To add the certificate to Domoticz the wiki site https://www.domoticz.com/wiki/Native_secure_access_with_Lets_Encrypt mentions:

sudo rm ~/domoticz/server_cert.pem
sudo cat /etc/letsencrypt/live/gschmidt.duckdns.org/privkey.pem >> ~/domoticz/server_cert.pem
sudo cat /etc/letsencrypt/live/gschmidt.duckdns.org/fullchain.pem >> ~/domoticz/server_cert.pem
sudo cp ~/domoticz/server_cert.pem ~/domoticz/letsencrypt_server_cert.pem
sudo cat /etc/ssl/certs/dhparam.pem >> ~/domoticz/server_cert.pem
sudo /etc/init.d/domoticz.sh restart

Note: I added the “dhparam” command because this happens as mentioned on the wiki site

So how do I crontab these actions? with a bash script maybe?

I found something:

nano /home/pi/cert-domoticz-update.sh

Paste the following into the editor:

/etc/letsencrypt/letsencrypt-auto renew --manual-public-ip-logging-ok
rm ~/domoticz/server_cert.pem
rm ~/domoticz/letsencrypt_server_cert.pem 
cat /etc/letsencrypt/live/gschmidt.duckdns.org/privkey.pem >> ~/domoticz/server_cert.pem
cat /etc/letsencrypt/live/gschmidt.duckdns.org/fullchain.pem >> ~/domoticz/server_cert.pem
cp ~/domoticz/server_cert.pem ~/domoticz/letsencrypt_server_cert.pem
cat /etc/ssl/certs/dhparam.pem >> ~/domoticz/server_cert.pem 
/etc/init.d/domoticz.sh restart
sudo chmod a+x /home/pi/cert-domoticz-update.sh

Add this to crontab (runs every sunday):

3 0 * * 7  /home/pi/cert-domoticz-update.sh

Could this work? because your crontab to renew looks so different?

Greetzzz,

Gerben

Apart from the question of whether this will work on other ways, I find this script kind of odd because it doesn't check whether the certificate was renewed or not before replacing the domoticz certificate and restarting the service!

In Certbot's current design, it would make more sense to set this script as a --deploy-hook (although without the /etc/letsencrypt/letsencrypt-auto renew --manual-public-ip-logging-ok line at the beginning), and then use the original cron job.

The difference is mostly about an attempt to add a random delay to each renewal attempt, something which we've asked people to do in order to reduce load spikes at the CA, but which is now also handled in Certbot itself. Also, the cron job that @_az mentioned runs twice a day, rather than once a week.

Hmmm… i can’t follow you.

I had to add the --manual-public-ip-logging-ok parameter to renew the certificate whitout errors.

Do you mean that I have to create a crontab like:

0 0,12 * * * python -c 'import random; import time; time.sleep(random.random() * 3600)' && /etc/letsencrypt/letsencrypt-auto renew --deploy-hook /path/to/deploy-hook-script

and let the deploy-hook-script copy the certificates to the domoticz domain?

Yes, that's right. This has the advantage that the deploy-hook-script will only run when the certificate has actually been renewed, as opposed to running once a week. Also, you can then check for renewals twice a day (nothing will happen if the renewals aren't due yet).

(I don't remember whether --manual-public-ip-logging-ok has to be specified during the renewal process, or just once.)

I see what you mean…a cleaner solution…however does it actually matter that the certificate was renewed or not? The script copies either an existing certificate or a new one to the domoticz domain.

bye the way I ran the /etc/letsencrypt/letsencrypt-auto renew --dry-run again without the --manual-public-ip-logging-ok parameter, and it got the same error as i mentioned in a previous reply…does this error has to do with the fact that i use in my conf --preferred-challenges dns to avoid exposing port 80 on my router?

You shouldn't need to overwrite the certificate files and restart the service unless there's a real reason to. :slight_smile:

It probably means that Certbot isn't remembering that you previously said --manual-public-ip-logging-ok.

True!...So best way for renewal would be?:

  1. Create a crontab like:

0 0,12 * * * python -c 'import random; import time; time.sleep(random.random() * 3600)' && /etc/letsencrypt/letsencrypt-auto renew --manual-public-ip-logging-ok --deploy-hook /home/pi/cert-domoticz-update.sh

  1. In cert-domoticz-update.sh the following lines are added:
sudo rm /home/pi/domoticz/server_cert.pem
sudo rm /home/pi/domoticz/letsencrypt_server_cert.pem
sudo cat /etc/letsencrypt/live/gschmidt.duckdns.org/privkey.pem >> /home/pi/domoticz server_cert.pem
sudo cat /etc/letsencrypt/live/gschmidt.duckdns.org/fullchain.pem >> /home/pi/domoticz/server_cert.pem
sudo cp /home/pi/domoticz/server_cert.pem /home/pi/domoticz/letsencrypt_server_cert.pem
sudo cat /etc/ssl/certs/dhparam.pem >> /home/pi/domoticz/server_cert.pem
sudo /etc/init.d/domoticz.sh restart

I think this will work. Note that specifying the hook in the renew command isn’t a good thing if you have Certbot managing multiple certificates on the same system (since the deploy-hook script would run for the renewal of any certificate).

These commands might not have the exact effect that you expect. The >> is evaluated by the shell and runs with the privileges of the user running the script, not with sudo privileges. (However, in this case the script might already be run by root!)

$ sudo echo hello > /bin/hello
bash: /bin/hello: Permission denied

Thanx for the help man, I think I understand it now…a bit :wink:
Don’t have multiple certificates…only one

greetzzz,

Gerben

Well this “Add the certificate to Domoticz” part of the wiki tutorial https://www.domoticz.com/wiki/Native_secure_access_with_Lets_Encrypt is what I followed when I installed duckdns and letsencrypt for the first time on my Pi running Domoticz…and it worked.

Those instructions seem like they would work fine, but they don’t really address automating renewal at all!

Correct, that’s why I was surprised when I received an e-mail from letsencrypt to renew the certificate…lazy as I am, I was trying to find a script for automatic renewal and ended up here :wink: