Updating certbot renewal configuration inside the prehook

Hello,
I am not really sure where to ask this question so I'll just do here.

I am trying to automate the certbot renew process, so that

  • if a webserver is running/port 80 is used, it should use the webroot method to renew
  • if no webserver is running/port 80 is not used, it should use the standalone

I was reading about the reconfigure subcommand and wrote a script that updates all renewal configurations depending on the "status" of port 80. When running it on the commandline, it worked how it should, but when I tested it in the renewal as the pre-hook, I just got the error that two instances of certbot are running at the same time, so I dropped that idea.

My next idea was to update the configurations directly in the pre-hook. I wrote the following script that also works outside of certbot:

#!/bin/bash
# Ansible managed

for cert in /etc/letsencrypt/renewal/*
do
  # Check if port 80 is in use
  if netstat -tln | grep -q :80; then
      # Port 80 is in use, set the authenticator to "webroot"
      authenticator="webroot"

      if grep -q "^webroot_path" "$cert"; then
        # If webroot_path exists, update it
        sed -i "s|^webroot_path.*|webroot_path = /data/www/sites|" "$cert"
      else
        # If webroot_path doesn't exist, add it after the authenticator line
        sed -i "\|^authenticator.*|a webroot_path = /data/www/sites" "$cert"
      fi
  else
      # Port 80 is not in use, set the authenticator to "standalone"
      authenticator="standalone"
  fi

  # Modify the renewal configuration file based on the authenticator
  sed -i "s|^authenticator.*|authenticator = $authenticator|" "$cert"
  echo -e "\nchanged certificate $cert conf to:"
  cat "$cert"
done

Now I tested it by generating two certificates using the standalone method, so both configuration have these options:

# Options used in the renewal process
[renewalparams]
account = {account}
pref_challs = http-01,
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory

I expect it to work like this:

  1. Run pre-hook that updates the renewal configurations if necessary
  2. Load the first (updated) renewal configuration
  3. Run the challenge for that certificate
  4. Load the second (updated) renewal configuration
  5. Run the challenge for that certificate

When running certbot renew --dry-run with not used port 80, there aren't any complications.
But when I start apache before running it, it seems certbot is loading the first renewal configuration before running the pre-hook (I printed the new configuration inside the pre-hook to make sure they are updated correctly) so that it tries to start its own standalone webserver but can't because of apache. The second configuration is loaded correctly because its not using the standalone method and finishing the challenge (which makes sense because the pre-hook already ran and updated the confs).

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ansible01.bitvision.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Hook 'pre-hook' ran with output:
 changed certificate /etc/letsencrypt/renewal/ansible01.bitvision.de.conf conf to:
 # renew_before_expiry = 30 days
 version = 1.12.0
 archive_dir = /etc/letsencrypt/archive/ansible01.bitvision.de
 cert = /etc/letsencrypt/live/ansible01.bitvision.de/cert.pem
 privkey = /etc/letsencrypt/live/ansible01.bitvision.de/privkey.pem
 chain = /etc/letsencrypt/live/ansible01.bitvision.de/chain.pem
 fullchain = /etc/letsencrypt/live/ansible01.bitvision.de/fullchain.pem

 # Options used in the renewal process
 [renewalparams]
 account = {account}
 pref_challs = http-01
 authenticator = webroot
 webroot_path = /data/www/sites
 server = https://acme-v02.api.letsencrypt.org/directory

 changed certificate /etc/letsencrypt/renewal/ansible02.bitvision.de.conf conf to:
 # renew_before_expiry = 30 days
 version = 1.12.0
 archive_dir = /etc/letsencrypt/archive/ansible02.bitvision.de
 cert = /etc/letsencrypt/live/ansible02.bitvision.de/cert.pem
 privkey = /etc/letsencrypt/live/ansible02.bitvision.de/privkey.pem
 chain = /etc/letsencrypt/live/ansible02.bitvision.de/chain.pem
 fullchain = /etc/letsencrypt/live/ansible01.bitvision.de/fullchain.pem

 # Options used in the renewal process
 [renewalparams]
 account = {account}
 pref_challs = http-01,
 authenticator = webroot
 webroot_path = /data/www/sites
 server = https://acme-v02.api.letsencrypt.org/directory
Simulating renewal of an existing certificate for ansible01.bitvision.de and www.ansible01.bitvision.de
Failed to renew certificate ansible01.bitvision.de with error: Could not bind TCP port 80 because it is already in use by another process on this system (such as a web server). Please stop the program in question and then try again.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ansible02.bitvision.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for ansible02.bitvision.de

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following simulated renewals succeeded:
  /etc/letsencrypt/live/ansible01.bitvision.de/fullchain.pem (success)

The following simulated renewals failed:
  /etc/letsencrypt/live/ansible01.bitvision.de/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

So i have some questions:

  • Can the reconfigure subcommand somehow run in the pre-hook?
  • Is there a way to reload the renewal configuration inside the pre-hook?
  • Can I maybe tell certbot directly from inside the pre-hook to use a certain method (e.g. through a variable like RENEWAL_METHOD)
  • Is there maybe a different approach I can try or is there anything I did wrong in my approaches?

Let me know if more details are needed!

There's an easier (and way dirtier) method.

certbot renew --webroot || certbot renew --standalone
4 Likes

Certbot has a locking mechanism in place to make sure only one instance of Certbot can run at any time. So running Certbot from within Certbot cannot work.

Also, I'm very confused. Why is this necessary in the first place?

4 Likes

If we're talking about a renewal, then it was already issued in one of those two ways and the renewal should continue in the same way.

If we are trying to futureproof the renewals so that even if the system changes from no web service to something using port 80...
Then, yes, a standalone renewal attempt would fail.
But how would you determine the exact webroot path programatically for each vhost?
In your code, all are being set to "/data/www/sites".
[it might be possible to include global web code pointing all vhosts to the same ACME challenge path - which could be used by webroot]
If it went in the opposite direction [from webroot to standalone], it would be much easier.
[but that doesn't seem a likely scenario]

3 Likes

So webroot method is tried and if not succeeding the standalone method is used?

I want that the "scheduled" renewal works everytime, no matter if a webservice is running or not.
Also I want to prevent downtime of the webserver if one is running.

  1. I can't control the scenarios (webservice on or off) because customers will use the system(s) and they can do whatever they want with it. So yes, webroot to standalone is unlikely but not impossible.
  2. Why would I need to determine the exact webroot path for each vhost? Can't I just run all challenges in this one webroot path?

Yup.

That... depends on how your webserver is configured. You can play with try_files to solve this issue.

But remember that --standalone is an IP address level decision, not a virtualhost one.

4 Likes

Why is it so uncertain a webservice is running or not? Usually a webservice runs 24/7?

I get the feeling you're trying to work around something that should be fixed elsewhere.

4 Likes

As I mentioned, the server(s) will not be used by me, but by customers and I want to make sure, that the certificates are renewed independently of a running webservice. So if they decide to stop the webservice for whatever reason, the certificates should be renewed anyway. And when deciding to restart it, the renewal should run through the webservice.

Sorry if I am missing something but let's say you have

customer1.example.com DNS points to IP 1.2.3.4

But, if that server is down you want requests sent to that domain name and IP 1.2.3.4 to be handled by the certbot --standalone. Is that right?

How do you share that IP between these two services? You could certainly do that with a proxy of some kind but if you have that you can manage this differently.

3 Likes

If no Webservice is running, I want to run the renewal of the certificates in standalone mode.
I don't want to handle every request, only the challenge.

I don't quite understand that part.
Why would I want to share that IP between the two service that are on the same server?

Certbot is the ACME Client which initiates the cert request. But, the Let's Encrypt Server will make the HTTP Challenge to the IP address named in the public DNS.

The format of the URL for these requests is like:
http://company1.example.com/.well-known/acme-challenge/Token

It's pretty easy to understand how a webserver would handle the incoming request on port 80.

But, if that webserver is down you plan to start Certbot with --standalone which requires exclusive use of port 80. Certbot makes the cert request and the LE Server makes the same kind of HTTP Challenge request to the same IP from the public DNS. So, Certbot must reply properly to that request arriving on that IP for the challenge to succeed.

I think we are wondering how you plan to structure that. Because there are probably better methods than the path you are on.

4 Likes

How many customers would ever share one single IP?
Would there be a case where some customers [of the same IP] would use HTTP and some would not?

4 Likes

I don't think this usage is possible with Certbot, and I can't imagine any way of safely invoking Certbot using other scripts to accomplish it.

My suggestion is to standardize all the renewals onto the Standalone plugin as follows:

  • configure your webserver to proxypass the acme-challenge directory on port 80 to port 8080
  • instead of invoking Certbot directly, invoke a wrapper script that will check to see if a webserver is running and - if not - spins up a simple webserver that simply does the same proxypass - then..
  • invoke Certbot to use the standalone plugin on the higher port

Certbot can run on port 8080 via the --http-01-port commandline argument, however, the LetsEncrypt ACME Server MUST communicate with your domains(s) on port 80.

4 Likes

I think I was a bit unprecise with me explanation.
I am creating virtual servers e.g. on Amazon aws. Then I am configuring them automatically and give them (the customers) access to it. So every customer gets its own server.
Regarding your question: Every customer has its own IP.

1 Like

So how do you plan to route HTTP requests sent to the customer IP to the Certbot you want to run in --standalone mode. Are you going to run a Certbot --standalone on each virtual server just in case the customer has stopped the web server yet left other services running? If so, why? What good does a fresh cert have if the web service isn't even running? Usually Certbot renew is run multiple times a day so it would renew a cert shortly after the web server is restarted.

Sounds like you have a plan for some kind of hosting service. Be sure to read the following topics for further background

5 Likes

You are right!
But what if the customer isn‘t running a webservice for a long time and the cert expired. Can I still renew it after expiry and if so is there a “deadline“ while I can do so?

yes, with no deadline. and you can always get a new certificate.

5 Likes

Certs last for 90 days and renewal is recommended after 60 days so customer would have to be down for 30 days consecutively. Is that realistic?

If your customers are that, um, volatile you might consider a DNS Challenge instead. If you are not managing their DNS you could have them add a CNAME to a domain name you do control. Then you don't have to worry about what is running or how you'd have Certbot standalone running on a dormant EC2 instance.

5 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.