Certbot renewal takes a very long time to parse site config with very large list of redirects

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: acmssearch.sl.nsw.gov.au

I ran this command: certbot renew

It produced this output: None. Times out while parsing site config with very large list of redirects.

My web server is (include version): nginx 1.24.0

The operating system my web server runs on is (include version): Ubuntu 24.04

My hosting provider, if applicable, is: NA

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.11.0

2

OK, so now to describe the behaviour, historical method of managing this and why Im asking for advice.

I run a site whose purpose is to provide redirects for resources in a retired library catalogue system. Some years ago the catalogue admin exported direct links for each legacy catalogue resource and the corresponding direct link to the each resource in the new catalogue system.

We have been using nginx to ensure our visitors and partners with links to old resources can still access the resource they are interested in by redirecting (with permanently moved HTTP header for web indexers to update their data) to the resource they want in the new catalogue.

This resulted in the site's config including very large lists of redirects via include files. The web server handles this just fine but understandably Certbot takes so longer parsing the site config that it does not get around to requesting the certificate.

On older hosts I worked around this by running certbot renew in a cronjob preceded by a script to temporarily remove the redirect include files from the site's config, run the renewal, then update the site config with the include files again. Worked just fine.

On moving to Ubuntu 24.04, I understand that the renewal is scheduled automatically by the Certbot snap, in some part to help reduce the load on the LE servers by randomising the time the renewal runs.

This is fine and I tried to work within this approach and have my scripts run using renewal pre- and post-hooks.

What I have found in troubleshooting is that the pre-hook is not run until Certbot has parsed the site config so the large redirect includes remain in the site config and Certbot times out.

Now as a temporary workaround, I can manually run the pre-hook script and then run certbot renew. This works fine. Certbot can parse the modified site config, it runs the pre-hook script (which is superfluous but does no harm), renews the certificate and then runs the pos-hook script.

This is fine to get the host properly configured and ensure all the certs are up to date initially but the scheduled renewal is still going to run and time out.

I can obviously create a cronjob similar to that on my old server to process the renewal but this will run at a fixed time and will not stop the other Certbot-scheduled renewal attempt from running as well. From my reading even if I'm able to disable the Certbot schedule it will be recreated with any updates to Certbot.

Sorry for the essay.

Thank you for any ideas or recommendations.

1 Like
3

How fast is your storage and how large is the actual file it's reading?

1 Like
4

Thanks for replying.

There are several redirect include files but three large ones of 23MB, 43MB and 45MB.

The server runs on a hyperconverged virtualisation cluster with many disks.

1 Like
5

Do you use the nginx plugin? If you do you may just be running into a problem that can be solved by increasing the sleep that Cerbot needs after making the temp changes to your config.

--nginx-sleep-seconds NGINX_SLEEP_SECONDS
Number of seconds to wait for nginx configuration
changes to apply when reloading. (default: 1)

You could also avoid Certbot needing to parse your config at all using --webroot method

3 Likes
6

I assume Certbot uses the nginx plugin but the issue isn't with reloading nginx, it's with parsing the large redirect include files. Sorry if I misunderstand your point.

I include logging in my pre-hook script which lets me see that the pre-hook (to modify the site config, hiding the redirects) that I expected to be run before Certbot parses the site config is not called until after Cerbot parses the site config. Manually running that script first allows Certbot to parse the site config very quickly and then Certbot calls the pre-hook script. If I don't run the script manually, Certbot doesn't get through parsing the site config at all.

7

Certbot would only parse the nginx config as part of the challenge prep.

Or, for new domain requests to create the https server block and related changes. But that isn't part of renew

If you add the sleep I describe you may not need those hooks

Or switch to --weboot method then no parsing is needed

4 Likes
8

Sorry, I do not understand the nginx-sleep-seconds. There's no configuration of site configs being performed. These are all sites configured to use LE certs with renewal config files.

The delay seen during certbot renew only occurs if the large redirect include files are referenced in the site's config. My script removes those temporarily and certbot renew runs promptly as expected.

I understand, correct me if I'm wrong, that the systemd timer created automatically for renewal runs:

certbot renew

Are you suggesting I modify this to run a separate timer for each vhost on the server and run

certbot certonly --webroot [path to web root]

for each of the vhosts?

9

Depending on how Certbot was configured initially for those certs (i.e., with the --nginx option), Certbot will read and change the configuration for the renewal of any certificate so it can add (and remove) the challenge.

This kinda proves Mike was right.

2 Likes
10

Thank you all for your help and advice.

2 Likes
11

Do you need help adding the nginx sleep to your renewal config ?

I think switching from --nginx to --webroot is the better solution. We can walk you through that too. For either change, just show us the contents of the renewal config file for an affected domain in the /etc/letsencrypt/renewal folder. You can remove the account id if you wish but leave the rest.

We can instruct how to re-issue your original command to update the renewal config. Possibly even by using the reconfigure command.

At the risk of repeating myself ... here is roughly how the --nginx plugin works (ignoring hooks)

Certbot adds temp code to the nginx server block for the domain(s)

It "reloads" nginx to make those changes effective. This is an asynchronous request so Certbot waits 1 second for nginx to reload. For large nginx systems that is not long enough and you need to use the sleep seconds I described.

After that sleep it tells the Let's Encrypt server to make the challenge. If successful Certbot retrieves the cert files. In either case it removes the temp changes from your nginx config and issues another async reload. This reload removes the effect of the temp changes but also has nginx seeing the fresh cert if one was issued.

In your case, using those hooks is working around the async reload sleep "problem" by simplifying your nginx config so that it reloads within 1 second.

For this the --webroot is probably better than --nginx though. The webroot option never reads your nginx config and does not auto-issue any reload. It just places a file in a folder named by the root in your nginx config. We usually setup a location in nginx for these.

Let us know if you need help with any of this.

4 Likes
12

I agree with this, but I want to note the following:

Switching away from the nginx Certbot plugin is the most important thing here. The nginx plugin does not deal with large files or numerous domains very well.

The plugin is useful for setting up an initial system for novice uses. Once it is set up, however, it's not really needed anymore. The other plugins are all considerably faster and less buggy.

Most users here prefer webroot; I prefer using the standalone plugin on a higher port, or do everything by DNS. The key point though is that you're at a place where you've outgrown the utility of the nginx plugin and should migrate to another one.

6 Likes
13

It does reload nginx at renewal automatically. But of course this can also be done using a --deploy-hook.

3 Likes
14

Hi everyone. Thank you all for your expertise and patient advice. Sorry, I have responded sooner: it was night time here when your newer posts were added.

It took me a while to catch on but yesterday's advice had time to reach the right spots in my old brain overnight and by morning I had an understanding of what I needed to do.

For the benefit of others who might wish to switch from using the nginx plugin to the webroot plugin here is what I did to get my renewals running.

High level

For each existing site on the host:

  1. Update nginx site config to allow clear text HTTP requests for /.well-known/acme-challenge/ so that LE server can access a temporary file created by Certbot to confirm the host is authorised to request a cert for the site
  2. Run a cerbot command to request a certificate for the site using the webroot authentication method
  3. Confirm that the site's certbot renewal script was updated to use webroot authentication for future renewals
  4. Test a dry run renewal

Example

  1. nginx config update

Add the following to the clear text HTTP server section of your nginx site config file:

    # allow LE cert renewal authentication
    location /.well-known/acme-challenge/ {
        allow all;
    }

    # redirect all other requests to HTTPS
    location / {
            return 301 https://$host$request_uri;
    }
  1. Test the change

nginx -t

  1. If the test is successful, reload nginx

systemctl reload nginx

  1. Request a certificate for the site using the webroot command

certbot certonly --webroot --webroot-path "[FULL PATH TO SITE WEB ROOT]" -d [WEB SITE FQDN]

  1. Check the content of the site's Certbot renewal config file

cat /etc/letsencrypt/renewal/[WEB SITE FQDN].conf

Example output:

# renew_before_expiry = 30 days
version = 2.11.0
archive_dir = /etc/letsencrypt/archive/[WEB SITE FQDN]
cert = /etc/letsencrypt/live/[WEB SITE FQDN]/cert.pem
privkey = /etc/letsencrypt/live/[WEB SITE FQDN]/privkey.pem
chain = /etc/letsencrypt/live/[WEB SITE FQDN]/chain.pem
fullchain = /etc/letsencrypt/live/[WEB SITE FQDN]/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = [REDACTED]
authenticator = webroot
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa
webroot_path = [FULL PATH TO SITE WEB ROOT],
[[webroot_map]]
[WEB SITE FQDN] = [FULL PATH TO SITE WEB ROOT]

Note particularly the authenticator line and the last three lines which have been added for webroot authentication.

  1. Create a script to reload nginx after a successful cert renewal

Place your script in /etc/letsencrypt/renewal-hooks/deploy.

  1. Simulate a full renewal run

certbot renew --dry-run

Review the output and confirm that all sites would have been renewed successfully.

Thank you again for all your patience advice. Maybe my dimness will help someone else out there.

Happy cert requesting.

5 Likes
15

Excellent! I know it sometimes takes my brain a bit to disrupt old concepts :slight_smile:

One tip for future readers who may not be as comfortable with hook scripts. You can instead put --deploy-hook 'systemctl reload nginx' in the Certbot command itself. That places this option into the renewal config file just for that certificate. Of course, use the appropriate command for any other environ.

Congrats on working all that out !

3 Likes
16

The typical (canonical?) way to do this in nginx is to have a "macro" text file with those lines in, saved under /etc/nginx/macros (or similar), then just place an include line on all the sites that need this:

include /etc/nginx/macros/letsencrypt.conf

This strategy not only lets you consolidate the logic into a shared config file, but you can quickly change your integration for every hosted site by editing that file.

2 Likes
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.