Certbot does not call cleanup script after deleting and recreating a certificate

Help
1

My domain is: refimpl.maxant.ch

I ran this command:

certbot certonly --manual --preferred-challenges=http --email certs@maxant.ch --agree-tos --manual-public-ip-logging-ok --non-interactive --force-renewal --manual-auth-hook mycertbot-authenticator.sh --manual-cleanup-hook mycertbot-cleanup.sh -d refimpl.maxant.ch

the cleanup script creates some mappings for my web server.

certbot delete -n --cert-name refimpl.maxant.ch

then I rerun the first script, and this time, mycertbot-cleanup.sh is not executed. is that normal? if so, why? I was hoping that certbot would be idempotent.

My web server is (include version): custom
The operating system my web server runs on is (include version): ubuntu 22.04.1 LTS
My hosting provider, if applicable, is: n/a
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.21.0

2

Please stop testing this kind of stuff on the production environment. Currently you've maxed out the 5 duplicate certs per week already and thus have hit the duplicate rate limit. See crt.sh | refimpl.maxant.ch for your issued certs. Please use the staging environment for testing.

Also, the script should run just fine every time you're using the manual plugin. That said, I'm not sure what "creates some mappings" entails, but I'm pretty sure that should be in a deploy hook script and not in the cleanup script for the manual authenticator.

That said, if you think you've found a bug in Certbot, please first update to the most up to date version, which is currently 2.4.0.

4 Likes
3

Notably, none of your --manual hooks will get invoked if the Certificate Authority chooses to use an already-valid authorization for your order, so this advice is very on point:

Since you have issued a bunch of duplicate certificates, I assume that is indeed what happened. Certificates 2 through 5 would not have called either script.

Use --dry-run so that a) Certbot use the staging server and b) Certbot deactivates any already valid authorizations, so your hooks will be guaranteed to be called.

5 Likes
4

Probably don't want that option, but I will let the knowledgeable Let's Encrypt community volunteers advise.

1 Like
5

Ah, good point indeed! Didn't realise that :slight_smile:

Anyway, OP might want to take a look at the Certbot documentation before proceeding with anything else.

4 Likes
6

I'm using Ubuntu 22.04 and after an upgrade, I still only have certbot 1.21.0. Why doesn't it upgrade to 2.4.0?

7

I'll redesign my renewals - I was hung up on using create for simplicity, but can't see why I shouldn't be using renew. Thanks for the tip on --dry-run

1 Like
8

Please see https://certbot.eff.org/ for the most up to date instructions: Certbot has moved to snap for their main method of distribution.

There is nothing wrong with certbot renew, but there is much wrong with using --force-renewal.

4 Likes
9

My original question is still not quite answered. I am not trying to renew certificates, I am trying to recreate them after deleting them. I have a case where a customer of mine decides to delete a domain which I am hosting for them, so I run certbot delete -n --cert-name theirdomain.com

A few hours later, the customer returns and so I re-run the original certbot certonly --manual --preferred-challenges=http --email certs@maxant.ch --agree-tos --manual-public-ip-logging-ok --non-interactive --manual-auth-hook mycertbot-authenticator.sh --manual-cleanup-hook mycertbot-cleanup.sh -d theirdomain.com

(with or without --force-renewal)

The point is that I deleted the certificate when they left. And now that they are returning, I want to recreate the certificates since they are no longer on my disk.

What is the correct process for doing that? As I originally said, the cleanup script is not called when I recreate the certificates. I tested this with other domain names, not just "refimpl.maxant.ch". See crt.sh | refimpl3.maxant.ch - the first certificate was created "when the customer first hosted their domain on my server" and the second one after I removed the certificates and tried to recreate them, "when the customer returned".

certbot logs:

2023-03-07 22:40:15,123:DEBUG:certbot._internal.main:certbot version: 1.21.0
2023-03-07 22:40:15,124:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2023-03-07 22:40:15,124:DEBUG:certbot._internal.main:Arguments: ['--manual', '--preferred-challenges=http', '--email', 'certs@maxant.ch', '--agree-tos', '--manual-public-ip-log
ging-ok', '--non-interactive', '--force-renewal', '--manual-auth-hook', 'certbot-authenticator.sh', '--manual-cleanup-hook', 'certbot-cleanup.sh', '-d', 'refimpl3.maxant.ch']
...

NOW MY CLEANUP SCRIPT RUNS

2023-03-07 22:40:28,109:DEBUG:certbot._internal.error_handler:Calling registered functions
2023-03-07 22:40:28,110:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-03-07 22:40:28,110:INFO:certbot.compat.misc:Running manual-cleanup-hook command: certbot-cleanup.sh
2023-03-07 22:40:28,124:DEBUG:certbot._internal.display.obj:Notifying user: Hook '--manual-cleanup-hook' for refimpl3.maxant.ch ran with output:
 certbot-cleanup> cleanup is running for refimpl3.maxant.ch
...
2023-03-07 22:40:29,982:DEBUG:certbot._internal.display.obj:Notifying user: 
Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/refimpl3.maxant.ch/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/refimpl3.maxant.ch/privkey.pem
This certificate expires on 2023-06-05.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.
2023-03-07 22:40:29,984:DEBUG:certbot._internal.display.obj:Notifying user: If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le

...
NOW THE REMOVAL STARTS, about 90 seconds later:

2023-03-07 22:41:54,647:DEBUG:certbot._internal.main:certbot version: 1.21.0
2023-03-07 22:41:54,647:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2023-03-07 22:41:54,647:DEBUG:certbot._internal.main:Arguments: ['-n', '--cert-name', 'refimpl3.maxant.ch']
2023-03-07 22:41:54,647:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-03-07 22:41:54,656:DEBUG:certbot._internal.log:Root logging level set at 30
2023-03-07 22:41:54,657:INFO:certbot._internal.storage:Removed /etc/letsencrypt/renewal/refimpl3.maxant.ch.conf
2023-03-07 22:41:54,657:DEBUG:certbot._internal.storage:Removed /etc/letsencrypt/live/refimpl3.maxant.ch/cert.pem
2023-03-07 22:41:54,657:DEBUG:certbot._internal.storage:Removed /etc/letsencrypt/live/refimpl3.maxant.ch/privkey.pem
2023-03-07 22:41:54,657:DEBUG:certbot._internal.storage:Removed /etc/letsencrypt/live/refimpl3.maxant.ch/chain.pem
2023-03-07 22:41:54,657:DEBUG:certbot._internal.storage:Removed /etc/letsencrypt/live/refimpl3.maxant.ch/fullchain.pem
2023-03-07 22:41:54,657:DEBUG:certbot._internal.storage:Removed /etc/letsencrypt/live/refimpl3.maxant.ch/README
2023-03-07 22:41:54,657:DEBUG:certbot._internal.storage:Removed /etc/letsencrypt/live/refimpl3.maxant.ch
2023-03-07 22:41:54,658:DEBUG:certbot._internal.storage:Removed /etc/letsencrypt/archive/refimpl3.maxant.ch
2023-03-07 22:41:54,658:DEBUG:certbot._internal.display.obj:Notifying user: Deleted all files relating to certificate refimpl3.maxant.ch.

NOW THE RECREATION STARTS, 22 seconds later:

2023-03-07 22:42:16,385:DEBUG:certbot._internal.main:certbot version: 1.21.0
2023-03-07 22:42:16,386:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2023-03-07 22:42:16,386:DEBUG:certbot._internal.main:Arguments: ['--manual', '--preferred-challenges=http', '--email', 'certs@maxant.ch', '--agree-tos', '--manual-public-ip-logging-ok', '--non-interactive', '--force-renewal', '--manual-auth-hook', 'certbot-authenticator.sh', '--manual-cleanup-hook', 'certbot-cleanup.sh', '-d', 'refimpl3.maxant.ch']

...

-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

2023-03-07 22:42:19,850:DEBUG:acme.client:Storing nonce: C878sA-WK58f8FIBBTBTU-FkIt8fKHjmZppPzmXEQe5VnFY
2023-03-07 22:42:19,854:DEBUG:certbot._internal.storage:Creating directory /etc/letsencrypt/archive/refimpl3.maxant.ch.
2023-03-07 22:42:19,855:DEBUG:certbot._internal.storage:Creating directory /etc/letsencrypt/live/refimpl3.maxant.ch.
2023-03-07 22:42:19,855:DEBUG:certbot._internal.storage:Writing certificate to /etc/letsencrypt/live/refimpl3.maxant.ch/cert.pem.
2023-03-07 22:42:19,856:DEBUG:certbot._internal.storage:Writing private key to /etc/letsencrypt/live/refimpl3.maxant.ch/privkey.pem.
2023-03-07 22:42:19,856:DEBUG:certbot._internal.storage:Writing chain to /etc/letsencrypt/live/refimpl3.maxant.ch/chain.pem.
2023-03-07 22:42:19,856:DEBUG:certbot._internal.storage:Writing full chain to /etc/letsencrypt/live/refimpl3.maxant.ch/fullchain.pem.
2023-03-07 22:42:19,856:DEBUG:certbot._internal.storage:Writing README to /etc/letsencrypt/live/refimpl3.maxant.ch/README.
2023-03-07 22:42:19,871:DEBUG:certbot._internal.plugins.selection:Requested authenticator manual and installer <certbot._internal.cli.cli_utils._Default object at 0x7f679a2dcbb0>
2023-03-07 22:42:19,871:DEBUG:certbot._internal.cli:Var pref_challs=http (set by user).
2023-03-07 22:42:19,872:DEBUG:certbot._internal.cli:Var authenticator=manual (set by user).
2023-03-07 22:42:19,872:DEBUG:certbot._internal.cli:Var manual_auth_hook=certbot-authenticator.sh (set by user).
2023-03-07 22:42:19,872:DEBUG:certbot._internal.cli:Var manual_cleanup_hook=certbot-cleanup.sh (set by user).
2023-03-07 22:42:19,873:DEBUG:certbot._internal.storage:Writing new config /etc/letsencrypt/renewal/refimpl3.maxant.ch.conf.
2023-03-07 22:42:19,883:DEBUG:certbot._internal.display.obj:Notifying user: 
Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/refimpl3.maxant.ch/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/refimpl3.maxant.ch/privkey.pem
This certificate expires on 2023-06-05.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.
2023-03-07 22:42:19,885:DEBUG:certbot._internal.display.obj:Notifying user: If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le

As you can see almost at the end, the cleanup script is even referenced. But it is not called. My cleanup script logs stuff, that is displayed the first time round, but not the second time around.

Perhaps it is necessary to revoke the certificate, instead of just deleting it locally?

10

No, no, no. Revoking the cert is never (OK, almost never) the answer.

...and as you've already been told, the cleanup script is used to clean up the residue of a validation attempt (as noted in the docs too: User Guide — Certbot 2.6.0 documentation). If no validation is being done (because you've already validated that domain with your account recently enough), then there's no cleanup to do, and thus the cleanup script isn't called.

3 Likes
11

This is a workflow that is a tricky match for the default Certbot and Let's Encrypt issuance concepts.

As you've seen, people here are quite concerned about the Let's Encrypt rate limits, which are imposed because creating new certificates uses some certificate authority resources, so the certificate authority wants to prevent people from doing it in a duplicative way, or more often than necessary.

If you have a customer who stops being your customer and then starts again in a short period of time, Let's Encrypt's software is effectively going to conclude (at some point) that you are using the service in a wasteful way, because you failed to anticipate whether or not the customer would continue to be your customer or not, and used the service every time that changed (in a short period of time). You would not be permanently banned for this or anything, but you could be forced to wait for a week before continuing to issue duplicative certificates.

If you think it's likely to be a frequent or foreseeable situation that the same customer goes away and then comes back, I would suggest not deleting the certificate locally at all until a week has passed following the customer leaving. That is, I would make some kind of notation in a database or file in your system that a customer is "pending deletion" and then actually run the delete command only after a week has gone by. If the customer returns during that time, I would suggest detecting the "pending deletion" and simply marking it as no longer applying.

Other options are rather difficult because the certificate authority never possesses your private key, without which your issued certificate is useless. So the certificate authority doesn't really have any kind of "undo" feature that would help you here; it can't give you necessary secret information that it never possessed, after you've deleted (potentially) your only copy of that information!

I agree with other posters who pointed out that revocation and the cleanup-hook in Certbot are both concepts that are intended for entirely different situations than what you describe and probably neither will be useful in any way or to any extent for what you're trying to do.

As an analogy, some people have a workflow where they request Let's Encrypt certificates inside of Docker container instances. That's totally fine, except that some people do this with ephemeral instances which then get deleted and recreated from scratch periodically. Those people are likely to hit rate limits, because they're issuing new certificates every time the container is recreated or restarted. We always advise people in this situation to create some kind of persistent storage that the certificates and private keys will be persistently stored in, so that they don't have to recreate them if the container is recreated. Whether or not you're actually using Docker containers, the situation you describe is kind of analogous—you have some kind of integration that's being automatically created, then deleted, then recreated again. For just the same reason as with the Docker users, it would be more appropriate if you could persistently store the certificates (ideally during most of their period of validity, perhaps only deleting them when Certbot is going to attempt to renew them automatically if you happen to know that the renewal is not desired by the customer, but at least for a week!).

7 Likes
12

Thank you @schoen , that is a very clear answer.

2 Likes
13

I would add:
What is the need of immediately deleting the certificate?
[given that it will autoexpire within a few months]

Unless you are obtaining a single cert with many different domains on it, I don't really see the need for such an immediate change.

4 Likes
14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.