Errors attempting to auto-renew a certificate created using --manual

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
https://homecotton.co.uk

I ran this command:

certbot certonly --dry-run --manual --preferred-challenges=dns --manual-auth-hook /etc/certbotscripts/godad_authenticator_eg.sh --manual-cleanup-hook /etc/certbotscripts/godad_cleanup_eg.sh -d homecotton.co.uk

It produced this output:

$ certbot certonly --dry-run --manual --preferred-challenges=dns --manual-auth-hook /etc/certbotscripts/godad_authenticator_eg.sh --manual-cleanup-hook /etc/certbotscripts/godad_cleanup_eg.sh -d homecotton.co.uk
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for homecotton.co.uk

NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?

(Y)es/(N)o: Y

Running manual-auth-hook command: /etc/certbotscripts/godad_authenticator_eg.sh
Error output from manual-auth-hook command godad_authenticator_eg.sh:
Traceback (most recent call last):
File "", line 1, in
KeyError: 'result'
Traceback (most recent call last):
File "", line 1, in
KeyError: 'result'

Waiting for verification...
Challenge failed for domain homecotton.co.uk
dns-01 challenge for homecotton.co.uk
Cleaning up challenges
Running manual-cleanup-hook command: /etc/certbotscripts/godad_cleanup_eg.sh
Some challenges have failed.

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: homecotton.co.uk
    Type: unauthorized
    Detail: Incorrect TXT record
    "" found at
    _acme-challenge.homecotton.co.uk

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address.

Note: the log indicates an exception raised in the pythonry:
raise errors.AuthorizationError('Some challenges have failed.')

My web server is (include version):

Server version: Apache/2.4.41 (Ubuntu)
Server built: 2023-03-08T17:32:54

The operating system my web server runs on is (include version):
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="Bugs : Ubuntu"
PRIVACY_POLICY_URL="Data privacy | Ubuntu"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

My hosting provider, if applicable, is:
N/A
(godaddy is my DN provider)

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.40.0

Hi
I have a web server running on a local machine and a DN provided by godaddy. I originally used the following command to generate a certificate:
certbot certonly --manual --preferred-challenges dns -d homecotton.co.uk

Which worked, but requires me to manually enter the value for acme txt record in my go daddy account.

I'd like to be able to renew this automatically when the certificate expires.
Initially I thought I would simply be able to run renew and provide a script that updated the txt record with the new key in place of manually adding it - something like this:
$ certbot renew --dry-run

However from my searches and the documentation (User Guide ā€” Certbot 2.5.0 documentation) it looks like I need to use --manual-auth-hook and --manual-cleanup-hook and provide appropriate scripts to do the up date. Is that correct ? Should this allow me to automatically update the certificate ?

As a start I took the example scripts from the documentation, edited the the servername to api.godaddy.com and ran the above command, but getting the errors indicated.

When I ran the command it came up requiring user input regarding the ip address - does anyone know how I can prevent that ?

Does anyone know how I should write the script (for godaddy) in accordance with certbot requirements ?
Note: Not sure which parts of the script are icloud specific. For example in my scripts the auth header looks like "Authorization: sso-key :", but in the example scripts I got it seems to use "X-Auth-Key: $API_KEY" - possibly thats icloud specific ? Or the key format is different ?

And also the best way to debug the scripts ?

2

If you want to use the --manual with automatic renewal, then yes, you should provide correct and working scripts for the hooks you've mentioned.

Two things:

  • the documentation also explicitely says the script mentioned is an example only and should not be used as-is;
  • the example script in the documentation is for Cloudflare. Simply changing the API URL from the Cloudflare API to the GoDaddy API doesn't work, as Cloudflare and GoDaddy have their own, different API.

Yes, either update Certbot to a more recent version or use the now removed option --public-ip-logging-ok. Also, the question shouldn't popup when renewing.

With regard to your other questions:

It's probably better to forget about the --manual plugin and use the incorrect approach of using the example script as start for a GoDaddy script, but use an authenticator plugin which works with the GoDaddy DNS API to begin with. Such as:

Note that these are third party plugins and are not supported by the Certbot team, although the first one is actually developed by one of the Certbot developers.

6 Likes
3

Automating DNS-01 authentication requires an ACME client that can update the DNS zone via API.
I would recommend researching which ACME clients work with your operating system and can also work with your DSP [DNS Service Provider = GoDaddy DNS].
OR
Switch DSPs to one that works better with your desired ACME client.

Trying to write your own DNS plugin would be the very last choice.

4 Likes
4
2 Likes
5

HI

Thank you for your replies.

Not knowing how the renewal works underneath I am a bit confused why there is a dns provider specific element to this. What is it that means the renewal is supplier specific ?

If you want to use the --manual with automatic renewal, then yes, you should provide correct and working scripts for the hooks you've mentioned.

Do you know if there are any examples of scripts that would work for godaddy ? Is it simply that the api is different ? In which case I would simply need to find the correct commands to get the 'ZONE_ID' and post the validation with a return 'RECORD_ID' (whatever they are - do you know what zoneid and record id refer to?) ? Or are there other differences (different protocol, different tools) ?

It's probably better to forget about the --manual plugin and use the incorrect approach of using the example script as start for a GoDaddy script, but use an authenticator plugin which works with the GoDaddy DNS API to begin with. Such as:

It does look like I should probably be using certbot-dns-godaddy. However I also notice that the certbot version on my ubuntu is quite old (0.4) - the go-daddy-dns site is suggesting I should have >1.7 - for some reason updating this does not seem to be as simple as it should be, but I will give this a go.

As a back up plan, given that I was able to use --manual to get the certificate by updating my record manually I thought I could simply write a script to automate that process - user input,/output updating the record using the apis. Do you know if it is possible to run a dummy cerbot --manual so I can get examples of the output it gives and input it expects ? (I do not have my original output)

1 Like
6

The idea is the same, but the procedure is different. The idea is the adding (and subsequent removal) of a specific TXT resource record. That's always the same. But the API for how to add (and remove) that TXT resource record in an automated fashion differs per DNS provider.

I'm afraid it's probably the latter: different protocols. ZONE_ID and RECORD_ID are probably Cloudflare specific.

For Ubuntu it's recommended to use snap instead of the OS package. See Certbot Instructions | Certbot on how to install Certbot using snap for Ubuntu.

4 Likes
7

Thank you for your replies.

For Ubuntu it's recommended to use snap instead of the OS package. See Certbot Instructions | Certbot on how to install Certbot using snap for Ubuntu.
Yes, I initially had problems with trying to install with snap, but after I completely cleaned up anything I had installed related to certbot and re-installed via snap it now seems to have installed OK.
$ certbot --version
certbot 2.5.0

But the API for how to add (and remove) that TXT resource record in an automated fashion differs per DNS provider.

I found a this site:
https://chariotsolutions.com/blog/post/automating-lets-encrypt-certificate-renewal-using-dns-challenge-type/
which gives an example of a godaddy version and does indeed suggest the type of quries required are provider specific.

When I try to use these scripts things get a bit further, but I get the following error on doing a dry-run renewal:

certbot --dry-run -d homecotton.co.uk --agree-tos --register-unsafely-without-email --manual --preferred-challenges=dns --manual-auth-hook /etc/certbot/scripts/gdaddy.sh --manual-cleanup-hook /etc/certbot/scripts/gdaddy-clean.sh --manual-public-ip-logging-ok certonly
Use of --manual-public-ip-logging-ok is deprecated.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Use of --manual-public-ip-logging-ok is deprecated.

An RSA certificate named homecotton.co.uk already exists. Do you want to update
its key type to ECDSA?

(U)pdate key type/(K)eep existing key type: K
Simulating renewal of an existing certificate for homecotton.co.uk
Hook '--manual-auth-hook' for homecotton.co.uk ran with error output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 109 0 0 100 109 0 137 --:--:-- --:--:-- --:--:-- 137
100 109 0 0 100 109 0 137 --:--:-- --:--:-- --:--:-- 137

Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
Domain: homecotton.co.uk
Type: unauthorized
Detail: Incorrect TXT record found at _acme-challenge.homecotton.co.uk

Hint: The Certificate Authority failed to verify the DNS TXT records created by the --manual-auth-hook. Ensure that this hook is functioning correctly and that it waits a sufficient duration of time for DNS propagation. Refer to "certbot --help manual" and the Certbot User Guide.

Hook '--manual-cleanup-hook' for homecotton.co.uk ran with error output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 71 0 0 100 71 0 181 --:--:-- --:--:-- --:--:-- 180
100 71 0 0 100 71 0 180 --:--:-- --:--:-- --:--:-- 180
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details

It looks like at least part of the requests are working since is indeed the value I have in my txt record, I see what looks like valid record information in the log file and the POST command in the clean script seems to have been executed successfully (at least there is a txt record named _acme-challenge.homecotton with a value of 'clean' in my DNS records)

Do you know if there are likely to be problems because I am using an updated version of certbot ? Would I need to do a manual run from scratch using the new version ?

When I run the script it comes up with a prompt:

An RSA certificate named homecotton.co.uk already exists. Do you want to update
its key type to ECDSA?

I assume this is a certbot version thing. I am currently pressing 'K' for keep - is that likely to cause an issue ?
Eventually this will need to be automated, so is there a way of preventing the necessity for user input on this ?

Not sure how a dry run works, but I assume it knows whether the operation would have passed and if so report success ?

Have you any pointers on how to debug the error ? It looks like it does some comparison with a txt record and does not like it, but I have not seen where the match that causes the failure occurs. Also since this is a dry run I would not expect the actual txt record to be updated.

1 Like
8

Try increasing the sleep in your gdaddy.sh script to 120 seconds. 30s might be too low for GoDaddy authoritive DNS servers to sync.

No. That's fine but an ECDSA key is probably fine too.

Yes, exactly that.

The TXT value should change during --dry-run. A --dry-run goes through all the steps but using the Let's Encrypt Staging System and also does not save the resulting test certs.

Also, the godaddy-clean.sh sets the TXT value to 'clean'. But, right now there is an actual value in your DNS so maybe it isn't working as you expect.

I didn't study it and I am not a GoDaddy DNS expert. But, a different ACME Client (acme.sh) has a GoDaddy script. Maybe compare the one you are using with that to see if it should work. (link here)

3 Likes
9

Also, the godaddy-clean.sh sets the TXT value to 'clean'. But, right now there is an actual value in your DNS so maybe it isn't working as you expect.

Mmm - yes you are correct. The script(s) appear(s) to be assuming a record name _acme-challenge.homecotton, but when I created the record I was told to give the name _acme-challenge. When I changed the auth and cleanup script to use the raw _acme-challenge it came back as a success, but the _acme-challenge txt record value ends up as 'clean'. Not sure if that should happen ? Do I need to change it back to the old value ?

Do you know if the default record name changed with later versions ?

Thank you for link to the script - I'll take a look to see what it does.

1 Like
10

TXT records should be deleted after each use.
So, no, you don't have to put anything back.

3 Likes
11

Most scripts do that but the one they used sets the value to the literal "clean" during cleanup.

That's not horrible if it successfully rewrites that the next time with the new value. It even helps debug like in this situation :slight_smile:

4 Likes
12

Ok. Thank you.

As regards the user prompt asking about ECDSA:
If I press 'Y' to update the key will that be harmless - i.e the key will still work and be valid ?
If I do that, do you know if it will it prevent the question being asked again - so there is no requirement for user input when I run it ?

13

For most uses, yes.
But there still exist legacy software that can't use ECDSA certs.

Ideally, you won't be running it manual [after the first success].
So, you don't need to worry about that.

3 Likes
14

Ideally, you won't be running it manual [after the first success].
? I thought I would need to run a cron script that runs the certbot command when the certificate is about to expire.

15

That or a systemd timer. But, the renew command is just certbot renew which reviews and renews all certs based on their renewal config files in /etc/letsencrypt/renewal

3 Likes
16

Beats me, but if you're using an up to date version of Certbot using snap now I'd suggest to stop messing with those manual authenticator scripts and just try certbot-dns-multi.

3 Likes
17

But, the renew command is just certbot renew which reviews and renews all certs based on their renewal config files in /etc/letsencrypt/renew

How would I tell that about --manual-auth-hook and --manual-cleanup-hook ? Under [renewalparams] in the conf file ? or putting then in the right place under /etc/letsencrypt/renewal-hooks/ ?

Beats me, but if you're using an up to date version of Certbot using snap now I'd suggest to stop messing with those manual authenticator scripts and just try certbot-dns-multi.

Yes I guess now I have I could try to install the dns module - would feel like starting again though and no doubt there would be problems going that route also.

18

Certbot will update the renewal conf file once you get a successful cert.

3 Likes
19

Ah. Ok, Thank you

So I do a real renewal and after that I should simply be able to do certbot renew or certbot --dry-run renew (and my cron/systemd timer can simply run certbot renew) ?

2 Likes
20

Yes. In your case the "real renewal" is getting a new cert with a different method that allows automated renewals.

3 Likes