Renewal Failure - ListHostedZones / Security Token Invalid - Centos 7 / Route 53 plugin

Help
1

My domain is:
violet-book.com

I ran this command:

certbot renew

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/renewalvars.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Cert is due for renewal, auto-renewing...
Found credentials in environment variables.
Plugins selected: Authenticator dns-route53, Installer None
Renewing an existing certificate

Performing the following challenges:
dns-01 challenge for [subdomain 1 - edited for security reasons]
dns-01 challenge for violet-book.com
dns-01 challenge for [subdomain 2 - edited for security reasons]
dns-01 challenge for www.violet-book.com

Cleaning up challenges

Attempting to renew cert (violet-book.com) from /etc/letsencrypt/renewal/renewalvars.conf produced an unexpected error: An error occurred (InvalidClientTokenId) when calling the ListHostedZones operation: The security token included in the request is invalid.

To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.. Skipping.

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/violet-book.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/violet-book.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

My web server is (include version):
The operating system my web server runs on is (include version):
CentOS 7
My hosting provider, if applicable, is:
Digital Ocean
I can login to a root shell on my machine (yes or no, or I don't know):
yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No, via command line / SSH
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.37.0

Hi all, I've been trying to fix this error all day, but can't seem to figure out a way forward. The cert that expired (today) was a Let's Encrypt cert so certbot likely worked at some stage, but my DevOps engineer has gone AWOL and I'm not attempting to renew without luck. We have 3 subdomains, so ideally would renew with a wildcard cert. The DNS is managed via Route 53 and the official docker repo with the certbot-route-53 plugin are both installed on the server. Nginx reverse proxy is used for routing.

I checked the IAM access token and it is valid with full Route53 access, so don't know where else to turn?

1 Like
2

Please show this file:
[you can obfuscate/hide/remove any account/password info, if included]

1 Like
3

The error code seems pretty unambiguous:

InvalidClientTokenId
The X.509 certificate or AWS access key ID provided does not exist in our records

If something has gone wrong with this credential, you could consider creating a new one, attaching the same policy, and seeing if that helps?

2 Likes
4

Thanks for the reply, much appreciated!

The renewalsvars.conf (which I assume is the config file for all this, says the following:

archive_dir = /etc/letsencrypt/archive/violet-book.com
cert = /etc/letsencrypt/live/violet-book.com/cert.pem
privkey = /etc/letsencrypt/live/violet-book.com/privkey.pem
chain = /etc/letsencrypt/live/violet-book.com/chain.pem
fullchain = /etc/letsencrypt/live/violet-book.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = dns-route53
account = [hidden for security reasons]
server = https://acme-v02.api.letsencrypt.org/directory
1 Like
5

I thought about trying this, but currently (apart from the cfg file I just posted in my last reply), I'm not sure how it was all configured. There isn't (to my knowledge as I can't find it), a ".aws" folder with any additional config settings on the server.

I read through the linked instructions on the certbot-dns-route52 site, but none of those seem to be matching up with what is on the server, yet it appears to be running and failing for another reasons, so the config must be somewhere in there?

6

I would look in /var/log/letsencrypt/letsencrypt.log, following a failed renewal.

For example, on my system there is this line:

2020-11-18 21:52:08,819:INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials

You should have something similar. Otherwise you would have an Unable to locate credentials error.

7

Hmmm...nothing in the letsencrypt.log on credentials, but there is on the certbot.log which says :

Found credentials in environment variables.

In the letsencrypt.log it says:

/ # cat /var/log/letsencrypt/letsencrypt.log
2020-11-18 11:05:05,729:DEBUG:certbot.main:certbot version: 0.37.0
2020-11-18 11:05:05,730:DEBUG:certbot.main:Arguments: []
2020-11-18 11:05:05,730:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#certbot-route53:auth,PluginEntryPoint#dns-route53,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-11-18 11:05:05,790:DEBUG:certbot.log:Root logging level set at 20
2020-11-18 11:05:05,791:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-11-18 11:05:05,842:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7fc594f6cc10> and installer <certbot.cli._Default object at 0x7fc594f6cc10>
2020-11-18 11:05:05,893:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2020-11-17 02:09:09 UTC.
2020-11-18 11:05:05,893:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2020-11-18 11:05:05,893:INFO:certbot.renewal:Non-interactive renewal: random delay of 276 seconds
8

Edit: Hang on, this line was in your original post as well, and I didn't see it. Oops!

What actually runs the Docker certbot-dns-route53 container? Is there some kind of script which does it?

It seems the likely place that any AWS_ environment variables would be getting injected.

1 Like
9

Ahh there they are. Both the variables are outputted when I run that.

AWS_ACCESS_KEY_ID=[]
AWS_SECRET_ACCESS_KEY=[]

I'm running it directly. To note, I am inside the certbot docker container (where this all lives). Initially when I run certbot renew, it throws up an error stating:

Another instance of Certbot is already running.

to fix I have to run :

 find / -type f -name ".certbot.lock" -exec rm {} \;

Looks like there isn't any file called cli.ini inside the letsencrypt folder

10

Ah, so they are showing up when you run env inside the Docker container.

I guess the next step would be to figure out how they are getting into the container to begin with, so you have the ability to change them to fresh credentials.

11

the whole of the certbot installation seems to be inside the container itself (running certbot on the server root produces an error). The env vars were actually outputted from the sh inside the container itself.
I suppose I could try and generate a new API Token and update these, but it seems to be a different error (otherwise it would say that Auth failed right?).
Could it be that "AWS_DEFAULT_REGION" is a required variable in this mix? (assuming that is relating to ListHostedZones).

There must be more some config somewhere though as what is currently telling letencrypt what URLs to generate for you know?

12

Allllright.
I created a new key and added the default region variable (as it seemed like AWS wants it), updated the environment variables with the new ones, and presto, certificate renewal succeeded. Very odd and I would still love to know what is going on in here to be honest.

Do you know by chance where there may be any other config files hiding? This is not a wildcard cert as far as I can tell (given that it was challenging specified subdomains). I'd love to find where these challenges are being set.

Thanks a ton for all your help on this. Really very much appreciated.

1 Like
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.