Certbot not working with route53 (complaining about policy that it should have access to)

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: foundryvtt.arandmoor-gaming.com

I ran this command: sudo certbot certonly --dns-route53 -d *.arandmoor-gaming.com -i nginx --debug -v --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-route53, Installer nginx
Simulating a certificate request for *.arandmoor-gaming.com
Performing the following challenges:
dns-01 challenge for arandmoor-gaming.com
Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
File "/snap/certbot-dns-route53/current/lib/python3.8/site-packages/certbot_dns_route53/_internal/dns_route53.py", line 56, in perform
change_ids = [
File "/snap/certbot-dns-route53/current/lib/python3.8/site-packages/certbot_dns_route53/_internal/dns_route53.py", line 57, in
self._change_txt_record("UPSERT",
File "/snap/certbot-dns-route53/current/lib/python3.8/site-packages/certbot_dns_route53/_internal/dns_route53.py", line 107, in _change_txt_record
zone_id = self._find_zone_id_for_domain(validation_domain_name)
File "/snap/certbot-dns-route53/current/lib/python3.8/site-packages/certbot_dns_route53/_internal/dns_route53.py", line 85, in _find_zone_id_for_domain
for page in paginator.paginate():
File "/snap/certbot-dns-route53/current/lib/python3.8/site-packages/botocore/paginate.py", line 255, in iter
response = self._make_request(current_kwargs)
File "/snap/certbot-dns-route53/current/lib/python3.8/site-packages/botocore/paginate.py", line 334, in _make_request
return self._method(**current_kwargs)
File "/snap/certbot-dns-route53/current/lib/python3.8/site-packages/botocore/client.py", line 395, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/snap/certbot-dns-route53/current/lib/python3.8/site-packages/botocore/client.py", line 725, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListHostedZones operation: User: arn:aws:sts::618536401898:assumed-role/EC2-robot/i-09ed7459e8efd857d is not authorized to perform: route53:ListHostedZones because no identity-based policy allows the route53:ListHostedZones action

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/snap/certbot/1952/bin/certbot", line 8, in
sys.exit(main())
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/main.py", line 19, in main
return internal_main.main(cli_args)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/main.py", line 1723, in main
return config.func(config, plugins)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/main.py", line 1582, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/main.py", line 141, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/client.py", line 513, in obtain_and_enroll_certificate
cert, chain, key, _ = self.obtain_certificate(domains)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/client.py", line 441, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/client.py", line 493, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 86, in handle_authorizations
resps = self.auth.perform(achalls)
File "/snap/certbot-dns-route53/current/lib/python3.8/site-packages/certbot_dns_route53/_internal/dns_route53.py", line 67, in perform
raise errors.PluginError("\n".join([str(e), INSTRUCTIONS]))
certbot.errors.PluginError: An error occurred (AccessDenied) when calling the ListHostedZones operation: User: arn:aws:sts::618536401898:assumed-role/EC2-robot/i-09ed7459e8efd857d is not authorized to perform: route53:ListHostedZones because no identity-based policy allows the route53:ListHostedZones action
To use certbot-dns-route53, configure credentials as described at Configuration — Boto3 Docs 1.21.42 documentation and add the necessary permissions for Route53 access.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): nginx version: nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 20.04.4 LTS

My hosting provider, if applicable, is: EC2 + Route53

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.26.0

Additional Information:

I've got the AWS CLIv2 set up on the server, so I can directly test both my credentials file and my permissions.

The output of the following command: aws route53 list-hosted-zones

{
"HostedZones": [
{
"Id": "/hostedzone/[stuff]",
"Name": "arandmoor-gaming.com.",
"CallerReference": "RISWorkflow-RD:[moar stuff]",
"Config": {
"Comment": "HostedZone created by Route53 Registrar",
"PrivateZone": false
},
"ResourceRecordSetCount": 3
}
]
}

...and in python:

Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import boto3
client = boto3.client('route53')
client.list_hosted_zones()
{'ResponseMetadata': {'RequestId': '[stuff]', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '[stuff]', 'content-type': 'text/xml', 'content-length': '553', 'date': 'Mon, 18 Apr 2022 05:05:49 GMT'}, 'RetryAttempts': 0}, 'HostedZones': [{'Id': '/hostedzone/[stuff]', 'Name': 'arandmoor-gaming.com.', 'CallerReference': '[stuff]', 'Config': {'Comment': 'HostedZone created by Route53 Registrar', 'PrivateZone': False}, 'ResourceRecordSetCount': 3}], 'IsTruncated': False, 'MaxItems': '100'}

My credentials and permissions are fine as far as I can tell and according to the error messages certbot is handing me. I mean, if I use the exact same credentials it seems to be complaining about through the cli or boto3 on my own, everything works.

What am I doing wrong?

What is that EC2-robot/i-09ed7459e8efd857d? It's not one of mine as far as I can tell. Is it trying to borrow my permissions to make the cert or something?

Also, I've tried explicitly setting credentials through a secondary file with the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, and again with the AWS_PROFILE env variable, and again with the AWS_SHARED_CREDENTIALS_FILE env variable. I also tried changing accounts to both a second dedicated user AND my admin user account.

Nothing is working.

1 Like

I figured it out.

In case anyone else runs into this error, which has a VERY un-useful error message, but not because of certbot...) I'll leave my solution here.

sudo ln -s ~/.aws /root/.aws

Sudo was the problem, but was also a requirement. So it was a massive problem.

See, when you sudo a command, you run the command in an elevated shell that does NOT necessarily carry over things like set environment variables, or your current user for things like default file locations.

When you sudo anything involving python paths and environment variables get defaulted, so even if you set them as part of the execution by prepending the command with "env" and a bunch of assignments, it won't work unless you put the env command after sudo. Which I did not do when I tried to explicitly set my credentials.
Likewise, since sudo executes as root and not as the current user, the .aws directory in ~ wasn't being checked by boto3 for default config and credential files. Hence my confusion when I double-checked my credentials with the aws cli2 and boto3 in the python3 interpreter.

By creating a sym-link to my user's .aws directory in /root, sudo stopped effectively murdering my configurations and credentials and then hiding the bodies.

5 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.