Dns-google certbot renewal is giving me 403 error

Help
1

I created my certificates using dns-google for a compute engine VM instance.

Here's the documentation I followed: DNS-Google

The certificates were created perfectly, they all work.

The problem lies in the renewal.

I've been doing:

sudo certbot renew --dry-run

I keep getting errors:

sudo certbot renew --dry-run

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/DOMAINs.conf

Cert not due for renewal, but simulating renewal for dry run

Plugins selected: Authenticator dns-google, Installer None

Renewing an existing certificate

Performing the following challenges:

dns-01 challenge for DOMAIN.co

dns-01 challenge for DOMAIN.com

dns-01 challenge for DOMAIN.co

dns-01 challenge for DOMAIN.com

URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dns/v1/rest

URL being requested: GET https://dns.googleapis.com/dns/v1/projects/DOMAINs/managedZones?alt=json&dnsName=DOMAIN.co.

Encountered 403 Forbidden with reason "forbidden"

Cleaning up challenges

URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dns/v1/rest

URL being requested: GET https://dns.googleapis.com/dns/v1/projects/DOMAINs/managedZones?alt=json&dnsName=DOMAIN.co.

Encountered 403 Forbidden with reason "forbidden"

Error finding zone. Skipping cleanup.

URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dns/v1/rest

URL being requested: GET https://dns.googleapis.com/dns/v1/projects/DOMAINs/managedZones?alt=json&dnsName=DOMAIN.com.

Encountered 403 Forbidden with reason "forbidden"

Error finding zone. Skipping cleanup.

URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dns/v1/rest

URL being requested: GET https://dns.googleapis.com/dns/v1/projects/DOMAINs/managedZones?alt=json&dnsName=DOMAIN.co.

Encountered 403 Forbidden with reason "forbidden"

Error finding zone. Skipping cleanup.

URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dns/v1/rest

URL being requested: GET https://dns.googleapis.com/dns/v1/projects/DOMAINs/managedZones?alt=json&dnsName=DOMAIN.com.

Encountered 403 Forbidden with reason "forbidden"

Error finding zone. Skipping cleanup.

Attempting to renew cert (DOMAINs) from /etc/letsencrypt/renewal/DOMAINs.conf produced an unexpected error: Encountered error finding managed zone: <HttpError 403 when requesting https://dns.googleapis.com/dns/v1/projects/investize

ns/managedZones?alt=json&dnsName=DOMAIN.co. returned "Request had insufficient authentication scopes.">. Skipping.

All renewal attempts failed. The following certs could not be renewed:

/etc/letsencrypt/live/DOMAINs/fullchain.pem (failure)

** DRY RUN: simulating 'certbot renew' close to cert expiry

** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:

/etc/letsencrypt/live/DOMAINs/fullchain.pem (failure)

** DRY RUN: simulating 'certbot renew' close to cert expiry

** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)

Is there something I need to add to my /etc/letsencrypt/renewal/DOMAINs.conf?

I also gave DNS administrator permissions to the service account. What else can I do?

2

Can you reach the requested URLs?
try:
curl https://www.googleapis.com/discovery/v1/apis/dns/v1/rest

3

Yes I can definitely reach those urls.

For the second url, this is the result I get:

{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"errors": [
{
"message": "Login Required.",
"domain": "global",
"reason": "required",
"location": "Authorization",
"locationType": "header"
}
],
"status": "UNAUTHENTICATED"
}
}

I am assuming there's either a permission error or certbot is not sending right info?

4

So we have established that you "root user" can reach those URLs.

  • DNS: Check.
  • Outbound firewall rules: Check.
  • Routing: Check.
  • Curl program works: Check

Now we need to better understand the "problem"... why certbot fails.
Can we review the recent LE logs (unfiltered)?
tail -n 99 /var/log/letsencrypt/letsencrypt.log
[replace any names with REPLACED if you like]

5

I found a solution for those using Google Compute Engine (GCE) with Google DNS. I installed GCE using pip and not the native console APIs&Services tool. I had to give permissions to the APIs to be able to run.

For anyone with a similar issue, go to your Compute Engine instance page. At the bottom, you will find Cloud API access scopes. I set it to Allow full access to all Cloud APIs. It’s at the bottom of the instance page. You need to press edit the instance.

For those looking to install Google DNS, do it through APIs&Services tool in the menu instead of through pip.

Here’s the full instructions:

  1. Install Google DNS API through APIs&Services in the left hand menu on Google Cloud Console.
  2. Make sure that Cloud API access scopes is set to Allow full access to all Cloud APIs on the instance’s page. You can also go onto the Google DNS API after you install it and click on the right hand side and click “Create Credentials”. It will most likely say that you do not need them if you are running a GCE.
  3. Run sudo certbot certonly --cert-name CERTNAME --dns-google -d 'DOMAIN.COM,*.DOMAIN.COM' and replace or add DOMAIN and add name to your certificate using --cert-name CERTNAME if you want to do so. Else remove --cert-name CERTNAME if you want a default certificate name. You will see “congratulations” if successful.
  4. Wait a few mins and run sudo certbot renew --dry-run to check if everything with permissions is good to go. This might take a few min. If you get a txt records error, remove the txt records manually from your domains DNS records. Give it a few mins, and retry sudo certbot renew --dry-run. After a successful dry run, you’ll notice new txt records created in your DNS records, do not delete those. You will see “congratulations” if successful.
  5. If you want to make sure you have renewed before expiry, add 0 0 * * * certbot renew to your crontab. It will attempt to renew daily and succeed about 30 days before expiry to keep your certificates valid.
6

Use something like this to run twice a day at random times:

0 */12 * * * perl -e 'sleep int(rand(43200))' && certbot -q renew

Some OSes and some Certbot installation methods will have already installed an appropriate systemd timer or cron job, though.

1 Like
7

Is there a reason for running twice a day? This is what confuses me a bit. If I am allowed to renew 30 days before, realistically, I only need to run cron renew about once every 2-3 weeks. @rg305 & @mnordhoff thanks for your help overall.

8

What happens if that (small window of opportunity) is missed?
Then 2-3 weeks later from that may be too late.
The check is minimal and basically inconsequential (>98% of the time).
[I would not go any lower than twice a week.]

1 Like
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.