Dns-google plugin failure to find managed domains

itsthejb · March 11, 2019, 10:44am

My domain is: jcrooke.net

I ran this command: certbot --dns-google --dns-google-credentials /etc/letsencrypt/credentials/credentials.json --server https://acme-v02.api.letsencrypt.org/directory renewal --dry-run

It produced this output:

Attempting refresh to obtain initial access_token Refreshing access_token Encountered 403 Forbidden with reason "forbidden" Cleaning up challenges URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dns/v1/rest URL being requested: GET https://www.googleapis.com/dns/v1/projects/web-server-208814/managedZones?alt=json&dnsName=jcrooke.net. Attempting refresh to obtain initial access_token Refreshing access_token Encountered 403 Forbidden with reason "forbidden" Error finding zone. Skipping cleanup. Attempting to renew cert (jcrooke.net) from /etc/letsencrypt/renewal/jcrooke.net.conf produced an unexpected error: Encountered error finding managed zone: <HttpError 403 when requesting https://www.googleapis.com/dns/v1/projects/web-server-208814/managedZones?alt=json&dnsName=jcrooke.net. returned "Forbidden">. Skipping.

My web server is (include version): n/a

The operating system my web server runs on is (include version): CentOS7, Certbot docker image (https://hub.docker.com/r/certbot/certbot/)

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 0.31.0

Having the same issue as Dns-google plugin failure to find managed domains. Previously this was working, but I regenerated my service account and now I get the 403. All steps followed as https://certbot-dns-google.readthedocs.io/en/stable/, but no luck. I’ve triple checked that the credentials file is in the right location, and is the same as downloaded after creating a service account key. All seems fine

Might DNSSec have something to do with this? I did enable/disable/enable it. Perhaps this has messed everything up?

Hope someone can help

JuergenAuer · March 11, 2019, 10:57am

Hi @itsthejb

your DNSSEC is broken ( https://check-your-website.server-daten.de/?q=jcrooke.net ):

1 DS RR in the parent zone found

1 RRSIG RR to validate DS RR found

Algorithm: 8, 2 Labels, original TTL: 86400 sec, Signature-expiration: 18.03.2019, 05:45:51, Signature-Inception: 11.03.2019, 04:35:51, KeyTag 51638, Signer-Name: net

• Status: Good - Algorithmus 8 and DNSKEY with KeyTag 51638 used to validate the DS RRSet in the parent zone

2 DNSKEY RR found

Public Key with Algorithm 8, KeyTag 21407, Flags 257 (SEP = Secure Entry Point)

Public Key with Algorithm 8, KeyTag 61984, Flags 256

1 RRSIG RR to validate DNSKEY RR found

• Algorithm: 8, 2 Labels, original TTL: 300 sec, Signature-expiration: 01.04.2019, 10:26:27, Signature-Inception: 10.03.2019, 10:26:27, KeyTag 21407, Signer-Name: jcrooke.net

• Status: Good - Algorithmus 8 and DNSKEY with KeyTag 21407 used to validate the DNSKEY RRSet

Fatal error: DNSKEY 21407 signs DNSKEY RRset, but no confirming DS RR in the parent zone found. No chain of trust created.

Fatal error: Parent zone has a signed DS RR (Algorithm 8, KeyTag 22811, DigestType 2, Digest 53j1OEiu1olm+kDYV3n7Vk1a+h9tmCBwxhus6eudJaQ=), but the destination DNSKEY doesn’t exist or doesn’t validate the DNSKEY RR set. No chain of trust created.

There is a DS record in the parent zone, but not a DNSKEY in your zone that matches this DS.

Rechecked with https://dnssec-analyzer.verisignlabs.com/jcrooke.net to see if my tool works, there is the same result:

None of the 2 DNSKEY records could be validated by any of the 1 DS records

So

fix your DNSSEC so it works (you need a DNSKEY which has a DS in the parent zone) or
remove the DS entry in the parent zone.

DNSSEC is broken -> Google may not change your dns entries.

itsthejb · March 11, 2019, 1:16pm

Hi,

Thanks very much for the quick and courteous response! This does indeed make sense - I cleaned out my records recently and have obviously broken everything. Verifying everything now, but makes sense this is the cause.

I’ll close this once I’ve got it working again

itsthejb · March 12, 2019, 10:15am

Hi again,

Well I’ve fixed my DNSSEC (https://dnssec-analyzer.verisignlabs.com/jcrooke.net), but I’m still getting the same 403. I’ve waited some time to make sure the changes propagate, but it looks like there’s some other problem…

I hope someone can help?

JuergenAuer · March 12, 2019, 10:40am

Yep, your DNSSEC is now fixed.

There is another thread

The DNS authenticator only works with Google Cloud DNS. Google Domains doesn’t offer a API that Let’s Encrypt can use.

but if it had worked, that can't be the problem.

Do you have a different error message if you use wrong written credentials?

If not, your credentials may be wrong, perhaps the config file is in the wrong format (utf-16 etc.).

Is your zone name correct?

itsthejb · March 12, 2019, 10:49am

Hi Jürgen,

Thanks again for helping.

As for the credentials, I downloaded and SCP’ed the file, so I’m fairly sure this isn’t the problem. However, I did do some things that could’ve broken it:

Set up Certbot via docker with a previous domain. Everything was working
Created a new domain (jcrooke.net), and migrated over. Old domain no longer has any configuration
Everything seemed to be working correctly for the new domain, and it was trying to recreate it. I had some issue with timeout on the DNS update, so I thought I should clean out the service accounts and recreate things just-in-case
Now getting the 403 errors.

My zone name has changed, but I don’t see anywhere in the configuration to specify this. It is at least looking for the right DNS name (https://www.googleapis.com/dns/v1/projects/web-server-208814/managedZones?alt=json&dnsName=jcrooke.net)…

Otherwise config seems ok:

/etc/letsencrypt/renewal/jcrooke.net.conf

# renew_before_expiry = 30 days
version = 0.27.1
archive_dir = /etc/letsencrypt/archive/jcrooke.net
cert = /etc/letsencrypt/live/jcrooke.net/cert.pem
privkey = /etc/letsencrypt/live/jcrooke.net/privkey.pem
chain = /etc/letsencrypt/live/jcrooke.net/chain.pem
fullchain = /etc/letsencrypt/live/jcrooke.net/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = dns-google
account = dc88108ef4a4a469f962c6e6820cc757
dns_google_credentials = /etc/letsencrypt/credentials/credentials.json
server = https://acme-v02.api.letsencrypt.org/directory
post_hook = touch /etc/letsencrypt/renewed

/etc/letsencrypt/credentials/credentials.json is the credentials file downloaded for the service account. These files are of course mounted into the docker container.

Full log:

2019-03-12 10:14:07,179:DEBUG:certbot.main:certbot version: 0.31.0
2019-03-12 10:14:07,179:DEBUG:certbot.main:Arguments: ['--dns-google', '--dns-google-credentials', '/etc/letsencrypt/credentials/credentials.json', '--server', 'https://acme-v02.api.letsencrypt.org/directory', '--dry-run']
2019-03-12 10:14:07,179:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-google,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-03-12 10:14:07,197:DEBUG:certbot.log:Root logging level set at 20
2019-03-12 10:14:07,197:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-03-12 10:14:07,227:DEBUG:certbot.plugins.selection:Requested authenticator dns-google and installer <certbot.cli._Default object at 0x7f4778d45a50>
2019-03-12 10:14:07,227:DEBUG:certbot.cli:Var server=https://acme-v02.api.letsencrypt.org/directory (set by user).
2019-03-12 10:14:07,227:DEBUG:certbot.cli:Var authenticator=dns-google (set by user).
2019-03-12 10:14:07,254:INFO:certbot.renewal:Cert not due for renewal, but simulating renewal for dry run
2019-03-12 10:14:07,254:DEBUG:certbot.plugins.selection:Requested authenticator dns-google and installer None
2019-03-12 10:14:07,260:DEBUG:certbot.plugins.selection:Single candidate plugin: * dns-google
Description: Obtain certificates using a DNS TXT record (if you are using Google Cloud DNS for DNS).
Interfaces: IAuthenticator, IPlugin
Entry point: dns-google = certbot_dns_google.dns_google:Authenticator
Initialized: <certbot_dns_google.dns_google.Authenticator object at 0x7f4778d48810>
Prep: True
2019-03-12 10:14:07,261:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_dns_google.dns_google.Authenticator object at 0x7f4778d48810> and installer None
2019-03-12 10:14:07,261:INFO:certbot.plugins.selection:Plugins selected: Authenticator dns-google, Installer None
2019-03-12 10:14:07,263:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=None, only_return_existing=None, contact=(), key=None, external_account_binding=None), uri=u'https://acme-staging-v02.api.letsencrypt.org/acme/acct/7251889', new_authzr_uri=None, terms_of_service=None), 6f625e748d8a79646a9eb3d97c8d9541, Meta(creation_host=u'48cac13b0bf5', creation_dt=datetime.datetime(2018, 10, 31, 17, 21, 59, tzinfo=<UTC>)))>
2019-03-12 10:14:07,264:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
2019-03-12 10:14:07,265:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
2019-03-12 10:14:07,589:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 724
2019-03-12 10:14:07,589:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 724
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 12 Mar 2019 10:14:07 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 12 Mar 2019 10:14:07 GMT
Connection: keep-alive

{
  "dh-QUKoD0ZM": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}
2019-03-12 10:14:07,590:INFO:certbot.main:Renewing an existing certificate
2019-03-12 10:14:07,776:DEBUG:acme.client:Requesting fresh nonce
2019-03-12 10:14:07,778:DEBUG:acme.client:Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
2019-03-12 10:14:07,977:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2019-03-12 10:14:07,977:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Link: <https://acme-staging-v02.api.letsencrypt.org/index>;rel="index"
Replay-Nonce: gqJw56FZJeUrmHo0Zh0kRa4P-Qjz1w7-f5VKB3pYaKA
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Content-Length: 0
Expires: Tue, 12 Mar 2019 10:14:07 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 12 Mar 2019 10:14:07 GMT
Connection: keep-alive


2019-03-12 10:14:07,978:DEBUG:acme.client:Storing nonce: gqJw56FZJeUrmHo0Zh0kRa4P-Qjz1w7-f5VKB3pYaKA
2019-03-12 10:14:07,978:DEBUG:acme.client:JWS payload:
{
  "identifiers": [
    {
      "type": "dns",
      "value": "*.jcrooke.net"
    }
  ]
}
2019-03-12 10:14:07,979:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJub25jZSI6ICJncUp3NTZGWkplVXJtSG8wWmgwa1JhNFAtUWp6MXc3LWY1VktCM3BZYUtBIiwgInVybCI6ICJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1vcmRlciIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzcyNTE4ODkiLCAiYWxnIjogIlJTMjU2In0",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwgCiAgICAgICJ2YWx1ZSI6ICIqLmpjcm9va2UubmV0IgogICAgfQogIF0KfQ",
  "signature": "t-ZoJOxkIvaSqIcPLm9-b26fHmc1ZZ9apPbuS314GpWkDFlISl2KAIKCZp8Ha782Y1rG4kfnqFl1Q-WtXSxhXtaleRqrGkc8hqILRySVw9Bd_UYDd3DL1TWN19EHj3qLSYg__5eWebghYc98ptvo4jNDaFs0aEff5wwMZB7yv07Sm4jIKSzu0VBQhqqvoOgXP9NjDHABUCzfbshBx-6uiK2Lzl4hAQyp-c5U2TJffzl1XONvrvxWv2qhEWPCZbWsszNTSk-OJY5LGMFLOfkkICRBJu-q1wXVupicuYKnQXcuMaskN9hnCbTbRgaelcRyVfU7PxUsV2daMfHBfejhEA"
}
2019-03-12 10:14:08,211:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 376
2019-03-12 10:14:08,212:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 376
Boulder-Requester: 7251889
Link: <https://acme-staging-v02.api.letsencrypt.org/index>;rel="index"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/order/7251889/26383659
Replay-Nonce: OgJDs4iYIFhDt3TpIN-FHEBe1rPQL3H0pFVtqEiITNs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 12 Mar 2019 10:14:08 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 12 Mar 2019 10:14:08 GMT
Connection: keep-alive

{
  "status": "pending",
  "expires": "2019-03-17T22:06:05Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "*.jcrooke.net"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz/DmQgDt0VU_1Hrda_oqCXxqGyENwqnAKgS08aMUfDNzY"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/7251889/26383659"
}
2019-03-12 10:14:08,212:DEBUG:acme.client:Storing nonce: OgJDs4iYIFhDt3TpIN-FHEBe1rPQL3H0pFVtqEiITNs
2019-03-12 10:14:08,212:DEBUG:acme.client:JWS payload:

2019-03-12 10:14:08,213:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz/DmQgDt0VU_1Hrda_oqCXxqGyENwqnAKgS08aMUfDNzY:
{
  "protected": "eyJub25jZSI6ICJPZ0pEczRpWUlGaER0M1RwSU4tRkhFQmUxclBRTDNIMHBGVnRxRWlJVE5zIiwgInVybCI6ICJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2F1dGh6L0RtUWdEdDBWVV8xSHJkYV9vcUNYeHFHeUVOd3FuQUtnUzA4YU1VZkROelkiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC83MjUxODg5IiwgImFsZyI6ICJSUzI1NiJ9",
  "payload": "",
  "signature": "CdNiBCP_bYFEnUPIr1Y4fOXou4xwEakocrpOToetkf2x6_x2aMv8_nuf3FNkKPerfz5clYyk2o03g7RaZ42OL6wkroA0dyn0rhW4h49m6H51Otr-VpqQwNTWxnyZXbg_YlfcNRu5lpyp3ojfVUpf_Lii2Xdmxe2GujIcRjH2Vxg4CY0_vGOcIbFvcyGix2aJAs6wtNWaWQKwzUZrGlp4Hn-8G_Ns6K-6pV_5WU4yXouSeZnglC5htwxfFCBGL6iyrlj1SeVsIT8gTaLuEt_wDT63yVsrEpcEiotE5t83YEu4dBzxDtA1FUbEthiJWI5ULAN8qAZ5xrpXmNgjBw_RBA"
}
2019-03-12 10:14:08,420:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz/DmQgDt0VU_1Hrda_oqCXxqGyENwqnAKgS08aMUfDNzY HTTP/1.1" 200 428
2019-03-12 10:14:08,421:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 428
Boulder-Requester: 7251889
Link: <https://acme-staging-v02.api.letsencrypt.org/index>;rel="index"
Replay-Nonce: zj7datbeVmSrvUThE1MGJArKvFd3PBhz-NP_JsrrzHc
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 12 Mar 2019 10:14:08 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 12 Mar 2019 10:14:08 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "jcrooke.net"
  },
  "status": "pending",
  "expires": "2019-03-17T22:06:05Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/DmQgDt0VU_1Hrda_oqCXxqGyENwqnAKgS08aMUfDNzY/266575362",
      "token": "tMwFY1e7Y6BaalWNVfnZdHRN_gnjlbQFyUu4ECtruTY"
    }
  ],
  "wildcard": true
}
2019-03-12 10:14:08,421:DEBUG:acme.client:Storing nonce: zj7datbeVmSrvUThE1MGJArKvFd3PBhz-NP_JsrrzHc
2019-03-12 10:14:08,421:INFO:certbot.auth_handler:Performing the following challenges:
2019-03-12 10:14:08,422:INFO:certbot.auth_handler:dns-01 challenge for jcrooke.net
2019-03-12 10:14:08,424:INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dns/v1/rest
2019-03-12 10:14:08,627:INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/dns/v1/projects/web-server-208814/managedZones?alt=json&dnsName=jcrooke.net.
2019-03-12 10:14:08,627:INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
2019-03-12 10:14:08,629:DEBUG:oauth2client.crypt:['eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjRlYjFjNzk5NDE3NTI2NmJhYWZjYzZjMzhkZWUwZTk3OWRlMmYxOTgifQ', 'eyJpc3MiOiJjZXJ0Ym90QHdlYi1zZXJ2ZXItMjA4ODE0LmlhbS5nc2VydmljZWFjY291bnQuY29tIiwic2NvcGUiOiJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9hdXRoL25kZXYuY2xvdWRkbnMucmVhZHdyaXRlIiwiYXVkIjoiaHR0cHM6Ly9vYXV0aDIuZ29vZ2xlYXBpcy5jb20vdG9rZW4iLCJleHAiOjE1NTIzODkyNDgsImlhdCI6MTU1MjM4NTY0OH0', 'hMyT4bu6-sQT3k0BTmuaCp-fZQ4RKZ2JZkkhULwacSENp42GG65hSXBwBMFZmVNnfszPG72lVQ46OD4fk9SytfLbX8DmT80L3djrpOReLK7v3dW6F0gO5UKoW5xpSgWPDKtg7suh6dTirk1dvBCyMakOeRM0yxyFpdbIRaOxV0__PEhx7CsfTxqBvLmTXfWrW19siXrqtg_au_n-hnW5Mn40EP7L-5_j7M35qg1lTr2KenRqdtGpD2TuAj4xl9BDzEDrpxsigFZ3uQaqCb7fMyg3zryhKgakakF_irm32kcoDm763r50cGP1Ysci8IiBqasswqIU1jv-TevU44ySOg']
2019-03-12 10:14:08,629:INFO:oauth2client.client:Refreshing access_token
2019-03-12 10:14:09,222:WARNING:googleapiclient.http:Encountered 403 Forbidden with reason "forbidden"
2019-03-12 10:14:09,223:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/opt/certbot/src/certbot/auth_handler.py", line 75, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/opt/certbot/src/certbot/auth_handler.py", line 139, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/opt/certbot/src/certbot/plugins/dns_common.py", line 57, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/opt/certbot/src/certbot-dns-google/certbot_dns_google/dns_google.py", line 70, in _perform
    self._get_google_client().add_txt_record(domain, validation_name, validation, self.ttl)
  File "/opt/certbot/src/certbot-dns-google/certbot_dns_google/dns_google.py", line 113, in add_txt_record
    zone_id = self._find_managed_zone_id(domain)
  File "/opt/certbot/src/certbot-dns-google/certbot_dns_google/dns_google.py", line 275, in _find_managed_zone_id
    .format(e))
PluginError: Encountered error finding managed zone: <HttpError 403 when requesting https://www.googleapis.com/dns/v1/projects/web-server-208814/managedZones?alt=json&dnsName=jcrooke.net. returned "Forbidden">

2019-03-12 10:14:09,223:DEBUG:certbot.error_handler:Calling registered functions
2019-03-12 10:14:09,223:INFO:certbot.auth_handler:Cleaning up challenges
2019-03-12 10:14:09,225:INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dns/v1/rest
2019-03-12 10:14:09,414:INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/dns/v1/projects/web-server-208814/managedZones?alt=json&dnsName=jcrooke.net.
2019-03-12 10:14:09,414:INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
2019-03-12 10:14:09,416:DEBUG:oauth2client.crypt:['eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjRlYjFjNzk5NDE3NTI2NmJhYWZjYzZjMzhkZWUwZTk3OWRlMmYxOTgifQ', 'eyJpc3MiOiJjZXJ0Ym90QHdlYi1zZXJ2ZXItMjA4ODE0LmlhbS5nc2VydmljZWFjY291bnQuY29tIiwic2NvcGUiOiJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9hdXRoL25kZXYuY2xvdWRkbnMucmVhZHdyaXRlIiwiYXVkIjoiaHR0cHM6Ly9vYXV0aDIuZ29vZ2xlYXBpcy5jb20vdG9rZW4iLCJleHAiOjE1NTIzODkyNDksImlhdCI6MTU1MjM4NTY0OX0', 'Nw12THNUZXnIeLLwKIxZYAmQVCwxRsfSN_d-alPYeHs-CBYnvN0aecCbnIbZChYH6rpr0t_JeNae0Eugn0I4HIcy0-dHWO-eSxnc-BdxTq_jMprCp51sL34PhW3mEboK0kAUEUMJk9mG4r6hJz9BUp-Pb9K6vHots7yAMvayvb4VERUVBNuY4LIZHZF-JfgftESzoHZrXt_JR9U_UQhxBEIH0wtVR_6mfMq6LpzZGmg65aDocQauOHhhZGsGZ22-ldLG7ZJdoHwc7t9KkoS7p9_faPwOENfpwMj7Bsbr2Qy44JCcLv8cABNUIp1rvpLgzzvUqy0-STfOlHfjbw19Xw']
2019-03-12 10:14:09,416:INFO:oauth2client.client:Refreshing access_token
2019-03-12 10:14:10,537:WARNING:googleapiclient.http:Encountered 403 Forbidden with reason "forbidden"
2019-03-12 10:14:10,537:WARNING:certbot_dns_google.dns_google:Error finding zone. Skipping cleanup.
2019-03-12 10:14:10,537:WARNING:certbot.renewal:Attempting to renew cert (jcrooke.net) from /etc/letsencrypt/renewal/jcrooke.net.conf produced an unexpected error: Encountered error finding managed zone: <HttpError 403 when requesting https://www.googleapis.com/dns/v1/projects/web-server-208814/managedZones?alt=json&dnsName=jcrooke.net. returned "Forbidden">. Skipping.
2019-03-12 10:14:10,538:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/opt/certbot/src/certbot/renewal.py", line 452, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/opt/certbot/src/certbot/main.py", line 1193, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/opt/certbot/src/certbot/main.py", line 116, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/opt/certbot/src/certbot/renewal.py", line 310, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/opt/certbot/src/certbot/client.py", line 353, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/opt/certbot/src/certbot/client.py", line 389, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/opt/certbot/src/certbot/auth_handler.py", line 75, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/opt/certbot/src/certbot/auth_handler.py", line 139, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/opt/certbot/src/certbot/plugins/dns_common.py", line 57, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/opt/certbot/src/certbot-dns-google/certbot_dns_google/dns_google.py", line 70, in _perform
    self._get_google_client().add_txt_record(domain, validation_name, validation, self.ttl)
  File "/opt/certbot/src/certbot-dns-google/certbot_dns_google/dns_google.py", line 113, in add_txt_record
    zone_id = self._find_managed_zone_id(domain)
  File "/opt/certbot/src/certbot-dns-google/certbot_dns_google/dns_google.py", line 275, in _find_managed_zone_id
    .format(e))
PluginError: Encountered error finding managed zone: <HttpError 403 when requesting https://www.googleapis.com/dns/v1/projects/web-server-208814/managedZones?alt=json&dnsName=jcrooke.net. returned "Forbidden">

2019-03-12 10:14:10,538:ERROR:certbot.renewal:All renewal attempts failed. The following certs could not be renewed:
2019-03-12 10:14:10,538:ERROR:certbot.renewal:  /etc/letsencrypt/live/jcrooke.net/fullchain.pem (failure)
2019-03-12 10:14:10,540:INFO:certbot.hooks:Running post-hook command: /etc/letsencrypt/renewal-hooks/post/docker.sh
2019-03-12 10:14:18,213:INFO:certbot.hooks:Output from docker.sh:
ispconfig

2019-03-12 10:14:18,213:INFO:certbot.hooks:Running post-hook command: touch /etc/letsencrypt/renewed
2019-03-12 10:14:18,216:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 11, in <module>
    load_entry_point('certbot', 'console_scripts', 'certbot')()
  File "/opt/certbot/src/certbot/main.py", line 1365, in main
    return config.func(config, plugins)
  File "/opt/certbot/src/certbot/main.py", line 1272, in renew
    renewal.handle_renewal_request(config)
  File "/opt/certbot/src/certbot/renewal.py", line 477, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
Error: 1 renew failure(s), 0 parse failure(s)

JuergenAuer · March 12, 2019, 10:56am

It looks like a google problem, not a problem of your certbot:

2019-03-12 10:14:08,627:INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
2019-03-12 10:14:08,629:DEBUG:oauth2client.crypt:['eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjRlYjFjNzk5NDE3NTI2NmJhYWZjYzZjMzhkZWUwZTk3OWRlMmYxOTgifQ', 'eyJpc3MiOiJjZXJ0Ym90QHdlYi1zZXJ2ZXItMjA4ODE0LmlhbS5nc2VydmljZWFjY291bnQuY29tIiwic2NvcGUiOiJodHRwczovL3d3dy5nb29nbGVhcGlzLmNvbS9hdXRoL25kZXYuY2xvdWRkbnMucmVhZHdyaXRlIiwiYXVkIjoiaHR0cHM6Ly9vYXV0aDIuZ29vZ2xlYXBpcy5jb20vdG9rZW4iLCJleHAiOjE1NTIzODkyNDgsImlhdCI6MTU1MjM4NTY0OH0', 'hMyT4bu6-sQT3k0BTmuaCp-fZQ4RKZ2JZkkhULwacSENp42GG65hSXBwBMFZmVNnfszPG72lVQ46OD4fk9SytfLbX8DmT80L3djrpOReLK7v3dW6F0gO5UKoW5xpSgWPDKtg7suh6dTirk1dvBCyMakOeRM0yxyFpdbIRaOxV0__PEhx7CsfTxqBvLmTXfWrW19siXrqtg_au_n-hnW5Mn40EP7L-5_j7M35qg1lTr2KenRqdtGpD2TuAj4xl9BDzEDrpxsigFZ3uQaqCb7fMyg3zryhKgakakF_irm32kcoDm763r50cGP1Ysci8IiBqasswqIU1jv-TevU44ySOg']
2019-03-12 10:14:08,629:INFO:oauth2client.client:Refreshing access_token
2019-03-12 10:14:09,222:WARNING:googleapiclient.http:Encountered 403 Forbidden with reason "forbidden"

And now it looks like another error, not the problem “doesn’t find the managed zone”.

Now your access token is wrong.

itsthejb · March 12, 2019, 10:58am

Ah I see; so it’s possible this is a temporary issue on Google’s side? It would, however, be a strange coincidence that this started happening right when I regenerated my service account… Still possible, however.

I could at least try reporting it to Google DNS support…?

itsthejb · March 22, 2019, 10:37am

Hi all,

I’ve still got no further into fixing this:

I tried disabling DNSSec entirely, no change
I tried giving the service account full “DNS administrator” rights. No change
Verifying that the credentials are actually being read by playing with the path name. No change
Created a support ticket with Google in case it really is an issue with their API. No response from them yet. However, surely then many many people would be reporting issues? This is also unlikely since before I recreated the service account, everything was working
I tried recreating the service account yet more times; still no change. This is following the documentation to the letter, and I previously created the account the first time with no issue

I can’t believe that the plugin is entirely broken? It seems just like a simple credentials problem, but nothing I do seems to make any difference.

This is not urgent yet because my cert is not yet up for renewal, but I’m not close to find the problem yet…

Please I hope someone can help

JuergenAuer · March 22, 2019, 11:05am

Not directly a solution, but is it possible that you create the dns entry manual?

Using

--manual

instead of the plugin.

itsthejb · March 22, 2019, 12:27pm

Hi,

Thanks for helping out again! Currently trying to do it manually, but getting stuck on the .well-known/acme-challenge file challenge. This is something about my web server config; I can browse to the file, but it won’t show it and Certbot won’t be able to access it. Is there anyway to avoid this challenge?

In any case, I don’t suppose this is likely to prove much, since I can easily create the DNS record manually in the Cloud DNS Console. The problem is why the plugin isn’t authorized to do anything, in spite of it using the service account credentials

itsthejb · March 22, 2019, 2:02pm

Progress!

I created another service account with “All Rights”, to test if permissions were the issue. It works! Therefore, it would seem that something regarding permissions for Cloud DNS has changed, and the docs are out of date. Who should I approach about this?

JuergenAuer · March 22, 2019, 2:46pm

That happens always

Happy to read that it works now.

itsthejb · March 22, 2019, 3:19pm

Opened issue at https://github.com/certbot/certbot/issues/6877

obohovyk · March 28, 2019, 12:36pm

@itsthejb
I got the same error and couldn’t find appropriate solution, but in my case problem was in GSuite option Selection_008

P.S. DNSSEC activated for that zone

itsthejb · March 28, 2019, 2:56pm

Greyed out in my case:

Which I assume is because not using G Suite. Hopefully that might help someone else, though!

It’s working just fine with excessive credentials for the time being

system · April 27, 2019, 2:57pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.