I'm facing challenges renewing SSL certificates for several domains managed through the Google DNS plugin. Here's the list of affected domains:
*.accept.spend.cloud & accept.spend.cloud
*.stage.spend.cloud & stage.spend.cloud
*.spend.cloud & spend.cloud
*.test.spend.cloud & test.spend.cloud
Setup Details:
The domains are configured using the Google DNS plugin.
I'm using a service account with the correct permissions.
Initial Setup:
In February, I successfully added the required DNS records and obtained certificates for these domains.
Running a dry-run renewal command at that time worked perfectly.
Current Issue:
Despite this setup, automatic renewals have now failed for all mentioned domains.
When attempting to renew manually, I receive the following errors:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/accept.spend.cloud.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator dns-google, Installer None
Renewing an existing certificate for *.accept.spend.cloud and accept.spend.cloud
Performing the following challenges:
dns-01 challenge for accept.spend.cloud
dns-01 challenge for accept.spend.cloud
Encountered 403 Forbidden with reason "forbidden"
Encountered error adding TXT record: <HttpError 403 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "Forbidden". Details: "[{'message': 'Forbidden', 'domain': 'global', 'reason': 'forbidden'}]">
Cleaning up challenges
Encountered error deleting TXT record: <HttpError 404 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "The 'entity.change.deletions[_acme-challenge.accept.spend.cloud.][TXT]' resource named '_acme-challenge.accept.spend.cloud. (TXT)' does not exist.". Details: "[{'message': "The 'entity.change.deletions[_acme-challenge.accept.spend.cloud.][TXT]' resource named '_acme-challenge.accept.spend.cloud. (TXT)' does not exist.", 'domain': 'global', 'reason': 'notFound'}]">
Failed to renew certificate accept.spend.cloud with error: Error communicating with the Google Cloud DNS API: <HttpError 403 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "Forbidden". Details: "[{'message': 'Forbidden', 'domain': 'global', 'reason': 'forbidden'}]">
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/certtest.accept.spend.cloud.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/spend.cloud.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator dns-google, Installer None
Renewing an existing certificate for *.spend.cloud and spend.cloud
Performing the following challenges:
dns-01 challenge for spend.cloud
dns-01 challenge for spend.cloud
Encountered 403 Forbidden with reason "forbidden"
Encountered error adding TXT record: <HttpError 403 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "Forbidden". Details: "[{'message': 'Forbidden', 'domain': 'global', 'reason': 'forbidden'}]">
Cleaning up challenges
Encountered error deleting TXT record: <HttpError 404 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "The 'entity.change.deletions[_acme-challenge.spend.cloud.][TXT]' resource named '_acme-challenge.spend.cloud. (TXT)' does not exist.". Details: "[{'message': "The 'entity.change.deletions[_acme-challenge.spend.cloud.][TXT]' resource named '_acme-challenge.spend.cloud. (TXT)' does not exist.", 'domain': 'global', 'reason': 'notFound'}]">
Failed to renew certificate spend.cloud with error: Error communicating with the Google Cloud DNS API: <HttpError 403 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "Forbidden". Details: "[{'message': 'Forbidden', 'domain': 'global', 'reason': 'forbidden'}]">
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/stage.spend.cloud.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator dns-google, Installer None
Renewing an existing certificate for *.stage.spend.cloud and stage.spend.cloud
Performing the following challenges:
dns-01 challenge for stage.spend.cloud
dns-01 challenge for stage.spend.cloud
Encountered 403 Forbidden with reason "forbidden"
Encountered error adding TXT record: <HttpError 403 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "Forbidden". Details: "[{'message': 'Forbidden', 'domain': 'global', 'reason': 'forbidden'}]">
Cleaning up challenges
Encountered error deleting TXT record: <HttpError 404 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "The 'entity.change.deletions[_acme-challenge.stage.spend.cloud.][TXT]' resource named '_acme-challenge.stage.spend.cloud. (TXT)' does not exist.". Details: "[{'message': "The 'entity.change.deletions[_acme-challenge.stage.spend.cloud.][TXT]' resource named '_acme-challenge.stage.spend.cloud. (TXT)' does not exist.", 'domain': 'global', 'reason': 'notFound'}]">
Failed to renew certificate stage.spend.cloud with error: Error communicating with the Google Cloud DNS API: <HttpError 403 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "Forbidden". Details: "[{'message': 'Forbidden', 'domain': 'global', 'reason': 'forbidden'}]">
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/test.spend.cloud.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator dns-google, Installer None
Renewing an existing certificate for *.test.spend.cloud and test.spend.cloud
Performing the following challenges:
dns-01 challenge for test.spend.cloud
dns-01 challenge for test.spend.cloud
Encountered 403 Forbidden with reason "forbidden"
Encountered error adding TXT record: <HttpError 403 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "Forbidden". Details: "[{'message': 'Forbidden', 'domain': 'global', 'reason': 'forbidden'}]">
Cleaning up challenges
Encountered error deleting TXT record: <HttpError 404 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "The 'entity.change.deletions[_acme-challenge.test.spend.cloud.][TXT]' resource named '_acme-challenge.test.spend.cloud. (TXT)' does not exist.". Details: "[{'message': "The 'entity.change.deletions[_acme-challenge.test.spend.cloud.][TXT]' resource named '_acme-challenge.test.spend.cloud. (TXT)' does not exist.", 'domain': 'global', 'reason': 'notFound'}]">
Failed to renew certificate test.spend.cloud with error: Error communicating with the Google Cloud DNS API: <HttpError 403 when requesting https://dns.googleapis.com/dns/v1/projects/dev-pasc-vcdm/managedZones/219102773877062296/changes?alt=json returned "Forbidden". Details: "[{'message': 'Forbidden', 'domain': 'global', 'reason': 'forbidden'}]">
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following certificates are not due for renewal yet:
/etc/letsencrypt/live/certtest.accept.spend.cloud/fullchain.pem expires on 2024-08-06 (skipped)
All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/accept.spend.cloud/fullchain.pem (failure)
/etc/letsencrypt/live/spend.cloud/fullchain.pem (failure)
/etc/letsencrypt/live/stage.spend.cloud/fullchain.pem (failure)
/etc/letsencrypt/live/test.spend.cloud/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
4 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
Troubleshooting Steps:
I tested the service account permissions by obtaining a certificate for certtest.accept.spend.cloud using the same plugin and account. The certificate was issued successfully.
The dry-run renewal also worked seamlessly, adding a DNS record to the existing set.
Request for Help:
Given that the service account seems to function properly, I'm confused as to why renewal is failing for the existing certificates. Any insights or advice would be greatly appreciated.
Probably you executed the tests from the console, the renewal attempt is scheduled from crontab. Is there any configuration settings (especially Google DNS credential for the plug-in) from your profile that is accessible in the interactive session but not accessible in a session started from the crontab?
Hi @bruncsak , thanks and happy to be here to get help from awesome community. As far as I see, that's not the case. For a comparison, you can see the renewal configuration for certtest.accept.spend.cloud and accept.spend.cloud below.
# renew_before_expiry = 30 days
version = 2.10.0
archive_dir = /etc/letsencrypt/archive/certtest.accept.spend.cloud
cert = /etc/letsencrypt/live/certtest.accept.spend.cloud/cert.pem
privkey = /etc/letsencrypt/live/certtest.accept.spend.cloud/privkey.pem
chain = /etc/letsencrypt/live/certtest.accept.spend.cloud/chain.pem
fullchain = /etc/letsencrypt/live/certtest.accept.spend.cloud/fullchain.pem
# Options used in the renewal process
[renewalparams]
account = df1fe5b2b7cfa6976214c5710088517d
authenticator = dns-google
dns_google_credentials = /root/certbot-sa.json
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa
# renew_before_expiry = 30 days
version = 2.9.0
archive_dir = /etc/letsencrypt/archive/accept.spend.cloud
cert = /etc/letsencrypt/live/accept.spend.cloud/cert.pem
privkey = /etc/letsencrypt/live/accept.spend.cloud/privkey.pem
chain = /etc/letsencrypt/live/accept.spend.cloud/chain.pem
fullchain = /etc/letsencrypt/live/accept.spend.cloud/fullchain.pem
# Options used in the renewal process
[renewalparams]
account = df1fe5b2b7cfa6976214c5710088517d
authenticator = dns-google
dns_google_credentials = /root/certbot-sa.json
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa
renew_hook = /usr/bin/bash /root/certbot-post-hook/accept-spend-cloud.sh
Only differences you can see the version number and renewal_hook I added. When I check log files for the autorenewal messages, it was having same 403 error messages I posted in my main thread.
What @bruncsak described was more a different between your command prompt and the environ your auto-renew runs in. Does that run as a cronjob or a systemd timer? You didn't describe your O/S or Certbot version so I can't tell. You can review which option from these docs User Guide — Certbot 2.11.0.dev0 documentation
Sometimes the cronjob may not be using the same shell so gets a different PATH which then might see the wrong software version or config compared to something in the command line.
I found a very old but similar sounding problem below. Does any of the above or thread below sound plausible?
This is seemingly the important underlying error: Certbot doesn't have a correct API credential for your domain (or else the API has changed in some other way).
A Debian-based Compute Engine VM utilizes Certbot version 2.9.0 to obtain SSL certificates. The Certbot version has since been updated to 2.10.0. When I initially obtained the certificates, the renewal process was automatically set up using a systemd timer running as root, which matches the configuration in my console.
However, a recurring issue has emerged: when executing the certbot renew command either directly from the console or through the systemd timer's automatic renewal, I receive 403 errors for all certificates obtained in February. This indicates that the error isn't limited to one particular method, affecting both automated and manual renewals.
To investigate further, I requested a new certificate (certtest.accept.spend.cloud) on Wednesday using the same service account. It worked fine and didn't encounter any 403 errors, suggesting that the issue is specific to the Certbot API call made during the DNS record update for older certificates. Renewals for newly issued certificates don't encounter this problem.
As stated in my post, the system consistently returns a 403 error at present. I mention this to clarify that I tested the renewal process when initially setting up the certificates, expecting it to proceed smoothly.
_acme-challenge.certtest.spend.cloud canonical name = spend.cloud
_acme-challenge.stage.spend.cloud canonical name = stage.spend.cloud
_acme-challenge.accept.spend.cloud canonical name = accept.spend.cloud
@rg305 I was comparing the configuration for newly obtained (last Wednesday - the day I posted this thread) certificate and one of the certificate I obtained in February. certtest.accept.spend.cloud and accept.spend.cloud are in same DNS record set in our Google DNS manager.
For make it more clear, following are the list of certificates I'm having now:
When I run this command, certbot adds a DNS record automatically and waits for 60 seconds to verify and removes it when the verification is done. So I can get certificates for any CNAMEs as long as Certbot can add a DNS record to my record set.
To make it more clear when I run dry-run to a certificate that works fine, I get this: