At Heroku, domain considered "unsafe" by Let's Encrypt. Google says otherwise

Yesterday afternoon, I added "my.pdpworks.com" to our production application at Heroku utilizing their Automatic Certificate Management (Let's Encrypt) service. Normally, the Let's Encrypt certificates are applied within an hour. For "my.pdpworks.com", the status is stuck at "Domain verified". I received an email from Heroku indicating "Automated Certificate Management has failed for the following domains: my.pdpworks.com." The reason given is "Domain considered unsafe". When I check Google Transparency Report, the result is "Current status: No unsafe content found". Note that the Let's Encrypt certificate is applied to "**www.**my.pdpworks.com"

Heroku support states:

"The error domain is unsafe comes up when Let's Encrypt considers allocating a cert to this domain as unsafe as noted here.

As far as I know, there is not much we(Heroku) can do here. I realize that we built this feature on top of Let's Encrypt service, but it may be worth your time to inquire with them directly about any options that might be available for domains in that situation."

Please advise. What is Let's Encrypt considering unsafe? What corrective action is needed?

My domain is: my.pdpworks.com (no issue for www.my.pdpworks.com). The zone file is managed at AWS Route53.

I ran this command: Google Transparency Report

It produced this output: "Current status: No unsafe content found"

My web server is (include version): Heroku Common Runtime PaaS

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is: Heroku, Common Runtime

I can login to a root shell on my machine (yes or no, or I don't know): No

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): Heroku console

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

Hi @pdpglobalapp, and welcome to the LE community forum :slight_smile:

I'm not sure how Heroku implements certs, but this IP difference does seem rather interesting:

Name:      www.my.pdpworks.com.herokudns.com
Addresses: 54.204.238.15
           54.209.91.188
           54.221.251.148
           75.101.184.39
Aliases:   www.my.pdpworks.com

Name:    my.pdpworks.com
Address: 52.216.136.10
2 Likes

Thanks for looking. Maybe this is pertains:
Our app is running on Heroku Common Runtime where fixed IPs are not offered (without a paid third-party add-on). Our config utilizes 4 "dynos", hence the 4 IPs for www.my.pdpworks.com. Maybe the single IP for my.pdpworks.com is due to the fact that "my.pdpworks.com" is not yet used because the LE cert is not yet applied?

There hasn't been a cert issued to cover that name.
The "www" cert only covers "www":
crt.sh | 6351235299

2 Likes

From that link, Heroku states:

Let’s Encrypt checks domains against Google’s Safe Browsing API and will not issue certificates for domains considered unsafe.

However, that hasn't been the case any longer for some years now. See this blog post and the update in the "Our plan" section:

Update: As of January 10, 2019, we no longer check domains against the Safe Browsing API.

So, ultimately, I have NO clue what Heroku is refering to.. :roll_eyes: They either have to dig deeper into their own process or have to give you the correct advice.

7 Likes

I think @pdpglobalapp needs to ask Heroku for detailed logs about this issue on their end.

I would also try using Certbot locally with the DNS-01 validation, and seeing if the certificate issues or a clear errors from LetsEncrypt appears.

I would not be surprised if LetsEncrypt interpreted this domain as a blocklist (for being similar to the ADP payment processor), and Heroku is reporting the wrong error. if that's the case, LetsEncrypt staff needs to address this. trying to obtain cert(s) for the affected domain(s) from Certbot is the fastest way to see if there is a domain based issue.

1 Like

Yeah, it seems to me it's possible that Heroku hardcoded the UI for (what was historically the most common reasons for) "policy forbids issuance for name"-type errors, even though they can occur for multiple different reasons.

2 Likes

Let's Debug should try against staging and it doesn't complain: Let's Debug

1 Like

Thank you for the letsdebug.net tip. Test result for my.pdpworks.com using http-01 too.

I have installed Certbot for Windows. I just need to figure how to use it now.

1 Like

Here, have fun. This is what happens if I try asking for a certificate:

% docker run -ti certbot/certbot certonly --staging --webroot -w . -d "my.pdpworks.com" --agree-tos --register-unsafely-without-email
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Account registered.
Requesting a certificate for my.pdpworks.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: my.pdpworks.com
  Type:   unauthorized
  Detail: Invalid response from https://va-acm.heroku.com/challenge?host=www.my.pdpworks.com&token=qTbqoQXdrowiybclZXCA4eSszyVjNfZWH6yI6o1LgBE [3.208.201.80]: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

The challenge obviously fails, but there is no problem making the order.

Even production, it's the same:

% docker run -ti certbot/certbot certonly --webroot -w . -d "my.pdpworks.com" --agree-tos --register-unsafely-without-email
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Account registered.
Requesting a certificate for my.pdpworks.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: my.pdpworks.com
  Type:   unauthorized
  Detail: Invalid response from https://va-acm.heroku.com/challenge?host=www.my.pdpworks.com&token=zioBncnc-Ya-_amCm0KGBEVdhHdSOlBszw3oIVoJcH8 [52.200.99.78]: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

But the request reaching heroku is for www.my, not for my.

2 Likes

So, this is what I think is happening:

You have my.pdpworks.com on aws but www.my.pdpworks.com on heroku.

You try getting a certificate for my.pdpworks.com and the webserver redirects it to www.my.pdpworks.com.

At that point, heroku sees the challenge and responds "wtf, I haven't asked for this validation -- who the hell are you?"

So, either get your certificate from the aws machine (and configure it so that .well-known/acme-challenge is not redirected) or tell heroku about the second domain.

1 Like

I suspect you are on track.

In the AWS/Rout53 zone file for my.pdpworks.com I have an 'A' record with value 52.217.39.3 which is an AWS S3 based static website consisting of nothing more than an .index file with a meta-refresh to the target URL for the my.pdpworks.com domain at Heroku. Perhaps I am needing a certificate on that S3 website?

Ultimately, at Heroku, I want a single domian (my.pdpworks.com) and two CNAME records at AWS/Route53 my.pdpworks.com and www.my.pdpworks.com, both pointing to the Heroku target for my.pdpworks.com. It's a bit of a pickle before the cart situation. Once there is a certificate on my.pdpworks.com at Heroku, I can eliminate the A record and S3 website with meta-refresh.

Good to know! I didn't realize that went deep enough into the ACME process to trigger a blocklist error, but confirmed it did. (I trust you, but i don't TRUST you :wink: )

3 Likes

That is a very strange URL.
Some heavy redirection manipulation going on!

2 Likes

It does: Let's Debug

% docker run -ti certbot/certbot certonly --staging --webroot -w . -d ec2-54-202-156-183.us-west-2.compute.amazonaws.com --agree-tos --register-unsafely-without-email
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Account registered.
Requesting a certificate for ec2-54-202-156-183.us-west-2.compute.amazonaws.com
An unexpected error occurred:
The server will not issue certificates for the identifier :: Error creating new order :: Cannot issue for "ec2-54-202-156-183.us-west-2.compute.amazonaws.com": The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

I thought I might've been wrong because of this:

(I thought both domains were supposed to be blocked)

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.