Let's Encrypt can't see public challenge response

Hi,

I’m using AutoACME to use Let’s Encrypt on Windows Server 2016 and IIS 10. At other sites this has worked flawlessly however at this particular site I’m having the strange issue that AutoACME and therefore LE is returning 503 as the result when getting a challenge response or a timeout however I can access the challenge externally without any issues so I can’t see why LE servers wouldn’t be able to access it too.

The domain is ra.holybrookacademy.co.uk and I’ve left a challenge available at http://ra.holybrookacademy.co.uk/.well-known/acme-challenge/probe_14959e65-f1cc-4dbf-beda-2395ed7d40bc

My domain is: ra.holybrookacademy.co.uk

I ran this command: autoacme.exe addhost ra.holybrookacademy.co.uk

It produced this output:

c:\CertStore\AutoACME>autoacme.exe addhost ra.holybrookacademy.co.uk
Altairis AutoACME Manager version 1.5.4.0
Copyright © Michal A. Valasek - Altairis, 2017
www.autoacme.net | www.rider.cz | www.altairis.cz

Reading configuration from ‘c:\CertStore\AutoACME\autoacme.json’…OK
Checking host…OK
Requesting cerificate for ra.holybrookacademy.co.uk:
Accepting TOS at https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf...OK
Testing authorization:
Writing challenge to C:\InetPub\wwwroot\AutoAcme\probe_dd795b36-3e77-40f0-b162-f10b8531ad27…OK
Testing HTTP challenge:
Preparing request to http://ra.holybrookacademy.co.uk/.well-known/acme-challenge/probe_dd795b36-3e77-40f0-b162-f10b8531ad27...OK
Getting response…Failed!
The remote server returned an error: (503) Server Unavailable.
Testing HTTPS challenge:
Preparing request to https://ra.holybrookacademy.co.uk/.well-known/acme-challenge/probe_dd795b36-3e77-40f0-b162-f10b8531ad27...OK
Getting response…Failed!
The remote server returned an error: (503) Server Unavailable.
Deleting challenge from C:\InetPub\wwwroot\AutoAcme\probe_dd795b36-3e77-40f0-b162-f10b8531ad27…OK
Request failed: One or more errors occurred.
Failed!
Unable to get certificate for new host.

My web server is (include version): IIS 10

The operating system my web server runs on is (include version): Windows Server 2016 Standard

My hosting provider, if applicable, is: N/A

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

GEO-location blocking?
Check the server logs for clues.

I’d start by placing a test.txt file in the challenge folder and ensure it can be accessed from the Internet.

It’s possible but unlikely. The server logs show that Let’s Encrypt isn’t even touching the server so I thought it might be a DNS issue but it’s s fresh DNS record with a short TTL.

I’ve got a test file already up to check I can read from the directory and yes it’s accessible. See http://ra.holybrookacademy.co.uk/.well-known/acme-challenge/blob

blob seems to work.

I’d check the firewall logs for blocks to port 80.

If that leads nowhere, open the challenge folder and watch it while you run the script.
new files should be created in it.
If not… then the files are not going into the correct folder.

Yup checking the firewall logs when I load the blob file I see a record but when I run Let’s Encrypt nothing is generated in the firewall logs.

Yup I’ve checked the folder whilst running the script and it is creating the challenge responses in the right folder and in the right format.

Really running out of options unless there is an issue with the specific endpoint I’m trying to use with Let’s Encrypt although it is just looking at the default acme-v01.api.letsencrypt.org endpoint.

In this case the 503 error appears to be coming from Let’s Encrypt, not from your service. This might be a problem in Let’s Encrypt’s infrastructure, including with the CDN that we use. @jple, who could help investigate whether this is a CDN issue?

try using v02:
acme-v02.api.letsencrypt.org/directory

Unforuntately the LE client I’m using does not yet support v02

That’s what I was leaning towards. Let me know what information you need to diagnose any possible CDN issues.

Resolving acme-v01.api.letsencrypt.org returns e14990.dscx.akamaiedge.net [104.82.239.121]

Sorry for the delay, thanks for the tag @schoen.

Tagging @isk if y’all want to investigate further… I believe this would be Ops.

@joshftw Just caught up on this. Are you still having an issue? I can confirm that currently from the production servers we are able to reach ra.holybrookacademy.co.uk on 80 and 443.

Yes still getting a 503 error. I’ve left a challenge responce in the directory to see if you can reach it here http://ra.holybrookacademy.co.uk/.well-known/acme-challenge/probe_20e07015-aa41-4c8f-965b-a987285f602e

I’ve verified that I can pull that challenge from the production and staging validation servers just fine with a curl.

Looking in the logs at recent attempts via the API I see validation getting a 403 with the following text instead of the validation record:
\"\u003c!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\"\u003e \n\u003chtml xmlns=\"http\""

To make that a little more helpful, the validation attempt appears to be happening. Let’s Encrypt is getting the request and attempting to reach the server. DNS resolution appears to succeed, but it gets an invalid response from the webserver.

At this point, everything seems to be working right from the LE side, but the response from the server is not the expected challenge.

To update on this. I found the solution by monitoring the local firewall logs. The AutoACME application that I was using performed a DNS lookup and I think this was before requesting from the LE servers. The firewall and multiple layers of NAT complicated this by setting the incorrect source IP of the incoming packets from LE. By adding a zone file to the local DNS server for the FQDN of the server’s external address that pointed to the internal IP it resolved correctly and allowed LE to access the server and create a certificate.

Thanks for everyone’s contributions to this, certainly helped get to the right answer.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.