Renewal failure / Invalid response / 403 on IIS

[boowilbury]

Had been renewing ok for a few years

Am able to read extension-less files in acme-challenge folder on port 80 on remote system eg.
http://www.midlibrary.org/.well-known/acme-challenge/_v9QH54kCgnPYySBd_TIs65ojAdvbfxOaCKdfi1DV1M

letsdebug.net claims all ok
Let's Debug
although, in details states:
[LetsEncryptStaging]
Challenge update failures for www.midlibrary.org in order https://acme-staging-v02.api.letsencrypt.org/acme/order/5751349/3661752444
acme: error code 403 "urn:ietf:params:acme:error:unauthorized": 24.196.137.74: Invalid response from http://www.midlibrary.org/.well-known/acme-challenge/Rfum0zZYQXXoVRYaXbJHj1MNew0W4fX-s210v0CIgUc: 403

Firewall shows no blocks in log

My domain is: www.midlibrary.org

I ran this command: wacs.exe --renew --baseuri "https://acme-v02.
api.letsencrypt.org/" --verbose

It produced this output:
[DBUG] Scanning IIS bindings for hosts
[VERB] 12 named bindings found in IIS
[DBUG] Filtering based on binding type
[DBUG] Filtering by site(s) [2]
[VERB] 3 bindings remaining after site filter
[DBUG] Filtering by host: ^(www.midlibrary.org)$
[VERB] 1 bindings remaining after host filter
[VERB] 1 matching binding found
[DBUG] Scanning IIS sites
[VERB] Source converted into 1 order(s)
[VERB] Checking [IIS] midlibrary.org, www.midlibrary.org
[DBUG] Reading certificate cache
[VERB] v3 cache key not found, fall back to v2
[INFO] Renewing [IIS] midlibrary.org, www.midlibrary.org
[DBUG] Previous certificate found at C:\ProgramData\win-acme\acme-v02.api.letse
ncrypt.org\Certificates\llAXvcQ7L0uYK8tqnIgfpQ-80f32422d4d81331cc69304d00b28e75d
82de754-temp.pfx
[DBUG] Reading certificate cache
[VERB] v3 cache key not found, fall back to v2
[VERB] Obtain order details for Main
[DBUG] Refreshing cached order
[DBUG] Refreshing order...
[DBUG] Send POST to https://acme-v02.api.letsencrypt.org/acme/order/122827994/1
16465708317
[VERB] Request completed with status OK
[WARN] Cached order has status invalid, discarding
[VERB] Creating order for hosts: ["DnsName: www.midlibrary.org"]
[DBUG] Send POST to https://acme-v02.api.letsencrypt.org/acme/new-order
[VERB] Request completed with status Created
[VERB] Order https://acme-v02.api.letsencrypt.org/acme/order/122827994/11648039
8987 created
[DBUG] Send POST to https://acme-v02.api.letsencrypt.org/acme/authz-v3/14239038
3437
[VERB] Request completed with status OK
[WARN] Unable to scan for services
[WARN] Unable to scan for services
[VERB] Handle authorization 1/1
[WARN] Unable to scan for services
[INFO] [www.midlibrary.org] Authorizing...
[VERB] [www.midlibrary.org] Initial authorization status: pending
[VERB] [www.midlibrary.org] Challenge types available: ["http-01", "dns-01", "t
ls-alpn-01"]
[VERB] [www.midlibrary.org] Initial challenge status: pending
[INFO] [www.midlibrary.org] Authorizing using http-01 validation (FileSystem)
[VERB] Writing file to C:\inetpub\wwwroot\midlibrary.well-known\acme-challenge
\PGfCYAuACYKRq_6mihyFSh2ye7o0CsdsYZY83yLd87w
[DBUG] Writing web.config
[VERB] Writing file to C:\inetpub\wwwroot\midlibrary.well-known\acme-challenge
\web.config
[INFO] Answer should now be browsable at http://www.midlibrary.org/.well-known/
acme-challenge/PGfCYAuACYKRq_6mihyFSh2ye7o0CsdsYZY83yLd87w
[DBUG] Send GET to http://www.midlibrary.org/.well-known/acme-challenge/PGfCYAu
ACYKRq_6mihyFSh2ye7o0CsdsYZY83yLd87w
[VERB] Request completed with status OK
[INFO] Preliminary validation looks good, but the ACME server will be more thor
ough
[VERB] Starting commit stage
[VERB] Commit was succesful
[DBUG] [www.midlibrary.org] Submitting challenge answer
[DBUG] Send POST to https://acme-v02.api.letsencrypt.org/acme/chall-v3/14239038
3437/xx1Awg
[VERB] Request completed with status OK
[DBUG] Refreshing authorization (1/15)
[DBUG] Send POST to https://acme-v02.api.letsencrypt.org/acme/chall-v3/14239038
3437/xx1Awg
[VERB] Request completed with status OK
[EROR] [www.midlibrary.org] Authorization result: invalid
[EROR] [www.midlibrary.org] {
"type": "urn:ietf:params:acme:error:unauthorized",
"detail": "216.250.0.78: Invalid response from http://www.midlibrary.org/.well
-known/acme-challenge/PGfCYAuACYKRq_6mihyFSh2ye7o0CsdsYZY83yLd87w: 403",
"status": 403
}
[VERB] Starting post-validation cleanup
[DBUG] Deleting files
[VERB] Deleting file C:\inetpub\wwwroot\midlibrary.well-known\acme-challenge\P
GfCYAuACYKRq_6mihyFSh2ye7o0CsdsYZY83yLd87w
[VERB] Post-validation cleanup was succesful
[INFO] [www.midlibrary.org] Deactivating pending authorization
[DBUG] Send POST to https://acme-v02.api.letsencrypt.org/acme/authz-v3/14239038
3437
[VERB] Request completed with status OK
[VERB] Order 1/1 (Main): error
[VERB] Processing order 1/1: Main
[EROR] Renewal for [IIS] midlibrary.org, www.midlibrary.org failed, will retry
on next run
[VERB] Exiting with status code -1

My web server is (include version): IIS 6.1

The operating system my web server runs on is (include version): Win server 2008R2

My hosting provider, if applicable, is: (self)

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): (win-acme v 2.1.22.1289)

[MikeMcQ]

Welcome to the community @boowilbury

Very nice report. Thanks. Puzzling set of symptoms.

TL;DR: I see a quirk but not a reason for failure

I don't know win-acme well so may need to wait for another volunteer.

I don't see anything wrong in the log until the end where it says it failed. Odd. I also get that URL you describe just fine.
Aside: The response headers say IIS/7.5 not 6.1 as in your post.

The quirk I noticed is your DNS has different IP addresses for your apex and www domain. The "extra" 24. IP for www is related to Spectrum Biz. The other IP related to jvlnet.

Name:   midlibrary.org
Address: 216.250.0.78

Name:   www.midlibrary.org
Address: 24.196.137.74
Address: 216.250.0.78

Yet, requests to either IP give same response. So, that's good and likely not a problem. It is unusual to have multiple IPs, and different between apex and www, but as long as they respond the same it's fine.

[rmbolger]

They both give the same web content, but do both respond with the the ACME challenge? This looks potentially like a site was migrated from one provider to another and someone accidentally left the old IP on the www record. So now you have a 50/50 chance of getting the new or old server when you try to hit www.midlibrary.org.

[MikeMcQ]

I got the same response for both IP's using the test challenge URL provided (and still do). I agree it looks wrong and when done that way it usually responds poorly too.
(Most response headers omitted)

curl -i http://216.250.0.78/.well-known/acme-challenge/_v9QH54kCgnPYySBd_TIs65ojAdvbfxOaCKdfi1DV1M
HTTP/1.1 200 OK
Server: Microsoft-IIS/7.5

_v9QH54kCgnPYySBd_TIs65ojAdvbfxOaCKdfi1DV1M.8Tt0hpV6zhv6qmInkaU-3cHL3ZW30mKflfJvxbnZxK4

curl -i http://24.196.137.74/.well-known/acme-challenge/_v9QH54kCgnPYySBd_TIs65ojAdvbfxOaCKdfi1DV1M
HTTP/1.1 200 OK
Server: Microsoft-IIS/7.5

_v9QH54kCgnPYySBd_TIs65ojAdvbfxOaCKdfi1DV1M.8Tt0hpV6zhv6qmInkaU-3cHL3ZW30mKflfJvxbnZxK4

[boowilbury]

Thank you for your responses! Indeed it is IIs 7.5 (os is 6.1)
I've long had the duplicate A record with a backup ISP IP address, as our faster one would frequently go down - both pointing to the same server. I just removed the 24.196.137.74 from our DNS (provided by 007names.com)

I have read about AAAA and CAA perhaps causing issues, neither of which my domain provider has or gives me access too. I am solely using IPv4. I am not able to decipher the results from unboundtest.com for those records - if AAAA or CAA is my issue.

The API endpoint is accessible from a web browser on my server.

I tried and failed to renew again. I'll try again over the next few hours as the DNS update propagates.

[rg305]

Have you tried putting a "test" text file in the expected challenge folder?
[to see if that file can be reached from the Internet]

[boowilbury]

Indeed, I put a test file (no extension) in the folder and was able to see it remotely.
http://www.midlibrary.org/.well-known/acme-challenge/test

I just installed the CertifyTheWeb application, and that renewed it! I don't know if the DNS change did it, or the app does something different sending the request? I will go through its logs and also try another domain I have that failed with both win-acme and, if needed, CertifyTheWeb.

Thanks all! Nice to have a responsive group of smart people here.

[MikeMcQ]

@boowilbury

I have read about AAAA and CAA perhaps causing issues

Glad you have made progress. Certify The Web is especially good at Windows / IIS systems. It's integration is well-tested and clean. Perhaps there was some subtle issue it coped with better on your server. Totally a guess.

Just some info about above comment ...

The most common CAA problem is for certain kinds of faulty DNS. A CAA record is not required but if one exists the DNS should say "not found". Some DNS respond with SERVFAIL and that's a problem. That would show in Let's Debug and other places so was not happening here.

The AAAA records most often cause a problem when people have one when they don't have a working IPv6 or it points to the wrong server. Again, not the case here.

Both of these have symptoms that are (almost always) more obvious than what we saw here.

Thanks for the kind words.

[webprofusion]

Hi, Certify The Web author here, glad you got it working with CTW, you could probably have got it working with win-acme as well but you would ideally need to use it's self-hosting option, which avoids having to configure IIS to handle the challenge response file properly (extensionless files are problematic by default in IIS).