Getting 403 errors and other weird stuff when running acme V2.1.7.8072

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: online.uah.edu

I ran this command: wacs.exe

It produced this output:
Target generated using plugin IIS: online.uah.edu
First chance error calling into ACME server, retrying with new nonce…
Authorize identifier online.uah.edu
Authorizing online.uah.edu using http-01 validation (SelfHosting)
{
“type”: “urn:ietf:params:acme:error:unauthorized”,
“detail”: “Invalid response from http://online.uah.edu/.well-known/acme-challenge/weS4J6DbpQcARgNP2tHyh1o2GNaHJ7ECWm_yBewExiE [52.141.211.79]: “\r\n<!doctype html>\r\n<html lang=\“en\” \r\n\t\r\n>\r\n\r\n<meta http-equiv=\“X-UA-Compatible\” content=\“IE=edge,chrome=1\” />\r\n<meta chars””,
“status”: 403
}
Authorization result: invalid

My web server is (include version): IIS 10

The operating system my web server runs on is (include version): Windows 2019 datacenter (Azure vm)

My hosting provider, if applicable, is: Azure

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

1 Like

This is the response when I ran the wacs.exe and chose a new cert.

Background. We have been running sites on windows 2012 IIS version 8.5 with a Kemp load balancer (layer 4). we have migrated all our sites to new win 2019 IIS 10 servers in Azure. there are 2 servers in a backend pool running behind an application gateway. The gateway has listeners for both port 80 and 443 and both ports are accessible (http and https site). In spite of many tries we kept getting the error above. Then suddenly it worked on another site. We have yet to get it to run properly again.

Also sometimes the after selecting the options (bindings) and you are prompted to continue (y*/N) you type a “y” and the command prompt disapppears and nothing happens. I am at a lose as to what is happening
Mike

1 Like

Hi @michael.schindler

is Wacs.exe able to write in the correct subdirectory? Should be only one subdirectory, not two on different servers.

Your certificate expires Monday, that’s short. Do you have DNS access?

If yes, you can use dns validation (creating a TXT entry _acme-challenge.online.uah.edu is required) with something like manual validation, that should always work.

Then you have a new certificate and you have enough time to fix the http validation.

1 Like

Juergen,

I do have access to dns for about 80% of our domains I will try to use the manual for now
Mike

1 Like

I also have another cert that expires in July so I am going to use that one
Mike

1 Like

Your site (or inline protection systems) is handling that same HTTP request “differently”.

From all my systems I get an initial “301 moved” and then “404 not found” (from the redirected link)
This seems quite “normal”.
But comparing that to the wacs error shown above (from LE), it isn’t seen failing on the redirected link.
And LE will follow such a simple redirection.
So LE must NOT be getting the “301 moved” page.
And, as shown within the error message,

“type”: “urn:ietf:params:acme:error:unauthorized”,
“detail”: “Invalid response from http://online.uah.edu/.well-known/acme-challenge/weS4J6DbpQcARgNP2tHyh1o2GNaHJ7ECWm_yBewExiE [52.141.211.79]: “\r\n<!doctype html>\r\n<html lang=\“en\” \r\n\t\r\n>\r\n\r\n<meta http-equiv=\“X-UA-Compatible\” content=\“IE=edge,chrome=1\” />\r\n<meta chars””,
“status”: 403

LE gets a 403 unauthorized message.

I can only presume that “something” in your network* is deciding to handle the LE requests very differently than it should be and is seen to be for others.
[first hand witness]

Note: “your network” includes your IIS 10 web system as well as any security/proxy/load-balancing/etc. system within that environment and/or along the way there.

1 Like

Thanks all for the help. Unfortunately I do not feel any closer to a solution. Hopefully I can add more clarity.

We are using Microsoft azure application gateway V2. We have 4 multisite listeners for every domain. So for online.uah.edu we have listeners for:
online.uah.edu:80 this redirects to online.uah.edu:443
online.uah.edu:443 this goes to back end servers
www.online.uah.edu:80 this redirects to online.uah.edu:443
www.online.uah.edu:443 this redirects to online.uah.edu:443

this is why you see some 301 replies

also when I run the macs.exe I do not see the .well-known folder created

I gave all users full control to the root folder and still the .well-known folder was not created

Mike

1 Like

What are you running the ACME client on? One of the application servers or the gateway?

In clustered environments, I either run ACME clients on the gateway/loadbalancer or proxy all network traffic for the /.well-known/ directory on every domain to a single particular server. I don’t bother with redirects or specialized domains for authorization - I’ve found that is subject to too many local network/config problems. I just do a standard http proxy, which will stash the intended domain in the headers. IMHO that makes it much easier to troubleshoot.

1 Like

We are running wacs.exe on one of the backend servers (IIS). We run about 60 sites that are all subdomains of other entities.

Any references I can check out to run the the acme client on the azure application gateway (a layer 7 proxy)

Thanks

Mike

1 Like

The clarity added doesn’t even begin to explain the 403 unauthorized error message.
It does clearly explain the 301 messages.
But those were previously deemed quite “normal” and not relevant to the 403 error.

Are there any error log entries that cover the 403 errored requests on the /.well-known/acme-challenge/ files?

1 Like

RG305
you are correct it does not really explain the 403 errors. When I run the Macs.exe and try to renew the cert I never see the .well-known folder created. The very odd thing is that once the process ran exactly as it should.

When The tool runs I should see the .well-known folder be created shouldn’t I

Mike

1 Like

I can’t speak to that.

If you have a web server that has error logging enabled, perhaps there are some error log entries that may provide us some useful information about this problem.

1 Like

This is from the web site logs:
2020-05-13 22:35:44 10.1.13.81 GET /.well-known/acme-challenge/2nSSMur4UW5DlD5jUgECvgCJBjSj_ZFdOy85QGj_jJ8 - 443 - 10.1.12.164 Mozilla/5.0+(compatible;+Let’s+Encrypt+validation+server;++https://www.letsencrypt.org) http://onlinedegrees.murraystate.edu/.well-known/acme-challenge/2nSSMur4UW5DlD5jUgECvgCJBjSj_ZFdOy85QGj_jJ8 404 0 2 2
i ran this on a site that is not yet serving a partner of ours but dns and everything is set up

When I look on our old server (iis 8.5 behing a L4 load balancer) these get commands have a 301 response.

Thanks again
Mike

1 Like

That entry appears to be a 404 error.
[still an issue - but not the one we are working on]
Can you filter/search the logs for only 403 and acme-challenge entries?
Also, the IP entries listed appear to be only internal.

1 Like

this may be a simpler issue than I have lead on. We use a “hardened image” in azure. I created a different backend pool using a normal image. lets encrypt worked exactly as it should.

I will update with more info as I do more testing
Mike

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.