Having Trouble with an HTTP-01 Challenge

Help
1

My domain is: wartorngalaxy.com, torngalaxy.com, www.wartorngalaxy.com

I ran this command: /usr/bin/certbot --reinstall certonly --email certificates@haravikk.com --agree-tos --rsa-key-size 4096 --text --non-interactive --webroot --webroot-path /home/user/com.wartorngalaxy/public_html --domain torngalaxy.com --domain wartorngalaxy.com --domain www.wartorngalaxy.com --deploy-hook 'echo "$RENEWED_LINEAGE" > /tmp/28945.9116.YctUgRUvagu18GO5/certs_dir'

It produced this output:

Renewing an existing certificate
Performing the following challenges:
http-01 challenge for wartorngalaxy.com
http-01 challenge for www.wartorngalaxy.com
http-01 challenge for torngalaxy.com
Using the webroot path /home/dhc-user/com.wartorngalaxy/public_html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. torngalaxy.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: During secondary validation: 2606:4700:3032::ac43:86ac: Invalid response from https://wartorngalaxy.com/.well-known/acme-challenge/nr0UD0YDEnYrX8EcfWCWoPf0gK2To7RoV1uKiKaKnQY: 403
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: torngalaxy.com
   Type:   unauthorized
   Detail: During secondary validation: 2606:4700:3032::ac43:86ac:
   Invalid response from
   https://wartorngalaxy.com/.well-known/acme-challenge/nr0UD0YDEnYrX8EcfWCWoPf0gK2To7RoV1uKiKaKnQY:
   403

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is: nginx version: nginx/1.14.0 (Ubuntu)

The operating system my web server runs on is: Ubuntu 18.04.6 LTS

My hosting provider is: Dreamhost

I can login to a root shell on my machine: yes

I'm using a control panel to manage my site: no

The version of my client is: certbot 0.27.0

The above command is part of a script, which links the renewed certificate into position. This script has always worked well, I've been using it for years at this point, but suddenly it seems to be failing to get a renewed certificate for these domains as the HTTP-01 challenge(s) are failing.

I've verified that the path is accessible, if I throw a test files into .well-known/acme-challenge/test.html I can access it as expected, so somehow the ACME challenge thinks it's creating a file but is then unable to access it for some reason, which has never happened before.

I'm really at a loss as to why this is suddenly failing? Other domains haven't been affected so far, but none of them have needed to be renewed yet. Fortunately the affected domain is not one that's important (it's a placeholder that may or may not eventually be used) but I want to get this fixed before it breaks anything else.

Update: Just wanted to add, but the domains are all behind a Cloudflare reverse proxy, and always have been, so this has never been a problem before. But the fact that a specific IPv6 address is being given makes me wonder if that's part of the problem? I have no control over what IP address each domain appears as through Cloudflare, so they may well be different.

2

Check your nginx logs, there should be a reason it's responding 403 (if it's not cloudflare responding 403. What tls settings are you using in CF? The flexible ones are harder to debug.)

2 Likes
3

Have you always redirected torngalaxy to wartorngalaxy?

The "Secondary" means the request from the Let's Encrypt primary center succeeded. Do you block requests from certain geographic regions?

I also get a 403 using an AWS EC2 test server in US East Coast so do you have any kind of blocking that would affect all of AWS? Or, maybe requiring "human" challenge or something?

curl -I http://torngalaxy.com/.well-known/acme-challenge/Test404
HTTP/1.1 301 Moved Permanently
Location: https://wartorngalaxy.com/.well-known/acme-challenge/Test404

curl -I https://wartorngalaxy.com/.well-known/acme-challenge/Test404
HTTP/2 403
cf-mitigated: challenge
server: cloudflare
2 Likes
4

Have you always redirected torngalaxy to wartorngalaxy?

Yeah, all URLs not matched by another location in the nginx config should go to the same URL with the domain swapped.

The "Secondary" means the request from the Let's Encrypt primary center succeeded. Do you block requests from certain geographic regions?

I don't have anything configured that should be blocking anything geographically.

I also get a 403 using an AWS EC2 test server in US East Coast so do you have any kind of blocking that would affect all of AWS? Or, maybe requiring "human" challenge or something?

Ah, I think the issue that you're seeing is just that a missing file returns 403 rather than 404, you can compare against:
https://wartorngalaxy.com/.well-known/acme-challenge/test.html

I left this file in place for testing.

What's odd that it does seem that torngalaxy.com is redirecting acme challenges even though it's not supposed to, here's a trimmed down version of the nginx configuration for that domain:

server {
        listen 80;
        listen [::]:80;

        listen 443 ssl http2;
        listen [::]:443 ssl http2;

        server_name torngalaxy.com ~^(?<subdomain>(?:[^.]+\.)*)torngalaxy\.com$;
        root /home/dhc-user/com.wartorngalaxy/public_html;

        ssl_certificate         /path/to/cert.pub.pem;
        ssl_certificate_key     /path/to/cert.key.pem;

        # Block hidden files except for the acme challenge as required by Let's encrypt
        location ~ /\. { access_log off; error_log off; deny all; }
        location ~ ~$ { access_log off; error_log off; deny all; }
        location ^~ /.well-known/acme-challenge { try_files $uri $uri/ =403; }

        return 301 https://${subdomain}wartorngalaxy.com$request_uri;
}

When the acme challenge goes through it should return directly, not redirect, which is odd. Not been an issue before, and I haven't changed the configuration at all (in quite a while actually) – perhaps there's some new behaviour around how redirects are handled?

5

What's this all about? I'm pretty sure that doesn't work like that. Although I'm also pretty unsure about why it's there to begin with.

6

Dangit, so it turns out my redirect in the nginx configuration has been wrong all this time – moving this into its own location / block has allowed the acme challenge to return directly.

No idea why it's only now a problem, but once I fix all the similar configuration files (for other redirecting domains) I should be good again.

To summarise the fix. The following:

location ~ /\. { access_log off; error_log off; deny all; }
location ~ ~$ { access_log off; error_log off; deny all; }
location ^~ /.well-known/acme-challenge { try_files $uri $uri/ =403; }

return 301 https://${subdomain}wartorngalaxy.com$request_uri;

Was changed to:

location ~ /\. { access_log off; error_log off; deny all; }
location ~ ~$ { access_log off; error_log off; deny all; }
location ^~ /.well-known/acme-challenge { try_files $uri $uri/ =403; }

location / { return 301 https://${subdomain}wartorngalaxy.com$request_uri; }

Thanks everyone for suggestions and leading me to the problem – I'm still curious why it's only now an issue (as I thought up to ten redirects were okay?) but I'm just happy to get it fixed. I'm guessing it must be some kind of new IP address check or something?

7

Sorry, I had the script spit out the command it was using but I think it's missing some escaping/quotes (as the actual command has variables I wanted swapped for their values to post here), that specific part should look like:

--deploy-hook 'echo "$RENEWED_LINEAGE" > /tmp/28945.9116.YctUgRUvagu18GO5/certs_dir'

Basically that whole chunk is eval'd after the certificate is renewed and there are some useful variables you can put in there, so the value of $RENEWED_LINEAGE (path to the certificates directory) gets saved to a temporary file I can process afterwards.

8

No, I don't think that was it. Although, it would be helpful if you changed that 403 to a 410 or 418 to better isolate that condition.

There were a large number of response headers I got with that 403 that were unique to Cloudflare. Especially the one about "mitigation".

Interestingly, I can only reproduce that "mitigation" 403 using HTTPS. The HTTP 403's have the more "normal" response headers. And, since you have now fixed that redirect it doesn't happen unless I explicitly check using HTTPS

Probably this. Perhaps something regarding their human validation of requests.

2 Likes
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.