Http-01 challenge fails for one domain - nginx - 404

aharown · May 2, 2022, 8:15pm

I ran this command:
certbot --nginx --agree-tos --redirect --hsts --staple-ocsp -d n22x.iacpdev.us --email myaddress@myemail.com

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for n22x.iacpdev.us
Waiting for verification...
Challenge failed for domain n22x.iacpdev.us
http-01 challenge for n22x.iacpdev.us
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: n22x.iacpdev.us
   Type:   unauthorized
   Detail: 2600:3c02::f03c:92ff:feab:a96a: Invalid response from
   http://n22x.iacpdev.us/.well-known/acme-challenge/9gUj3W8V-yp21EW_HjCIP4apLfZhON-ThpdMQSgOP7c:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

If I run

certbot certonly --nginx -d n22x.iacpdev.us --dry-run

I get a 503 instead...

certbot certonly --nginx -d n22x.iacpdev.us --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for n22x.iacpdev.us
Waiting for verification...
Challenge failed for domain n22x.iacpdev.us
http-01 challenge for n22x.iacpdev.us
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: n22x.iacpdev.us
   Type:   unauthorized
   Detail: 2600:3c02::f03c:92ff:feab:a96a: Invalid response from
   http://n22x.iacpdev.us/.well-known/acme-challenge/jj_oM8oPt0yTfwa2gg0FhH3YsKrYrhy-YOTaDs0I15A:
   503

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version):
nginx/1.18.0

The operating system my web server runs on is (include version):
Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-109-generic x86_64)

My hosting provider, if applicable, is:
Linode

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.40.0

What I've checked:

DNS is pointed right. (the domain works currently as http)
I don't see anything wrong in the nginx server blocks
I have one certificate working fine already on this server for a different site and domain name. It has a much simpler nginx server config. (The phpmyadmin)
The one failing is a Drupal 9 site and, as far as I can tell, has a server config the same as one I have SSL working for on another server (except that certbot made the expected changes to it, which it has not made to this one)
I have a CAA record with letsencrypt.org as issue. [Edit: I just found a typo in that... which could be the whole problem. Fixed that. Still failing but now 503 is consistent, rather than 404]

MikeMcQ · May 2, 2022, 8:39pm

Hmmm. I get an http 503 trying the acme challenge path. Only fails with IPv6 as IPv4 works fine (gave 404). IPv6 seems fine with index page URL. Let's Encrypt will use IPv6 if there is an AAAA record. Maybe check your listen for this server?

curl -I6 n22x.iacpdev.us/.well-known/acme-challenge/ForumTest

HTTP/1.1 503 Service Unavailable
Server: nginx/1.18.0 (Ubuntu)
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Cache-Control: must-revalidate, no-cache, private
Date: Mon, 02 May 2022 20:31:19 GMT
X-UA-Compatible: IE=edge
Content-language: en
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Permissions-Policy: interest-cohort=()
Expires: Sun, 19 Nov 1978 05:00:00 GMT
X-Generator: Drupal 9 (https://www.drupal.org)

aharown · May 2, 2022, 9:06pm

I don't currently have a listen for that site. Usually certbot adds the listen block when I run it the first time for the domain.
But maybe what's breaking it is that there is already a listen in the domain I set up previously? I don't remember that being a problem before though.

(My Drupal server blocks look pretty much like this: Drupal | NGINX ... a few edits for root, logging details, otherwise the same)

MikeMcQ · May 2, 2022, 9:53pm

Even for the port 80 server? I don't recall ever seeing that. I thought certbot would create a port 443 server if needed based on the port 80 config. I could be mis-remembering the details.

In any event, the challenge request from the Let's Encrypt server starts with HTTP. It would only try HTTPS if you redirect it. Your first error showed an HTTP URL. Is that still what shows even after fixing the CAA record?

The curl using IPv6 I showed gives different results with IPv4 (use curl -I4). They should not return different things.

You can check the /var/log/letsencrypt/letsencrypt.log file. It will show the temp nginx conf the nginx plugin makes. If you want help upload that here. Or, paste the results of nginx -T here.

Either the AAAA record for this domain is wrong or something wrong with IPv6 listening in nginx. My best guesses.

aharown · May 2, 2022, 10:18pm

Yes, the error still shows a check via http. Hadn't thought of that.
I double checked the IPv6 address in the DNS and that's good. But maybe there's something in my default nginx config that is interfering. I have had some difficulty understanding how to set up a default nginx landing page, but thought I'd finally gotten it right. Maybe not.
Also, nginx server block rules get confusing to me pretty quickly--it's a very dense syntax--but I can definitely experiment with some listen 80 directives in there and see if something falls into place.

_az · May 2, 2022, 10:30pm

I'd try putting in an explicit listen on that Drupal virtualhost. This should give the virtualhost identical behavior on both IPv4 and IPv6:

listen [::]:80 ipv6only=off;

Keep in mind ipv6only may only be present on a single virtual host. On others, just write listen [::]:80; and they will inherit the ipv6only modifier.

aharown · May 3, 2022, 12:00am

Yep, that did it.
Thanks!

Edit:
The certificate went through, and doing nginx reload it seemed stable, but I was getting 403s for URLs of the Drupal site. When I attempted an nginx restart, it failed.
In the log: 2022/05/02 19:49:28 [emerg] 2732#2732: bind() to [::]:80 failed (98: Address already in use)

So that ipv6only line seems to break things. When I comment it out, the nginx starts, but n22x domain is 403.

aharown · May 30, 2022, 3:40pm

Fixed: I was never able to identify exactly what the problem was, but I decided to try restructuring my PHPMyadmin nginx config. I had it set up to use it's own domain name, so it had it's own server block in a /sites-available file and enabled in /sites-enabled.
I had gone over it in detail multiple times and couldn't see any problem. But removing it and routing phpma via a subfolder on another domain (that already had a valid SSL certificate) seems to have eliminated some kind of conflict.
I also deleted all the old certificates, though (sudo certbot delete) and then requested new. I don't recall attempting that before reconfiguring phpma, so it's possible that the delete and reissue solved the problem. But I think more likely there was some kind of conflict with the phpma nginx serverblock.

system · June 29, 2022, 3:41pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.