Http-01 challenges working for some domains but not others

iprodeveloper · April 16, 2025, 10:00pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: ipro.rest

I ran this command: sudo certbot renew -v --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ipro.monster.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Simulating renewal of an existing certificate for ipro.monster
Performing the following challenges:
http-01 challenge for ipro.monster
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ipro.rest.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Simulating renewal of an existing certificate for ipro.rest and www.ipro.rest
Performing the following challenges:
http-01 challenge for ipro.rest
http-01 challenge for www.ipro.rest
Waiting for verification...
Challenge failed for domain ipro.rest
Challenge failed for domain www.ipro.rest
http-01 challenge for ipro.rest
http-01 challenge for www.ipro.rest

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: ipro.rest
  Type:   connection
  Detail: 172.206.168.127: Fetching http://ipro.rest/.well-known/acme-challenge/Ws1lMc2LeRN7Z7vDfSoREBZohMdt8MJ-K0kRub06DNc: Timeout during connect (likely firewall problem)

  Domain: www.ipro.rest
  Type:   connection
  Detail: 172.206.168.127: Fetching http://www.ipro.rest/.well-known/acme-challenge/bKhSPfg86o5zQzEIuYsZls6BD8ZxN1Ar5AdUSE9FOR8: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate ipro.rest with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following simulated renewals succeeded:
  /etc/letsencrypt/live/ipro.monster/fullchain.pem (success)

The following simulated renewals failed:
  /etc/letsencrypt/live/ipro.rest/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

My web server is (include version): nginx 1.18.0

The operating system my web server runs on is (include version): SMP Debian 5.10.234-1

My hosting provider, if applicable, is: Azure

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): Azure VM

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 4.0.0

MikeMcQ · April 16, 2025, 10:13pm

Welcome @iprodeveloper

iprodeveloper:

Domain: ipro.rest
  Type:   connection
  Detail: 172.206.168.127: Fetching http://ipro.rest/.well-known/acme-challenge/Ws1lMc2LeRN7Z7vDfSoREBZohMdt8MJ-K0kRub06DNc: 
Timeout during connect (likely firewall problem)

HTTP requests on port 80 to that domain and its www subdomain are failing with a timeout. The Let's Encrypt server cannot reach you to validate your request.

As the message says, this is often because of a firewall blocking port 80. Your other domain monster uses a different IP address so maybe has different firewall rules.

Maybe something changed since you last got a cert with both those names on Feb7?

I see you got certs more recently with just ipro.rest domain. Although I'm guessing that was on a different setup than you use for this combined cert.

iprodeveloper · April 16, 2025, 10:15pm

This is what I have set up in Azure if that helps. Unfortunately my predecessor (who is no longer at the company) was the one that renewed the certificate last. That other cert renewal was for a different VM (but I'm going to eventually have to do that one as well)

iprodeveloper · April 16, 2025, 10:29pm

I took a look at the other VMs certs and they have the same issue where they're failing on the http-01 challenge both with timeouts

MikeMcQ · April 16, 2025, 11:26pm

Do you have an Azure Load Balancer in front of your Debian servers?

If so, is that directing the HTTP (port 80) requests to those servers properly?

I ask because HTTPS (port 443) requests get a reply from something like this:

Microsoft-Azure-Application-Gateway/v2

iprodeveloper · April 16, 2025, 11:42pm

Yes, I have a Load Balancer in front of the servers. Where would I go to see if HTTP requests are going to the right server?

MikeMcQ · April 17, 2025, 12:23am

I'd start at the top of those settings and work your way through

I am not an Azure expert so maybe some other volunteer will have more specific advice.

But, we know requests using HTTP (port 80) are not getting a response. Not only did Let's Encrypt report a problem but also the Let's Debug site I linked.

And, also this test site which checks multiple locations world-wide: Check website performance and response : Check host - online website monitoring

webprofusion · April 17, 2025, 12:28am

Keep in mind that if you are using a load balancer or application gateway you still need to forward http (tcp port 80) to the target server (so that the acme client can help respond to the http challenge).

If there is more than one server that may respond for that domain then you likely cannot use http domain validation, because the traffic will inevitably be directed to the wrong server. For those situations you can use DNS domain validation instead of http.

iprodeveloper · April 17, 2025, 6:00pm

I stopped the load balancer/application gateway using the Azure CLI and tried again only for it to fail at the http-01 test again. Do you have any other suggestions or should I look into DNS domain validation?

MikeMcQ · April 17, 2025, 6:12pm

Is hard to say without more details.
What was the error message?
Did you have the DNS setup properly since it would no longer point to the load balancer.
What was the command you tried?

iprodeveloper · April 17, 2025, 6:15pm

I used this command to stop it (replaced the sub-id):

az network application-gateway stop --id /subscriptions/sub-id/resourceGroups/load-balancer-resources/providers/Microsoft.Network/applicationGateways/ipro-application-gateway-load-balancer

The output was this (same as before):

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ipro.rest.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Simulating renewal of an existing certificate for ipro.rest and www.ipro.rest
Performing the following challenges:
http-01 challenge for ipro.rest
http-01 challenge for www.ipro.rest
Waiting for verification...
Challenge failed for domain ipro.rest
Challenge failed for domain www.ipro.rest
http-01 challenge for ipro.rest
http-01 challenge for www.ipro.rest

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: ipro.rest
  Type:   connection
  Detail: 172.206.168.127: Fetching http://ipro.rest/.well-known/acme-challenge/C_sBKMg3sZ6PgslxIf8wH0VYS8hilXINRujM8Cq5CYQ: Timeout during connect (likely firewall problem)

  Domain: www.ipro.rest
  Type:   connection
  Detail: 172.206.168.127: Fetching http://www.ipro.rest/.well-known/acme-challenge/ppBbbR337OgRKmlvlGISYhlYlky23TqtX5JiTdCgLr8: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate ipro.rest with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following simulated renewals succeeded:
  /etc/letsencrypt/live/ipro.monster/fullchain.pem (success)

The following simulated renewals failed:
  /etc/letsencrypt/live/ipro.rest/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

I'm still pretty new to this, so I'm not sure how to set up the DNS properly, is there a resource you could guide me towards?

I also saw something about ipv4 vs tcp6 protocols, and I can confirm that the port 80 on my VM is using tcp6, I don't know if that's a real issue or not

iprodeveloper · April 17, 2025, 7:57pm

I don't know if this helps, but when I try to run sudo nginx I get this output which seems to confirm nginx is not able to access port 80?

nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
nginx: [emerg] still could not bind()

MikeMcQ · April 17, 2025, 8:09pm

It could not open port 443 either.

Something else was using them. certbot --nginx will try to start an nginx if one was not running. Probably what happened here.

You can see what is listening on those ports with this

sudo ss -pant | grep -i listen | grep -Ei ':80|:443|nginx'

I don't think stopping your load balancer will also reconfigure your DNS to point directly to your servers.

Without the load balancer the A and/or AAAA records should be the public IP for your servers. There are many ways to know what those are. One way is this

curl -4 https://ifconfig.io
curl -6 https://ifconfig.io

If you don't have an IPv6 address you shouldn't have an AAAA record either. Use https://unboundtest.com to check the values in the DNS for each type of record (for each domain name)

iprodeveloper · April 17, 2025, 8:25pm

When I run that sudo command I get this, looks like nginx is on all of them:

LISTEN    0      511          0.0.0.0:80           0.0.0.0:*     users:(("nginx",pid=11680,fd=6),("nginx",pid=11679,fd=6),("nginx",pid=11597,fd=6))
LISTEN    0      511          0.0.0.0:443          0.0.0.0:*     users:(("nginx",pid=11680,fd=9),("nginx",pid=11679,fd=9),("nginx",pid=11597,fd=9))
LISTEN    0      511             [::]:80              [::]:*     users:(("nginx",pid=11680,fd=7),("nginx",pid=11679,fd=7),("nginx",pid=11597,fd=7))
LISTEN    0      511             [::]:443             [::]:*     users:(("nginx",pid=11680,fd=8),("nginx",pid=11679,fd=8),("nginx",pid=11597,fd=8))

the curl-4 gave back the IP address of the VM but the curl -6 came back with "curl: (7) Couldn't connect to server" Seems like I don't have an IPv6 address?

MikeMcQ · April 17, 2025, 8:37pm

And did that match what you have in the public DNS?

dig +noall +answer ipro.rest
ipro.rest.              300     IN      A       172.206.168.127

As for the nginx bind errors to port 80 and 443, there is also a known bug with Certbot if you run certbot --nginx without nginx already running. Could you have done that?

What can happen in that case is Certbot starts nginx but not using systemd. So, you can essentially get multiple copies of nginx running and one won't let the other use the port(s).

These port errors can also happen if you use a non-standard nginx install.

What does this show?

sudo systemctl status --no-pager -l nginx

iprodeveloper · April 17, 2025, 10:07pm

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2025-04-17 20:39:37 UTC; 1h 27min ago
       Docs: man:nginx(8)
   Main PID: 669 (nginx)
      Tasks: 3 (limit: 4635)
     Memory: 14.2M
        CPU: 275ms
     CGroup: /system.slice/nginx.service
             ├─669 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─670 nginx: worker process
             └─671 nginx: worker process

Apr 17 20:39:37 Buster systemd[1]: Starting A high performance web server and a reverse proxy server...
Apr 17 20:39:37 Buster systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument
Apr 17 20:39:37 Buster systemd[1]: Started A high performance web server and a reverse proxy server.

I think it might be a DNS issue, I'm waiting to get credentials so I can get an API key hopefully tomorrow

webprofusion · April 18, 2025, 12:09am

The simplest test is to try accessing your site externally using HTTP (tcp port 80), not https. If that works OK then http domain validation will usually work. If you use something like curl you can perform a simple test:

curl -I http://ipro.rest/.well-known/acme-challenge/configcheck

Which should return a 404 or at least some response from your web server. If it doesn't you just don't have working http, and the usual reason for that is your cloud network settings don't allow http traffic to reach your server (or a machine firewall stops is).

Adding application gateway (which is load balanced reverse proxy which terminates TLS for you) to the mix makes it extra complex, so yes using DNS validation should be easier.

MikeMcQ · April 18, 2025, 12:20am

iprodeveloper:

     CGroup: /system.slice/nginx.service
             ├─669 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─670 nginx: worker process
             └─671 nginx: worker process

Those are the PIDs of the currently running nginx.

But, earlier you showed these PIDs as the ones listening. Did you restart nginx? Is this the same machine?

iprodeveloper · April 18, 2025, 6:31pm

So I ended up solving the issue. Rebooting the VM along with changing the DNS in GoDaddy solved it! Thanks for all your help guys

Topic		Replies	Views
Some challenges have failed; HTTP-01 challenge success Help	7	677	November 30, 2023
Http-01 challenge failed Help	9	887	March 9, 2023
Auto-renewing failed - http-01 challenge for Help	3	968	August 18, 2018
Challenge failed for domain http-01 Help	6	43672	April 1, 2020
Dns-01 challenge - something has changed Help	11	4358	January 2, 2020

Http-01 challenges working for some domains but not others

Related topics