Certbot HTTP challenge validation failing with 404 error

My domain is: w-ctilab.ims.dev.tdc.dk

I ran this command: certbot certonly --webroot -w /var/broadworks/webapps/default -d w-ctilab.ims.dev.tdc.dk -v

It produced this output:certbot certonly --webroot -w /var/broadworks/webapps/default -d w-ctilab.ims.dev.tdc.dk -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Requesting a certificate for w-ctilab.ims.dev.tdc.dk
Performing the following challenges:
http-01 challenge for w-ctilab.ims.dev.tdc.dk
Using the webroot path /var/broadworks/webapps/default for all unmatched domains.
Waiting for verification...
Challenge failed for domain w-ctilab.ims.dev.tdc.dk
http-01 challenge for w-ctilab.ims.dev.tdc.dk

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: w-ctilab.ims.dev.tdc.dk
Type: connection
Detail: 87.51.46.59: Fetching http://w-ctilab-xsp1.ims.dev.tdc.dk/.well-known/acme-challenge/KCmwGYtMsZ_FP-UU-nHZ4K3N2LiJO1G1M-Vrs4s2csk: Redirect loop detected

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Cleaning up challenges
Some challenges have failed.

My web server is (include version): tomcat 9.0.91 (its a Cisco's COTs product but webserver is Tomcat at the backend)

The operating system my web server runs on is (include version): Red Hat Enterprise Linux 8.10 (Ootpa)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no , the DNS is managed separately.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.22.0

Welcome @apoorva

There are a couple problems with your setup. You have two IP addresses for that domain

w-ctilab.ims.dev.tdc.dk. 0 IN A 87.51.47.59
w-ctilab.ims.dev.tdc.dk. 0 IN A 87.51.46.59

With --webroot the Let's Encrypt server requires both of them to reply with the correct challenge response. This is hard to make work with the standard Certbot request.

As shown by Let's Debug test: Let's Debug
Your two IP addresses respond different. The '.47.59' refuses connections and the '.46.59' causes an HTTP redirect loop.

Neither of these would work for a cert challenge. Can you describe the purpose of these two IP? Is there a load balancer, for example. Or is it just a mistake?

3 Likes

So based on your inputs I have changed the redirect rule to a internal FQDN that returns a single IP but now I am getting the below error:

Requesting a certificate for w-ctilab.ims.dev.tdc.dk
Performing the following challenges:
http-01 challenge for w-ctilab.ims.dev.tdc.dk
Waiting for verification...
Challenge failed for domain w-ctilab.ims.dev.tdc.dk
http-01 challenge for w-ctilab.ims.dev.tdc.dk

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
Domain: w-ctilab.ims.dev.tdc.dk
Type: dns
Detail: During secondary validation: DNS problem: looking up CAA for ims.dev.tdc.dk: DNSSEC: Bogus: validation failure <ims.dev.tdc.dk. CAA IN>: key for validation dev.tdc.dk. is marked as invalid because of a previous not all NSEC3 records secure

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

No, that is still not working. And, you still have two IP in the public DNS for that domain.

The Let's Encrypt ACME Server uses the public DNS to see which IP addresses to use. You still have two IP there. The LE Server may (randomly) choose either one. So, both must be able to reply correctly.

Certbot, the ACME Client, places the challenge token in the --webroot path you give it. Your system must be able to return that token for HTTP requests using either of those IP.

It would help us to know why you have two IP addresses. Can you explain that?

2 Likes

And when I try with webroot , it still fails

certbot certonly --webroot -w /var/broadworks/webapps/default -d w-ctilab.ims.dev.tdc.dk -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Requesting a certificate for w-ctilab.ims.dev.tdc.dk
Performing the following challenges:
http-01 challenge for w-ctilab.ims.dev.tdc.dk
Using the webroot path /var/broadworks/webapps/default for all unmatched domains.
Waiting for verification...
Challenge failed for domain w-ctilab.ims.dev.tdc.dk
http-01 challenge for w-ctilab.ims.dev.tdc.dk

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: w-ctilab.ims.dev.tdc.dk
Type: connection
Detail: 87.51.47.59: Fetching https://w-ctilab-xsp2.ims.dev.tdc.dk/.well-known/acme-challenge/aR3L5AZD_iGgAp3eIj5dVuBUWOqUcCaG0suJTfk1csI: Redirect loop detected

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

I have a main FQDN lets say w-ctilab.ims.dev.tdc.dk , which has two FQDNs and which resolves to two IP's , this is for redundancy. Also if you see, the certbot is on one of these IPs and I am trying to renew certificate for main FQDN. Handled the redirection within rules to point it to one of the internal FQDN, where certbot is installed but somehow certbot send requests to both

Then you probably need to use a DNS Challenge. It is difficult to organize your local system so that both IP can respond correctly for HTTP challenges.

I didn't notice you switched to trying --standalone. But, that probably won't work for you either. Requests to either of those IP have to be sent by your system to wherever you have Certbot --standalone running. If you can do that it should work. But, I think DNS Challenge is probably better way forward.

The problem with DNS Challenge for you will be automating the renewals. I have never seen your DNS provider before and am not sure if any ACME Client supports its API. Or, if your DNS provider even has an API. If not you will have to use a CNAME to delegate the name somewhere else.

2 Likes

Hi Mike,

I have the exact problem that you mentioned, our DNS provider doesn't have an API, they are working on an upgrade but it will take too long. I tried to look into the possibility of this CName based approach but I didn't find any good information on How Actually this needs to be handled. Is it at DNS level that we need to get this configured ?

Can you probably share some more information on this Cname based approach.

Hi @Apoorva,

From the link Mike posted, states
“Since Let’s Encrypt follows the DNS standards when looking up TXT records for DNS-01 validation, you can use CNAME records or NS records to delegate answering the challenge to other DNS zones. This can be used to delegate the _acme-challenge subdomain to a validation-specific server or zone. It can also be used if your DNS provider is slow to update, and you want to delegate to a quicker-updating server.“

Also see: CNAME record - Wikipedia

Edit

And here is on suggest example DNS-01 challenge acme-dns internal Bind - #12 by jvanasco

3 Likes

Hi Bruce,

So now after making some corrections, I am able to navigate to same IP where certbot is installed. But I am getting below error now:

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: w-ctilab.ims.dev.tdc.dk
Type: unauthorized
Detail: During secondary validation: 87.51.46.59: Invalid response from https://w-ctilab-xsp1.ims.dev.tdc.dk/.well-known/acme-challenge/.well-known/acme-challenge/V1XlJGe7EiRgVykAWbR7V2iUUwiET0CxEb37q6Do3qE: 404

You still have two IP for your domain. Are you somehow redirecting the HTTP Challenge requests to a single server that Certbot can update?

Because otherwise the situation looks the same as before: Certbot HTTP challenge validation failing with 404 error - #4 by MikeMcQ

2 Likes

Hi Mike,

Yes now I have added a redirection rule, that whenever the request comes it is redirected to the certbot server.
RewriteCond %{REQUEST_URI} !^/well-known*
RewriteRule ^/(.*) https://w-ctilab-xsp1.ims.dev.tdc.dk/.well-known/acme-challenge/$1 [L,R]

Also today I was able to successfully generate certificate for the subfqdn , have a look: But unfortunately we need a certificate for the main FQDN as well.

certbot certonly --webroot -w /var/broadworks/webapps/default -d w-ctilab-xsp2.ims.dev.tdc.dk -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Certificate is due for renewal, auto-renewing...
Renewing an existing certificate for w-ctilab-xsp2.ims.dev.tdc.dk
Performing the following challenges:
http-01 challenge for w-ctilab-xsp2.ims.dev.tdc.dk
Using the webroot path /var/broadworks/webapps/default for all unmatched domains.
Waiting for verification...
Cleaning up challenges

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/w-ctilab-xsp2.ims.dev.tdc.dk/fullchain.pem
Key is saved at: /etc/letsencrypt/live/w-ctilab-xsp2.ims.dev.tdc.dk/privkey.pem
This certificate expires on 2025-02-10.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.


If you like Certbot, please consider supporting our work by:


Yes, I see that domain name only has one IP address in the DNS. That is much easier to get working. UPDATE: I now see two IP again using https://unboundtest.com - did you change the DNS temporarily?

What is that domain name?

2 Likes

did you change the DNS temporarily? - No , can you share how it is showing you now?

What is that domain name? - w-ctilab.ims.dev.tdc.dk

From https://unboundtest.com I now see a single IP (again). Just the ".47.59" IP:
https://unboundtest.com/m/A/w-ctilab-xsp2.ims.dev.tdc.dk/BZ23VYRB

But, this domain w-ctilab.ims.dev.tdc.dk is showing two IP.
https://unboundtest.com/m/A/w-ctilab.ims.dev.tdc.dk/66RC7R2O

2 Likes

Yes Mike, the FQDN w-ctilab.ims.dev.tdc.dk is configured in a way that it redirects to one of those two IP. I have tried to write a redirection rule to send the request always to one IP. But now I am getting 404 on that node.

So if you see below, all requests I am firing for w-ctilab.ims.dev.tdc.dk is reaching to w-ctilab-xsp1.ims.dev.tdc.dk everytime , but when it reaches the .well-known directory it says 404.

  "type": "http-01",
  "url": "

https://acme-v02.api.letsencrypt.org/acme/chall-v3/429118317817/f-CBEw"
,
"status": "invalid",
"validated": "2024-11-12T14:11:56Z",
"error": {
"type": "urn:ietf:params:acme:error:unauthorized",
"detail": "87.51.46.59: Invalid response from
https://w-ctilab-xsp1.ims.dev.tdc.dk/.well-known/acme-challenge/.well-known/acme-challenge/f_aT6mofVf5J_BaOSoLdBnnfbrrp5PxaVxh8NlTi1sM:
404",
"status": 403
},

Does that even work?
It has condition "NOT" well-known, redirect to HTTPS...well-known...

1 Like

Yes it works well.

Thank you all for the support , as of now I was able to generate the certificate successfully, by doing two changes.

  1. Change in rewrite.config to correct URL:
    RewriteRule ^/(.*) https://w-ctilab.ims.dev.tdc.dk/$1 [L,R]
  2. manually creating the .well-known folder, looks like it might be a permission issue on our webroot for any other user
1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.