The key authorization file from the server did not match this challenge

My domain is: mediconfort.com

I ran this command:

I have a pod on kubernetes that run this command "certbot certonly --webroot --webroot-path /usr/share/nginx/html -d mediconfort.com"

It produced this output:

'Error accepting authorization: acme: authorization error for mediconfort.com: 403 urn:ietf:params:acme:error:unauthorized: The key authorization file from the server did not match this challenge. Expected "dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98.nX2cH20pStJZqsIBA9rt1ll1_2Iy2KPJsqTGEJkHKVI" (got "dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98.4E3VCTFsySjUrqnCg0ooULx-3kbdPBygi0aWkvg5Gd8"

Notice that if I do a "curl -i -v http://mediconfort.com/.well-known/acme-challenge/dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98".
I receive the correct challenge dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98.nX2cH20pStJZqsIBA9rt1ll1_2Iy2KPJsqTGEJkHKVI
although the ACME server states that it received dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98.4E3VCTFsySjUrqnCg0ooULx-3kbdPBygi0aWkvg5Gd8

that's the whole output and is always the same with multiple curl's...

root@archlinux: certificates:#curl -i -v http://mediconfort.com/.well-known/acme-challenge/dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98; echo 
* Host mediconfort.com:80 was resolved.
* IPv6: 2001:41d0:1:1b00:213:186:33:18
* IPv4: 195.154.139.76
*   Trying 195.154.139.76:80...
* Connected to mediconfort.com (195.154.139.76) port 80
> GET /.well-known/acme-challenge/dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98 HTTP/1.1
> Host: mediconfort.com
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< date: Wed, 17 Jan 2024 20:14:03 GMT
date: Wed, 17 Jan 2024 20:14:03 GMT
< content-type: text/plain; charset=utf-8
content-type: text/plain; charset=utf-8
< content-length: 87
content-length: 87
< cache-control: no-cache, no-store, must-revalidate
cache-control: no-cache, no-store, must-revalidate
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< referrer-policy: strict-origin
referrer-policy: strict-origin

< 
* Connection #0 to host mediconfort.com left intact
dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98.nX2cH20pStJZqsIBA9rt1ll1_2Iy2KPJsqTGEJkHKVI

Notice I am curling a public ip and i do not reside inside a VPN, also my webserver serving the challenge does not use any cache! It's weird that my curl is returning a different challenge than what the "letsencrypt ACME server" is stating... we are both curling the same url http://mediconfort.com/.well-known/acme-challenge/xxxxxxxxxxxx I suppose!

I have tried with several different letsencrypt accounts (having different emails) on different pods (meaning no chance to have multiple accounts running at once inside the pod...), and I always have the same error from the ACME server although my curl returns the correct challenge!

Notice it does work fine with other certs like xxx.mediconfort.com

I have waited several days before each try cause i suspected there was some sort of caching issue somewhere on the ACME server sides (not sure)!

I am getting a bit out of ideas :frowning:

My web server is (include version): .NET 8.0

The operating system my web server runs on is (include version): Linux

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): no

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.7.4

I don't have my normal tools available right at the moment but it looks like your IPv4 and IPv6 address don't resolve to the same server. It looks like your test ends up using the IPv4 but let's encrypt prefers the IPv6 which is being handled by an nginx server

Type=IPv6,Server=nginx,

See the let's debug result here

3 Likes

I have to agree with @MikeMcQ.
The IPv4 and IPv6 responses don't look like they are coming from the same system:

curl -Ii4 mediconfort.com/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 404 Not Found
date: Thu, 18 Jan 2024 11:39:51 GMT
content-type: text/html
content-length: 146
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
referrer-policy: strict-origin

curl -Ii6 mediconfort.com/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 404 Not Found
server: nginx
date: Thu, 18 Jan 2024 11:39:59 GMT
content-type: text/html; charset=utf8
x-ovh-webhosting-id: 77795a1f8ee1d476e182a2e3ae81d321
pragma: no-cache
cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
content-length: 4981
x-iplb-request-id: 260017005230A1900192016801990199:27E3_200141D000011B000213018600330018:0050_65A90E0F_31C72:4535
x-iplb-instance: 51825

Even the two IP networks don't seem to be from the same HSP:
ARIN Whois/RDAP - American Registry for Internet Numbers - 195.154.139.76
ARIN Whois/RDAP - American Registry for Internet Numbers - 2001:41d0:1:1b00:213:186:33:18

You mention .NET 8.0 web server
One of those responses shows an nginx server.

4 Likes

thank you guys for the tips, seems like an AAAA is set and forgotten!

unlike your's though the curl does not resolve on ipv6:

curl -Ii6 http://mediconfort.com/.well-known/testing
curl: (7) Failed to connect to mediconfort.com port 80 after 7 ms: Couldn't connect to server

tried to nmap (it won't resolve obviously!)

nmap -6 -sS mediconfort.com -oN nslookup.ipv6

# Nmap 7.94 scan initiated Thu Jan 18 17:34:33 2024 as: nmap -6 -sS -oN nslookup.ipv6 mediconfort.com
setup_target: failed to determine route to mediconfort.com (2001:41d0:1:1b00:213:186:33:18)
WARNING: No targets were specified, so 0 hosts scanned.
# Nmap done at Thu Jan 18 17:34:33 2024 -- 0 IP addresses (0 hosts up) scanned in 0.02 seconds
1 Like

Does your outbound IPv6 work for any others? Like

curl -6 https://ifconfig.io

Because Let's Debug sees an nginx server at that IPv6 just fine. As do I from my own test server

But, that nginx isn't yours is it?

3 Likes

Good point! yeah you are right, my curl -6 is down for any destination! the following is not working neither

root@archlinux: certificates:#
url=ifconfig.io
curl -v -6 https://$url
* Host ifconfig.io:443 was resolved.
* IPv6: 2606:4700:e6::ac40:c603, 2606:4700:e6::ac40:c703
* IPv4: (none)
*   Trying [2606:4700:e6::ac40:c603]:443...
* connect to 2606:4700:e6::ac40:c603 port 443 from fe80::e63d:22ea:cc10:f9cb port 56396 failed: No route to host
*   Trying [2606:4700:e6::ac40:c703]:443...
* connect to 2606:4700:e6::ac40:c703 port 443 from fe80::e63d:22ea:cc10:f9cb port 41428 failed: No route to host
* Failed to connect to ifconfig.io port 443 after 8718 ms: Couldn't connect to server
* Closing connection
curl: (7) Failed to connect to ifconfig.io port 443 after 8718 ms: Couldn't connect to server

The nginx might be mine! That's what I'am trying to figure out... cause the pod is behind a nginx ingress!

Not 100% sure if the nginx is mine, cause I cannot ping the pod with any ipv6...

I did some "ip route add" to fix the curl ipv6 issue,,, still curl -6 is not reaching any destinations/hosts...

ip route add ::/0 dev enp0s8

So iam left at figuring out if nginx is mine! (most probably is)... i will update once i have a final clue

Some updates on ipv6 not working if someone is looking for some answers!

looks like my ISP does not provide IPv6

to check browse to https://test-ipv6.com/...

Getting back to the question, is the nginx mine! well i do not have any other nginx running on my system and from the "ACME server response"

Expected "dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98.nX2cH20pStJZqsIBA9rt1ll1_2Iy2KPJsqTGEJkHKVI"

(got "dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98.4E3VCTFsySjUrqnCg0ooULx-3kbdPBygi0aWkvg5Gd8"

looks like the challenge part dKX8XN5WrdpPrAxELrfcJnkXkUeaUzcS3Q2IS_6cD98 is present on both the Expected and got parts... but the account part (after the dot .) differs!

So if the challenge exists in any form (as in Expected or as in got), I presume that must be my nginx which is routing to my pod... so i'am back to square one :frowning:

Then you should see those requests in your nginx access_log

It is still odd that your IPv4 is responded to by one server and the IPv6 by something else. Maybe your inbound IPv6 is just broken.

If your ISP does not support IPv6 you should remove the AAAA records

3 Likes

just as info, I'am working on a remote system... the servers are not tied to my own ISP.

I will be cleaning that remaining ivp6 AAAA record, and will check later how it turns out.

thanks for the tip

2 Likes

Ah, I understand what you meant by that now.

Which makes sites like https://letsdebug.net even more helpful. I may have mentioned this once or twice before :slight_smile:

It makes requests to a domain to test HTTP Challenges. It is very good for comms problems. It doesn't find everything. Sometimes it will say OK even if there is still a problem (like a wrongly configured server). But, if it fails you should fix what it says.

Run a test for your root and your www domain. Click its "Verbose Info" for detail results. To see a 404 Not Found is expected since the test systems look for files not expected to exist. It is just testing connectivity and related elements.

2 Likes

oh ok nice!

2 Likes

Just as an update, the "ACME Server is pinging my pod"! and the nginx is mine

kubectl logs -n $namespace ingress-nginx-internal-controller-ldnzj | grep $pattern

2024/01/19 06:07:29 [info] 309#309: connecting to upstream, client: 172.16.40.8, server: mediconfort.com, request: "GET /.well-known/acme-challenge/D4XJ26j8Ev3JUEIvklROC6x_c6qZ9OCc2dDZlNjaIb4 HTTP/1.1", upstream:
"http://10.244.5.158:8089/.well-known/acme-challenge/D4XJ26j8Ev3JUEIvklROC6x_c6qZ9OCc2dDZlNjaIb4", host: "mediconfort.com"

If this is your only "proof":

To me, that alone is inconclusive; The IP shown is likely that IP for all requests.
The log doesn't even show the UserAgent.
It could have been a web crawler that made those request.
It could have been anyone on this forum trying to troubleshoot your problem who made that request.
It could have been from the staging environment.

That said, and presuming it really did from from LE production systems...
We don't see anything related to what happened to the proxied request:

Perhaps that part never completes correctly.

2 Likes

Yes, those are valid points!

Just to describe how I proceeded:

1- I flushed the logs and rebooted the nginx, starting with zero logs and no requests logged in yet
2- meanwhile, with few seconds apart I created a new certificate request for "mediconfort.com"
3- and those are the logs i was tailing

In that small time window, both the acme challenge timestamp and the nginx logs timestamp match pretty closely...

1 Like

The AAAA record got cleaned. The test https://letsdebug.net is passing fine! :muscle:

I will be trying a new certificate request, and will update...

It worked :muscle:, so yes it was indeed the AAAA ipv6 junk record :man_facepalming:!

thank you guys

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.