Http-01 Challenge with central validation server

Help
1

Having a problem issuing a certificate via the http-01 challenge with a central validation server. We host about 50 sites across a number of load balanced servers. We don’t have direct control over most of our clients’ domains so we have them use a validation subdomain with an A record which then responds to the challenge.

Everything has been working great for some time now, but I have one client that I’ve been trying to issue a cert for for the last two days that doesn’t seem to be working, and it looks like it’s because the verification servers on LE’s end don’t seem to be picking up the DNS updates.

So, if I run:

dig domain-validation.CLIENT-DOMAIN.com -A

I see:

;; ANSWER SECTION:
domain-validation.CLIENT-DOMAIN.com. 60 IN	A	XX.XX.XX.XX

Where XX.XX.XX.XX is the correct IP address and was set a few days ago.

I run the command

    sudo certbot certonly --manual \
    --manual-auth-hook /etc/letsencrypt/hooks/oK2el1w3t99aLal1KRwnr81gi \
    --preferred-challenges http --manual-public-ip-logging-ok \
    -w /srv/www/XXX -d www.CLIENT-DOMAN.com \
    -d CLIENT-DOMAN.com -n

And I get:

Failed authorization procedure.
www.CLIENT-DOMAIN.com 
(http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://domain-validation.CLIENT-DOMAIN.com/fE76oevmsIqhoum2a--U-KNXw-X3X8rAxeK6eiKFz1o 
[2607:f1c0:1000:60a7:2b3b:79ac:2607:700e]: "<!DOCTYPE html>\n<html>\n    <head>\n        <meta charset=\"utf-8\">\n        <style type=\"text/css\">\n            html, body, #partne",
CLIENT-DOMAIN.com 
(http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://domain-validation.CLIENT-DOMAIN.com/-9r5c3Gf0tIy901_ngEDlzpZRjGYPeG_jahzGif5WfE 
[2607:f1c0:1000:60a7:2b3b:79ac:2607:700e]: "<!DOCTYPE html>\n<html>\n    <head>\n        <meta charset=\"utf-8\">\n        <style type=\"text/css\">\n            html, body, #partne"

So, here’s the weird thing: That IPv6 address is not the address for my server, nor is that the right HTML that would be seen from any of my machines. I traced that IPv6 address to a non-functional domain.

I can successfully load http://domain-validation.CLIENT-DOMAIN.com/-9r5c3Gf0tIy901_ngEDlzpZRjGYPeG_jahzGif5WfE on my local machine, my test machines, my C2 machine, all my remote servers.

I’m at a loss here. At first I thought it was because the DNS hadn’t propagated yet, but it seems like over 36 hours is plenty of time. Anyone have any ideas?

edit: readability

2

I realize that I may be needlessly vague - OPSEC never dies I suppose. Here’s the actual domains and IPs:

dig domain-validation.cavalierdaily.com -t A


; <<>> DiG 9.10.3-P4-Ubuntu <<>> domain-validation.cavalierdaily.com -t A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32679
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;domain-validation.cavalierdaily.com. IN	A

;; ANSWER SECTION:
domain-validation.cavalierdaily.com. 60	IN A	50.16.124.219

;; Query time: 44 msec
;; SERVER: 10.0.0.2#53(10.0.0.2)
;; WHEN: Wed Apr 10 15:53:39 UTC 2019
;; MSG SIZE  rcvd: 80

Then running:

sudo certbot certonly --manual 
--manual-auth-hook /etc/letsencrypt/hooks/oK2el1w3t99aLal1KRwnr81gi
--preferred-challenges http --manual-public-ip-logging-ok  
-w /srv/www/cav -d www.cavalierdaily.com -d cavalierdaily.com -n

Results in:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for cavalierdaily.com
http-01 challenge for www.cavalierdaily.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. cavalierdaily.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://domain-validation.cavalierdaily.com/haRRYuX30hF8cCIJ3ERTV6KLcbV_F67pfJTLJzGScrY [2607:f1c0:1000:60a7:2b3b:79ac:2607:700e]: "<!DOCTYPE html>\n<html>\n    <head>\n        <meta charset=\"utf-8\">\n        <style type=\"text/css\">\n            html, body, #partne", www.cavalierdaily.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://domain-validation.cavalierdaily.com/KmXed4Ltuj_fG_cJ7KdjyERzwyXBIkqlzpMaQ7YR78U [2607:f1c0:1000:60a7:2b3b:79ac:2607:700e]: "<!DOCTYPE html>\n<html>\n    <head>\n        <meta charset=\"utf-8\">\n        <style type=\"text/css\">\n            html, body, #partne"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: cavalierdaily.com
   Type:   unauthorized
   Detail: Invalid response from
   http://domain-validation.cavalierdaily.com/haRRYuX30hF8cCIJ3ERTV6KLcbV_F67pfJTLJzGScrY
   [2607:f1c0:1000:60a7:2b3b:79ac:2607:700e]: "<!DOCTYPE
   html>\n<html>\n    <head>\n        <meta charset=\"utf-8\">\n
   <style type=\"text/css\">\n            html, body, #partne"

   Domain: www.cavalierdaily.com
   Type:   unauthorized
   Detail: Invalid response from
   http://domain-validation.cavalierdaily.com/KmXed4Ltuj_fG_cJ7KdjyERzwyXBIkqlzpMaQ7YR78U
   [2607:f1c0:1000:60a7:2b3b:79ac:2607:700e]: "<!DOCTYPE
   html>\n<html>\n    <head>\n        <meta charset=\"utf-8\">\n
   <style type=\"text/css\">\n            html, body, #partne"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Again, not my IP, nor my HTML. And I can verify that both those verification URLs worked (until I removed the data).

3

You have an AAAA record with the IPv6 address:

dig domain-validation.cavalierdaily.com aaaa

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> domain-validation.cavalierdaily.com aaaa
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32981
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;domain-validation.cavalierdaily.com. IN	AAAA

;; ANSWER SECTION:
domain-validation.cavalierdaily.com. 3600 IN AAAA 2607:f1c0:1000:60a7:2b3b:79ac:2607:700e

;; Query time: 49 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Apr 10 17:04:47 IST 2019
;; MSG SIZE  rcvd: 92

If that’s wrong, I guess you should remove it? If you have both A (IPv4) and AAAA (IPv6) records, Let’s Encrypt will use the IPv6 for validation.

1 Like
4

Hi @mikej

I see that ip address ( https://check-your-website.server-daten.de/?q=domain-validation.cavalierdaily.com ):

Host T IP-Address is auth. ∑ Queries ∑ Timeout
domain-validation.cavalierdaily.com A 50.16.124.219 yes 1 0
AAAA 2607:f1c0:1000:60a7:2b3b:79ac:2607:700e yes
www.domain-validation.cavalierdaily.com A 192.195.77.137 yes 1 0
AAAA 2607:f1c0:1000:60a7:2b3b:79ac:2607:700e yes

So Letsencrypt checks that address.

1 Like
5

Son of a bee-sting. Didn’t even think if that - since we have clients handle their own DNS I didn’t even consider this one.

Thanks for pointing that out, I’ve been banging my head against the wall for two days.

1 Like
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.