LE seems to be doing some weird DNS caching

[roffe]

Please fill out the fields below so we can help you better.

Intro:
Several other domains i have tried in the same setup works fine. i get certs issued. However for the 3 below. LE keeps connecting to the OLD ip even thought we updated the DNS records BEFORE we tried to request a certificate.

We have verified also that the requests are hitting the OLD backend even though DNS was updated hours ago,

My domain is:
hptest-app.gel.camp
hptest-communication.gel.camp
hptest-communication-fe.gel.camp

I ran this command:
kube-lego in kubernetes

jetstack/kube-lego

It produced this output:
time=“2017-02-02T12:29:49Z” level=warning msg=“authorization failed after 1m0s: reachabily test failed: wrong status code ‘403’” context=acme domain=hptest-app.gel.camp
time=“2017-02-02T12:31:23Z” level=warning msg=“authorization failed after 1m0s: reachabily test failed: wrong status code ‘403’” context=acme domain=hptest-communication.gel.camp
time=“2017-02-02T12:32:30Z” level=warning msg=“authorization failed after 1m0s: reachabily test failed: wrong status code ‘403’” context=acme domain=hptest-communication-fe.gel.c

My operating system is (include version):
Kubernetes 1.5.2

My web server is (include version):
Nginx 1.11.*

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

[roffe]

This is the OLD backend on 185.60.160.122 that the domain names not are pointing to that is getting hit by LE.

Why does LE insist on keeping connecting to the old ip’s?

Last login: Thu Feb 2 13:43:03 on ttys013
for DOMAIN in hptest-app.gel.camp hptest-communication.gel.camp hptest-communication-fe.gel.camp; do dig $DOMAIN; done
➜ ~ for DOMAIN in hptest-app.gel.camp hptest-communication.gel.camp hptest-communication-fe.gel.camp; do dig $DOMAIN; done

; <<>> DiG 9.8.3-P1 <<>> hptest-app.gel.camp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21640
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;hptest-app.gel.camp. IN A

;; ANSWER SECTION:
hptest-app.gel.camp. 59 IN A 185.60.160.170

;; Query time: 29 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Thu Feb 2 13:47:38 2017
;; MSG SIZE rcvd: 53

; <<>> DiG 9.8.3-P1 <<>> hptest-communication.gel.camp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58791
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;hptest-communication.gel.camp. IN A

;; ANSWER SECTION:
hptest-communication.gel.camp. 59 IN A 185.60.160.170

;; Query time: 33 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Thu Feb 2 13:47:38 2017
;; MSG SIZE rcvd: 63

; <<>> DiG 9.8.3-P1 <<>> hptest-communication-fe.gel.camp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9510
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;hptest-communication-fe.gel.camp. IN A

;; ANSWER SECTION:
hptest-communication-fe.gel.camp. 59 IN A 185.60.160.170

;; Query time: 27 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Thu Feb 2 13:47:38 2017
;; MSG SIZE rcvd: 66

[serverco]

Let’s Encrypt doesn’t (as far as I’m aware) cache the DNS. It checks with the authoritative DNS to get the latest IP address.

If I try reaching hptest-communication-fe.gel.camp/.well-known/came-challenge/_selftest I also get a 404 (and my DNS is saying the IP is 185.60.160.170 )

Are you certain that it’s just the wrong DNS IP ? and not some routing set up in the AWS you are using ?

[roffe]

@serverco

hptest-communication-fe.gel.camp/.well-known/came-challenge/_selftest

spelling error in ur link m8

➜ ~ curl hptest-communication-fe.gel.camp/.well-known/acme-challenge/_selftest
rwAsGzOWcreotlLP%

[serverco]

acme, came … almost the same

OK - that get’s a response - and presumably hitting the correct (new not old) servers on your end.

can you get the full log from kube-lego ? it’s not a client I’ve used.

[roffe]

I appoligize.

This turns out the be our own fault as we are rolling own internal DNS servers. For some reason kube-lego connects to itself on the hostname before sending the request to LE ( which is good ) but i didn’t know about and to us it looked like it was LE connecting.

Please close issue!

[serverco]

No problem - glad you got it all sorted

[roffe]

If someone using kube-lego stumbles in here

To clarify. We had our own internal DNS records for the hostnames in question pointing to internal ip’s and that’s why kube-lego kept saying 403 and establishing connections to old backend

[MikeRob]

As an aside, Macs today use mDNSResponder daemons in such a way that they can cache-and-hold DNS entries without realizing that they are out-of-date.   Turning wireless off-then-on and/or unplugging network cables might cause the daemon to refresh; otherwise send it a SIGHUP signal with:

sudo killall -HUP mDNSResponder

Perhaps the most irritating “feature” of this particular daemon is that it will also cache NXDOMAIN responses.   If you “briefly had something wrong with an upstream DNS server,” the daemon will remember that the domain doesn’t exist, and it will tell you this again and again without looking again to see if it now does exist (as most other operating systems would have done).   You might not be expecting this sort of DNS-caching behavior to be occurring directly on the client machine, but it does, and it’s not quite the behavior you might have intuitively expected based on your experience with other operating systems.

Maybe not directly-relevant in this particular case, but annoying and unexpected behavior nonetheless.

(I run a DNS server on the other side of an OpenVPN link, and if I neglect to open the VPN tunnel before attempting to hit a particular web site that is on the other side of it, I have to go through these calisthenics because the responder, having failed to contact the DNS server that did know about the domain, will now remember that “it’s not there.”)