Traefik not using letsencrypt certificate

geoHeil · March 5, 2023, 4:27pm

I am trying to set up traefik with letsencrypt and DNS validation.
It looks like the letsencrypt certificates are generated - but not used by traefik

traefik  | time="2023-03-05T16:40:15Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
traefik  | time="2023-03-05T16:40:15Z" level=debug msg="Adding certificate for domain(s) *.dataweeder.cloud,dataweeder.cloud"
traefik  | time="2023-03-05T16:40:15Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default

GitHub - geoHeil/aceme-ssl-traefik: Debugging acme ssl traefik contains the details of the traefik configuration

After reading Default certificate from letsencrypt - #6 by jakubhajek - Traefik v2 (latest) - Traefik Labs Community Forum and Traefik TLS Documentation - Traefik I tried to set the default cert store to the ones generated from letsencrypt - but this does not work
aceme-ssl-traefik/docker-compose.yml at master · geoHeil/aceme-ssl-traefik · GitHub

questions:

My domain is: dataweeder.cloud

I ran this command: docker-compose up

It produced this output:

traefik  | time="2023-03-05T16:17:53Z" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default

My web server is (include version): traefik 2.9

The operating system my web server runs on is (include version): docker 4 mac

My hosting provider, if applicable, is: none/local

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

geoHeil · March 5, 2023, 5:16pm

switching to the production version of letsencrypt and waiting longer for DNS propagation seems to fix the issue with letsencrypt certificate generation

rg305 · March 6, 2023, 12:03am

How long did you set the wait time?

geoHeil · March 29, 2023, 7:03pm

2 minutes

Bruce5051 · March 29, 2023, 8:38pm

~~Looks like there is a DNS issue~~ A dataweeder.cloud 5m 127.0.0.1
The IPv4 Address of 127.0.0.1 is the localhost on all machines.

Bruce5051 · March 29, 2023, 8:46pm

And using this online tool https://unboundtest.com/ yields these results https://unboundtest.com/m/A/dataweeder.cloud/5G4ISUKV

Query results for A dataweeder.cloud

Response:
;; opcode: QUERY, status: NOERROR, id: 876
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;dataweeder.cloud.	IN	 A

----- Unbound logs -----
Mar 29 20:45:13 unbound[911775:0] notice: init module 0: validator
Mar 29 20:45:13 unbound[911775:0] notice: init module 1: iterator
Mar 29 20:45:13 unbound[911775:0] info: start of service (unbound 1.16.3).
Mar 29 20:45:13 unbound[911775:0] info: 127.0.0.1 dataweeder.cloud. A IN
Mar 29 20:45:13 unbound[911775:0] info: resolving dataweeder.cloud. A IN
Mar 29 20:45:13 unbound[911775:0] info: priming . IN NS

And with nslookup

$ nslookup -q=a dataweeder.cloud hera.ns.cloudflare.com
Server:         hera.ns.cloudflare.com
Address:        108.162.192.162#53

Name:   dataweeder.cloud
Address: 127.0.0.1

Bruce5051 · March 29, 2023, 9:45pm

Presently I see 2 DNS TXT Records for _acme-challenge.dataweeder.cloud which is fine.
So those 2 have made it through DNS Propagation delays.

https://unboundtest.com/m/TXT/_acme-challenge.dataweeder.cloud/5OROSQK6

Query results for TXT _acme-challenge.dataweeder.cloud

Response:
;; opcode: QUERY, status: NOERROR, id: 12451
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;_acme-challenge.dataweeder.cloud.	IN	 TXT

;; ANSWER SECTION:
_acme-challenge.dataweeder.cloud.	0	IN	TXT	"1ErmhKTTqTQrZbfWNW9_9eOhfFkZsDcmzH2B92Hyf9Y"
_acme-challenge.dataweeder.cloud.	0	IN	TXT	"9v14vg7i8AubNcDv9HRRiJolC3jncUB0zyOoJCub0Ew"

----- Unbound logs -----
Mar 29 21:42:44 unbound[912111:0] notice: init module 0: validator
Mar 29 21:42:44 unbound[912111:0] notice: init module 1: iterator
Mar 29 21:42:44 unbound[912111:0] info: start of service (unbound 1.16.3).
Mar 29 21:42:45 unbound[912111:0] info: 127.0.0.1 _acme-challenge.dataweeder.cloud. TXT IN
Mar 29 21:42:45 unbound[912111:0] info: resolving _acme-challenge.dataweeder.cloud. TXT IN
Mar 29 21:42:45 unbound[912111:0] info: priming . IN NS
Mar 29 21:42:45 unbound[912111:0] info: response for . NS IN

geoHeil · March 29, 2023, 10:26pm

Setting the acme challenges is not an issue - but recently these challenges no longer complete for me:

[*.dataweeder.cloud] The server validated our request"
[dataweeder.cloud] acme: Trying to solve DNS-01"
[dataweeder.cloud] acme: Checking DNS record propagation using [1.1.1.1:53 1.0.0.1:53]"
Wait for propagation [timeout: 2m0s, interval: 2s]"
Delaying 90000000000 rather than validating DNS propagation now." providerName=dns-cloudflare.acme

and am stuck until this timeouts.

What do you mean by DNS A issues? the 127.0.0.1 (for this particular domain) is deliberate to serve nice SSL certs for local development.

Bruce5051 · March 29, 2023, 10:28pm

I misspoke, my bad; sorry!

Bruce5051 · March 29, 2023, 10:30pm

You likely already know this DNS providers who easily integrate with Let's Encrypt DNS validation
And probably have reasons to not change DNS provider also.
But it is just a FYI.

geoHeil · March 29, 2023, 10:32pm

yes - this is why I am using cloudflares DNS servers. And as also written above - this worked well some weeks ago and seems to fail now by not completing the challenge

Bruce5051 · March 29, 2023, 10:47pm

I'll stop mudding the waters now.
Kindly wait for more knowledgeable Let's Encrypt community volunteers to assist.

rg305 · March 29, 2023, 11:32pm

Looks like the process isn't cleaning up after itself:

nslookup -q=txt _acme-challenge.dataweeder.cloud. max.ns.cloudflare.com
Server:  max.ns.cloudflare.com
Address:  173.245.59.132

_acme-challenge.dataweeder.cloud text = "1ErmhKTTqTQrZbfWNW9_9eOhfFkZsDcmzH2B92Hyf9Y"
_acme-challenge.dataweeder.cloud text = "9v14vg7i8AubNcDv9HRRiJolC3jncUB0zyOoJCub0Ew"
_acme-challenge.dataweeder.cloud text = "N9aMUzp1Oo3HME1gJylQ8wPHrAtQtg9WvrhZxiydRZU"
_acme-challenge.dataweeder.cloud text = "TsCGnK9-9yQcqIuxypGY6neQsCfUC1ZDPfLQgbsySCM"
_acme-challenge.dataweeder.cloud text = "WrJsFZJfHG_-zpWbds7RHpLTsDTXBrvFom4_PMz93hU"
_acme-challenge.dataweeder.cloud text = "jZ17VlvsHagFA6X-izO0LreGRhI8HvjlpcBrmKxfzkA"
_acme-challenge.dataweeder.cloud text = "oSkHBhh2_fiFpawKitIM0zwL6aBjgTw6c3B3Ucshjv8"
_acme-challenge.dataweeder.cloud text = "ymSSkww3HngJ0aFViPNwchZmb4SJ5HR58cxzxrYF-Wg"

You might want to remove all those before continuing [your tests].

MikeMcQ · March 30, 2023, 2:33am

Have you tried increasing the timeouts beyond 2m0s?

I ran across an FAQ for Cloudflare that hinted that 5m is tolerable limit. I am not a Cloudflare expert but worth checking if that resolves it. Could check with Cloudflare support or on their forums for an official response.

And, unless you are doing high volume of certs a 2s retry seems unnecessarily short.

geoHeil · March 30, 2023, 4:54am

I did this before but cleaning up does not help

geoHeil · March 30, 2023, 4:55am

I am using the: --certificatesResolvers.dns-cloudflare.acme.dnsChallenge.delayBeforeCheck=90

and then run the default check which retries for 2 minutes every 2 seconds.

this is only a single wildcard DNS

How could I reduce the 2s for longer wait? But I would expect the 90 seconds timeout to work

geoHeil · March 30, 2023, 5:01am

indeed I just figured it out some UDP traffic by my firewall was blocked - solved finally.

system · April 29, 2023, 5:02am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.