Certificates no longer authenticating

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: julapalli.club

I ran this command: DNS challenge with my domain and wildcard subdomains behind Traefik 2 reverse proxy running on Docker.

It produced this output: I had my domain and multiple subdomains all running with certificates issued by Let’s Encrypt, through Docker containers behind Traefik reverse proxy. The DNS is hosted by Cloudflare. Every subdomain was working except for my Nextcloud subdomain, which would not launch (404 Not Found) when issued a certificate. I couldn’t figure out why, and so I decided to rebuild all my containers again and try to get new certificates issued. The problem now is I don’t seem to be getting any new certificates issued. The Docker log for one container shows this statement "No ACME certificate generation required for domains [“julapalli.club”, followed by a TLS handshake error: bad certificate. I am now in over my head and could use some help.

My web server is (include version): Running on Docker

The operating system my web server runs on is (include version): Arch Linux

My hosting provider, if applicable, is: Cloudflare

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): Cloudflare

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

2

This was... unwise.

However, it looks like you have a valid wildcard certificate: crt.sh | julapalli.club

You should let us see that error, we need a link to a subdomain that actually answers.

3

Perhaps unwise, indeed, in hindsight. I know enough in SSL certificates to get myself into trouble. So I can learn from this, why specifically was it unwise? Does it have to do with DNS propagation?

I think the bad certificate error was because I had a delay in the DNS check in the docker-compose file, which allowed the Traefik container to issued its own self-signed certificate. I took out that delay, and here is the latest debug log:

2020-05-25T12:46:32.447336035Z time=“2020-05-25T12:46:32Z” level=info msg=“Starting provider aggregator.ProviderAggregator {}”
2020-05-25T12:46:32.447342489Z time=“2020-05-25T12:46:32Z” level=debug msg=“Start TCP Server” entryPointName=http
2020-05-25T12:46:32.447373558Z time=“2020-05-25T12:46:32Z” level=debug msg=“Start TCP Server” entryPointName=https
2020-05-25T12:46:32.447416174Z time=“2020-05-25T12:46:32Z” level=info msg=“Starting provider file.Provider {“directory”:"/rules",“watch”:true}"
2020-05-25T12:46:32.447496596Z time=“2020-05-25T12:46:32Z” level=debug msg=“Start TCP Server” entryPointName=traefik
2020-05-25T12:46:32.447576569Z time=“2020-05-25T12:46:32Z” level=info msg=“Starting provider traefik.Provider {}"
2020-05-25T12:46:32.447607305Z time=“2020-05-25T12:46:32Z” level=info msg=“Starting provider acme.Provider {“email":"vrjula@protonmail.com”,“caServer”:“https://acme-v02.api.letsencrypt.org/directory",“storage”:"/acme.json",“keyType”:“RSA4096”,“dnsChallenge”:{“provider”:“cloudflare”,“resolvers”:[“1.1.1.1:53”,“1.0.0.1:53”]},“ResolverName”:“dns-cloudflare”,“store”:{},“ChallengeStore”:{}}"
2020-05-25T12:46:32.447619130Z time=“2020-05-25T12:46:32Z” level=info msg=“Testing certificate renew…” providerName=dns-cloudflare.acme
2020-05-25T12:46:32.447638712Z time=“2020-05-25T12:46:32Z” level=info msg=“Starting provider docker.Provider {“watch”:true,“endpoint”:“unix:///var/run/docker.sock”,“defaultRule”:“Host({{ index .Labels \\\"com.docker.compose.service\\\" }}.julapalli.club)”,“network”:“t2_proxy”,“swarmModeRefreshSeconds”:15000000000}"
2020-05-25T12:46:32.448208972Z time=“2020-05-25T12:46:32Z” level=debug msg=“Configuration received from provider file: {“http”:{},“tcp”:{},“udp”:{},“tls”:{}}” providerName=file
2020-05-25T12:46:32.448223770Z time=“2020-05-25T12:46:32Z” level=debug msg=“Configuration received from provider internal: {“http”:{“services”:{“api”:{},“dashboard”:{},“noop”:{}},“models”:{“https”:{“tls”:{“certResolver”:“dns-cloudflare”}}}},“tcp”:{},“tls”:{}}” providerName=internal
2020-05-25T12:46:32.448248392Z time=“2020-05-25T12:46:32Z” level=debug msg=“Configuration received from provider dns-cloudflare.acme: {“http”:{},“tls”:{}}” providerName=dns-cloudflare.acme
2020-05-25T12:46:32.448477133Z time=“2020-05-25T12:46:32Z” level=debug msg=“No default certificate, generating one”
2020-05-25T12:46:32.452328180Z time=“2020-05-25T12:46:32Z” level=debug msg=“Provider connection established with docker 19.03.9-ce (API 1.40)” providerName=docker
2020-05-25T12:46:32.453937576Z time=“2020-05-25T12:46:32Z” level=debug msg=“Configuration received from provider docker: {“http”:{“routers”:{“http-catchall”:{“entryPoints”:[“http”],“middlewares”:[“redirect-to-https”],“service”:“traefik-docker”,“rule”:“HostRegexp({host:.+})”},“traefik-rtr”:{“entryPoints”:[“https”],“service”:“api@internal”,“rule”:“Host(traefik.julapalli.club)”,“tls”:{“certResolver”:“dns-cloudflare”,“domains”:[{“main”:“julapalli.club”,“sans”:[”.julapalli.club”]}]}}},“services”:{“traefik-docker”:{“loadBalancer”:{“servers”:[{“url”:“http://192.168.90.254:80”}],“passHostHeader”:true}}},“middlewares”:{“redirect-to-https”:{“redirectScheme”:{“scheme”:“https”}}}},“tcp”:{},“udp”:{}}” providerName=docker
2020-05-25T12:46:32.581832339Z time=“2020-05-25T12:46:32Z” level=debug msg=“No default certificate, generating one”
2020-05-25T12:46:32.787568038Z time=“2020-05-25T12:46:32Z” level=debug msg=“No default certificate, generating one”
2020-05-25T12:46:33.009320253Z time=“2020-05-25T12:46:33Z” level=debug msg=“Creating middleware” entryPointName=http routerName=http-catchall@docker serviceName=traefik-docker middlewareName=pipelining middlewareType=Pipelining
2020-05-25T12:46:33.009336922Z time=“2020-05-25T12:46:33Z” level=debug msg=“Creating load-balancer” entryPointName=http routerName=http-catchall@docker serviceName=traefik-docker
2020-05-25T12:46:33.009342767Z time=“2020-05-25T12:46:33Z” level=debug msg=“Creating server 0 http://192.168.90.254:80” routerName=http-catchall@docker serverName=0 serviceName=traefik-docker entryPointName=http
2020-05-25T12:46:33.009347696Z time=“2020-05-25T12:46:33Z” level=debug msg=“Added outgoing tracing middleware traefik-docker” entryPointName=http routerName=http-catchall@docker middlewareType=TracingForwarder middlewareName=tracing
2020-05-25T12:46:33.009350660Z time=“2020-05-25T12:46:33Z” level=debug msg=“Creating middleware” routerName=http-catchall@docker entryPointName=http middlewareName=redirect-to-https@docker middlewareType=RedirectScheme
2020-05-25T12:46:33.009353354Z time=“2020-05-25T12:46:33Z” level=debug msg="Setting up redirection to https " routerName=http-catchall@docker entryPointName=http middlewareName=redirect-to-https@docker middlewareType=RedirectScheme
2020-05-25T12:46:33.009358295Z time=“2020-05-25T12:46:33Z” level=debug msg=“Adding tracing to middleware” routerName=http-catchall@docker entryPointName=http middlewareName=redirect-to-https@docker
2020-05-25T12:46:33.009360845Z time=“2020-05-25T12:46:33Z” level=debug msg=“Creating middleware” middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=http
2020-05-25T12:46:33.009363355Z time=“2020-05-25T12:46:33Z” level=debug msg=“Added outgoing tracing middleware api@internal” middlewareName=tracing middlewareType=TracingForwarder entryPointName=https routerName=traefik-rtr@docker
2020-05-25T12:46:33.009509664Z time=“2020-05-25T12:46:33Z” level=debug msg=“Creating middleware” middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=https
2020-05-25T12:46:33.009552614Z time=“2020-05-25T12:46:33Z” level=debug msg=“No default certificate, generating one”
2020-05-25T12:46:33.082579576Z time=“2020-05-25T12:46:33Z” level=debug msg="Looking for provided certificate(s) to validate [“julapalli.club” ".julapalli.club”]…" providerName=dns-cloudflare.acme
2020-05-25T12:46:33.082589566Z time=“2020-05-25T12:46:33Z” level=debug msg="Domains [“julapalli.club” ".julapalli.club”] need ACME certificates generation for domains "julapalli.club,.julapalli.club”." providerName=dns-cloudflare.acme
2020-05-25T12:46:33.082593107Z time=“2020-05-25T12:46:33Z” level=debug msg=“Loading ACME certificates [julapalli.club .julapalli.club]…" providerName=dns-cloudflare.acme
2020-05-25T12:46:33.082595779Z time=“2020-05-25T12:46:33Z” level=debug msg=“Building ACME client…” providerName=dns-cloudflare.acme
2020-05-25T12:46:33.082598802Z time=“2020-05-25T12:46:33Z” level=debug msg=“https://acme-v02.api.letsencrypt.org/directory” providerName=dns-cloudflare.acme
2020-05-25T12:46:33.324524698Z time=“2020-05-25T12:46:33Z” level=debug msg=“Using DNS Challenge provider: cloudflare” providerName=dns-cloudflare.acme
2020-05-25T12:46:33.324611273Z time=“2020-05-25T12:46:33Z” level=debug msg=“legolog: [INFO] [julapalli.club, .julapalli.club] acme: Obtaining bundled SAN certificate"
2020-05-25T12:46:33.747230930Z time=“2020-05-25T12:46:33Z” level=debug msg="legolog: [INFO] [.julapalli.club] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4798080481”
2020-05-25T12:46:33.747278407Z time=“2020-05-25T12:46:33Z” level=debug msg=“legolog: [INFO] [julapalli.club] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4798080482”
2020-05-25T12:46:33.747289888Z time=“2020-05-25T12:46:33Z” level=debug msg="legolog: [INFO] [.julapalli.club] acme: use dns-01 solver”
2020-05-25T12:46:33.747296726Z time=“2020-05-25T12:46:33Z” level=debug msg=“legolog: [INFO] [julapalli.club] acme: Could not find solver for: tls-alpn-01”
2020-05-25T12:46:33.747303653Z time=“2020-05-25T12:46:33Z” level=debug msg=“legolog: [INFO] [julapalli.club] acme: Could not find solver for: http-01”
2020-05-25T12:46:33.747310315Z time=“2020-05-25T12:46:33Z” level=debug msg=“legolog: [INFO] [julapalli.club] acme: use dns-01 solver”
2020-05-25T12:46:33.747330045Z time=“2020-05-25T12:46:33Z” level=debug msg=“legolog: [INFO] [.julapalli.club] acme: Preparing to solve DNS-01"
2020-05-25T12:46:34.307698353Z time=“2020-05-25T12:46:34Z” level=debug msg=“legolog: [INFO] [julapalli.club] acme: Preparing to solve DNS-01”
2020-05-25T12:46:34.567645604Z time=“2020-05-25T12:46:34Z” level=debug msg="legolog: [INFO] [.julapalli.club] acme: Cleaning DNS-01 challenge”
2020-05-25T12:46:34.567685371Z time=“2020-05-25T12:46:34Z” level=debug msg=“legolog: [WARN] [.julapalli.club] acme: cleaning up failed: cloudflare: unknown record ID for ‘_acme-challenge.julapalli.club.’ "
2020-05-25T12:46:34.567694691Z time=“2020-05-25T12:46:34Z” level=debug msg=“legolog: [INFO] [julapalli.club] acme: Cleaning DNS-01 challenge”
2020-05-25T12:46:34.567701405Z time=“2020-05-25T12:46:34Z” level=debug msg="legolog: [WARN] [julapalli.club] acme: cleaning up failed: cloudflare: unknown record ID for ‘_acme-challenge.julapalli.club.’ "
2020-05-25T12:46:34.652494393Z time=“2020-05-25T12:46:34Z” level=debug msg=“legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4798080481”
2020-05-25T12:46:34.829311954Z time=“2020-05-25T12:46:34Z” level=debug msg=“legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/4798080482”
2020-05-25T12:46:34.936126641Z time=“2020-05-25T12:46:34Z” level=error msg="Unable to obtain ACME certificate for domains "julapalli.club,.julapalli.club” : unable to generate a certificate for the domains [julapalli.club .julapalli.club]: error: one or more domains had a problem:\n[.julapalli.club] [*.julapalli.club] acme: error presenting token: cloudflare: failed to create TXT record: error from makeRequest: HTTP status 400: content “{\“result\”:null,\“success\”:false,\“errors\”:[{\“code\”:81057,\“message\”:\“The record already exists.\”}],\“messages\”:}”\n[julapalli.club] [julapalli.club] acme: error presenting token: cloudflare: failed to create TXT record: error from makeRequest: HTTP status 400: content “{\“result\”:null,\“success\”:false,\“errors\”:[{\“code\”:81057,\“message\”:\“The record already exists.\”}],\“messages\”:}”\n" providerName=dns-cloudflare.acme

4

Here’s an updated log, after I cleared some existing challenge TXT records from Cloudflare:

020-05-25T16:58:31.345578936Z time=“2020-05-25T16:58:31Z” level=error msg=“Unable to obtain ACME certificate for domains “julapalli.club,*.julapalli.club” : unable to generate a certificate for the domains [julapalli.club .julapalli.club]: error: one or more domains had a problem:\n[.julapalli.club] time limit exceeded: last error: NS carlos.ns.cloudflare.com. did not return the expected TXT record [fqdn: julapalli.club., value: SpXPuxxawVtIfwCSXlFcN0Gc1aZeriBMlzuZdFFOD-s]: v=spf1 include:spf.efwd.registrar-servers.com ~all\n[julapalli.club] time limit exceeded: last error: NS carlos.ns.cloudflare.com. did not return the expected TXT record [fqdn: julapalli.club., value: 9s4oV0GRg6giEW-C4Gd_CePH5UDld2Pm9oN08MCKI5k]: v=spf1 include:spf.efwd.registrar-servers.com ~all\n” providerName=dns-cloudflare.acme

5

Well, your domain has a wildcard CNAME record. And your ACME client’s internal self-test is seeing that instead of the TXT records it’s creating.

The CNAME record has a TTL of 5 minutes; if you wait a little while – which you have, if that error is actually 4 hours old! – and try again it might work.

6

Thanks, I did a dig command and confirmed that the TTL on the A and CNAME records for julapalli.club and *. julapalli.club, respectively, is 5 minutes. But I keep getting a long series of “acme: waiting for DNS record propagation” when I run the challenge, followed by a timeout and the error seen on my last published log above. Attached is my DNS records at Cloudflare:

image
image1028×366 42.5 KB

Not sure what to do now.

7

Can you confirm that the DNS records are set correctly at Cloudflare while your ACME client is running?

(Cloudflare’s audit log should also have historical records.)

Can you run dig _acme-challenge.julapalli.club txt in a terminal to see what it’s getting?

8

I solved it; this link helped:

The issue was at Cloudflare; on the DNS challenge it kept looking at the CNAME record instead of the generated TXT record. I had to remove the CNAME record first, run the challenge, then add the CNAME record back, for the wildcard.

Thanks for pointing me in the right direction!

9

Depending on what’s going on, that’s a bug in the ACME client or, possibly, Cloudflare’s API or DNS service. You weren’t doing anything wrong.

10

because getting new certificates is unlikely to solve any problem, and can create serious new ones if you don't backup the old certificates.

11

Do you mean in terms of the rate limits? I was using the staging environment first, for the most part.

12

yes, yes. and some more characters.