X509 error while generating Let's Encrypt certificate with Traefik

[machone]

I keep getting an x509 certificate error while traefik is trying to generate a Let's Encrypt certificate. Currently set to Let's Encrypt staging, but I've tried production as well.

Config: https://pastebin.com/2Ph1zfun
Logs: https://pastebin.com/KWQPSyqB

See log lines starting at 55:

traefik |time="2021-06-20T23:07:22Z" level=debug msg="Building ACME client..." providerName=letsencrypt.acme
traefik |time="2021-06-20T23:07:22Z" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme
traefik |time="2021-06-20T23:07:22Z" level=error msg="Unable to obtain ACME certificate for domains "traefik.[redacted].ca": cannot get ACME client get directory at 'https://acme-staging-v02.api.letsencrypt.org/directory': Get "https://acme-staging-v02.api.letsencrypt.org/directory": x509: certificate signed by unknown authority" routerName=traefik@docker providerName=letsencrypt.acme rule="Host( traefik.[redacted].ca )"

and at the end of the log, when traefik does an update check (I think):

traefik |time="2021-06-20T23:30:53Z" level=warning msg="Error checking new version: Get "https://update.traefik.io/repos/traefik/traefik/releases": x509: certificate signed by unknown authority"

I did actually have this working a few days ago, but unfortunately I forgot to set it to staging while I was getting all of this configured, so I'm waiting for that to timeout. In the mean time I'm not sure what I changed that messed this all up.

I also tried switching from HTTP challenge to DNS, and I still have the same issue. I've also tried the same config on a different server and I have the same issue.

[griffin]

Welcome to the Let's Encrypt Community

It appears to me that the certificate trust store being used by traefik might be lacking the necessary CA root certificates needed to verify the certificate chains being sent by https://acme-staging-v02.api.letsencrypt.org/directory and https://update.traefik.io/repos/traefik/traefik/releases.

[webprofusion]

Check that your trust store has "Baltimore CyberTrust Root", which is the root CA for cloudflare, which is the service hosting update.traefik,io. Also if you are going through a network proxy ensure it's not something that uses it's own SSL termination (otherwise you'd need to trust that internal CA root).

[machone]

Hmm. As I said, I had this working a few days ago, so it seems unlikely but it also seems to make sense. I know very little about how certificates in general but also chain verification work so I'll have to figure out how to do that. I did as part of my troubleshooting make sure the ca-certificates package is up-to-date.. thanks for the reply and I'll update if that leads me anywhere.

[machone]

This is interesting:

************@sun:/opt/traefik$ openssl s_client -showcerts -connect update.traefik.io:443 | grep -i depth
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = sni.cloudflaressl.com
verify return:1
^C
************@sun:/opt/traefik$ awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | egrep -i -- "(baltimore|cloudflare)"
subject=C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root

so it looks to me like the root cert should be verifiable/trusted but not the others in the chain -- do they need to be?

[machone]

Looks like the root ca appears in /etc/ssl/certs/ca-certificates.crt but not the others in the chain. Do they need to?

[griffin]

The last two (depths 0 and1) should be being sent by https://update.traefik.io/repos/traefik/traefik/releases while the first one (depth 2) should be in the trust store, which it appears to be!?

I tested https://update.traefik.io/repos/traefik/traefik/releases with SSL Checker - Test Certificate and Installation to confirm my assumptions.

None of this addresses the root needed (DST Root CA X3) for https://acme-staging-v02.api.letsencrypt.org/directory.

Any thoughts here @webprofusion?

Maybe traefik is lacking permission to access the CA file?

[machone]

@griffin

Maybe traefik is lacking permission to access the CA file?

well, traefik is running in a docker container with limited access to the filesystem, so I'm not sure how it would access the CA file -- if that were the issue I think everyone trying to run Traefik in docker would have the same issue, or I'm misunderstanding how docker works. I am running docker with root privileges.

You need DST Root CA X3 for that.

Just confirmed, I have that one as well.
By the way, thanks so much for taking the time to reply. This has been driving me crazy for a while and it's at least a little comforting that it's not immediately obvious what the issue is!

[griffin]

Maybe this?

Traefik can't connect to lets encrypt directory

[machone]

OooooooOoooOoohhhhh. Of course. I re-mapped /etc as well, not realizing that it was being used to actually access various resources in my host machine's /etc. And that lines up with around the time I started having issues, as at the time I was trying to map a directory to store acme.json.

Okay, I'm going to try re-mapping that location and see if it clears up. Fingers crossed.

[griffin]

Here's to hoping:

[machone]

It worked! Thank you so much!

[griffin]

Really glad to hear it!