X509 error while generating Let's Encrypt certificate with Traefik

I keep getting an x509 certificate error while traefik is trying to generate a Let's Encrypt certificate. Currently set to Let's Encrypt staging, but I've tried production as well.

Config: https://pastebin.com/2Ph1zfun
Logs: https://pastebin.com/KWQPSyqB

See log lines starting at 55:

traefik |time="2021-06-20T23:07:22Z" level=debug msg="Building ACME client..." providerName=letsencrypt.acme
traefik |time="2021-06-20T23:07:22Z" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme
traefik |time="2021-06-20T23:07:22Z" level=error msg="Unable to obtain ACME certificate for domains "traefik.[redacted].ca": cannot get ACME client get directory at 'https://acme-staging-v02.api.letsencrypt.org/directory': Get "https://acme-staging-v02.api.letsencrypt.org/directory": x509: certificate signed by unknown authority" routerName=traefik@docker providerName=letsencrypt.acme rule="Host( traefik.[redacted].ca )"

and at the end of the log, when traefik does an update check (I think):

traefik |time="2021-06-20T23:30:53Z" level=warning msg="Error checking new version: Get "https://update.traefik.io/repos/traefik/traefik/releases": x509: certificate signed by unknown authority"

I did actually have this working a few days ago, but unfortunately I forgot to set it to staging while I was getting all of this configured, so I'm waiting for that to timeout. In the mean time I'm not sure what I changed that messed this all up.

I also tried switching from HTTP challenge to DNS, and I still have the same issue. I've also tried the same config on a different server and I have the same issue.

1 Like

Welcome to the Let's Encrypt Community :slightly_smiling_face:

It appears to me that the certificate trust store being used by traefik might be lacking the necessary CA root certificates needed to verify the certificate chains being sent by https://acme-staging-v02.api.letsencrypt.org/directory and https://update.traefik.io/repos/traefik/traefik/releases.

2 Likes

Check that your trust store has "Baltimore CyberTrust Root", which is the root CA for cloudflare, which is the service hosting update.traefik,io. Also if you are going through a network proxy ensure it's not something that uses it's own SSL termination (otherwise you'd need to trust that internal CA root).

2 Likes

Hmm. As I said, I had this working a few days ago, so it seems unlikely but it also seems to make sense. I know very little about how certificates in general but also chain verification work so I'll have to figure out how to do that. I did as part of my troubleshooting make sure the ca-certificates package is up-to-date.. thanks for the reply and I'll update if that leads me anywhere.

1 Like

This is interesting:

************@sun:/opt/traefik$ openssl s_client -showcerts -connect update.traefik.io:443 | grep -i depth
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = sni.cloudflaressl.com
verify return:1
^C
************@sun:/opt/traefik$ awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | egrep -i -- "(baltimore|cloudflare)"
subject=C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root

so it looks to me like the root cert should be verifiable/trusted but not the others in the chain -- do they need to be?

1 Like

Looks like the root ca appears in /etc/ssl/certs/ca-certificates.crt but not the others in the chain. Do they need to?

1 Like

The last two (depths 0 and1) should be being sent by https://update.traefik.io/repos/traefik/traefik/releases while the first one (depth 2) should be in the trust store, which it appears to be!?

I tested https://update.traefik.io/repos/traefik/traefik/releases with SSL Checker - Test Certificate and Installation to confirm my assumptions.

None of this addresses the root needed (DST Root CA X3) for https://acme-staging-v02.api.letsencrypt.org/directory.

Any thoughts here @webprofusion?

Maybe traefik is lacking permission to access the CA file?

1 Like

well, traefik is running in a docker container with limited access to the filesystem, so I'm not sure how it would access the CA file -- if that were the issue I think everyone trying to run Traefik in docker would have the same issue, or I'm misunderstanding how docker works. I am running docker with root privileges.

You need DST Root CA X3 for that.

Just confirmed, I have that one as well.
By the way, thanks so much for taking the time to reply. This has been driving me crazy for a while and it's at least a little comforting that it's not immediately obvious what the issue is!

2 Likes

Maybe this?

2 Likes

OooooooOoooOoohhhhh. Of course. I re-mapped /etc as well, not realizing that it was being used to actually access various resources in my host machine's /etc. And that lines up with around the time I started having issues, as at the time I was trying to map a directory to store acme.json.

Okay, I'm going to try re-mapping that location and see if it clears up. Fingers crossed.

1 Like

Here's to hoping:

:smiley:

:crossed_fingers:

1 Like

It worked! Thank you so much! :partying_face:

4 Likes

Really glad to hear it!

:partying_face:

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.