ACME / Traefik / Versio, no new certificates generated (previously working)

Help
1

My domain is:
tevk.nl (and various subdomains *.tevk.nl)

I ran this command:
I use Traefik v2.8.2 (and now updated to v2.9.8) to get wildcard certificates for my subdomains with the following settings:

'certificatesResolvers:
letsencrypt:
acme:
email: [my-email]
storage: acme.json
dnsChallenge:
provider: "versio"`

It produced this output:

time="2023-02-26T09:35:38+01:00" level=error msg="Error renewing certificate from LE: {tevk.nl [*.tevk.nl]}" providerName=letsencrypt.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" error="error: one or more domains had a problem:\n[*.tevk.nl] [*.tevk.nl] acme: error presenting token: versio: 400: request failed: ObjectDoesNotExist|Domain not found\n[tevk.nl] [tevk.nl] acme: error presenting token: versio: 400: request failed: ObjectDoesNotExist|Domain not found\n"

My web server is (include version): Debian 10 Buster (everything running in docker containers)

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is: Versio - Note: Versio changed their UI recently: maybe something changed there?

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): No idea: this is done by traefik.

So the weird thing is: This used to work flawlessly! Something changed (somewhere) and I can't figure out what. Anybody has any ideas?

What I've tried:

set LEGO_DISABLE_CNAME_SUPPORT=true as an env variable: since i read somewhere that traefik 2.9+ had a new lego version which had trouble with wildcard domains: didn't work.
Checked: this topic but i'm already using dns challenge.

2

There's not much log to work with, but I'm betting it's a reply from the Versio API saying it couldn't find your domain?

You might want to consider trying lego manually first using the Versio test API using the command line (i.e.: bypassing Traefik) and get that working first. See Versio.[nl|eu|uk] :: Let’s Encrypt client and ACME library written in Go. and Loading... for more info.

Also please use the Let's Encrypt staging environment while testing while you're at it, thanks!

3

I'm not really sure if i'm using it wrong or it's something else:

What I did:

install Lego 0.3.0 (just apt install lego).

add my environment variables (VERSIO_PASSWORD and VERSIO_USERNAME)

then: sudo lego --email [my-email] --dns versio --server https://acme-staging-v01.api.letsencrypt.org/directory --domains *.tevk.nl run

The response:

2023/02/26 11:14:24 Could not create client: get directory at 'https://acme-staging-v01.api.letsencrypt.org/directory': failed to get "https://acme-staging-v01.api.letsencrypt.org/directory": Get https://acme-staging-v01.api.letsencrypt.org/directory: dial tcp: lookup acme-staging-v01.api.letsencrypt.org on 192.168.1.6:53: no such host

What I don't get:

Why is this trying to connect to my local IP port 53? I'm using a dns challange: so i'd expect it to work with just my dns provider (versio).

4

It seems your server (or lego?) is configured to use 192.168.1.6:53 as the DNS resolver. This error is yet before anything related to Versio, but with generic DNS lookups to connect to anything on the internet. You could perhaps check /etc/resolv.conf for any misconfiguration or maybe Lego has some DNS resolver setting.

5

! it was something else: I (incorrectly assumed that the staging v01 existed: It doesn't. Checked on a newer linux version which uses a newer version of LEGO.

So I got the error reproduced with just lego with a (tiny bit...) more logging:
lego --email [my-email] --dns versio --server https://acme-staging-v02.api.letsencrypt.org/directory --domains tevk.nl run

results in:
2023/02/26 11:28:02 [INFO] [tevk.nl] acme: Obtaining bundled SAN certificate 2023/02/26 11:28:02 [INFO] [tevk.nl] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/5523617694 2023/02/26 11:28:02 [INFO] [tevk.nl] acme: Could not find solver for: tls-alpn-01 2023/02/26 11:28:02 [INFO] [tevk.nl] acme: Could not find solver for: http-01 2023/02/26 11:28:02 [INFO] [tevk.nl] acme: use dns-01 solver 2023/02/26 11:28:02 [INFO] [tevk.nl] acme: Preparing to solve DNS-01 2023/02/26 11:28:03 [INFO] [tevk.nl] acme: Cleaning DNS-01 challenge 2023/02/26 11:28:03 [WARN] [tevk.nl] acme: error cleaning up: versio: 400: request failed: ObjectDoesNotExist|Domain not found 2023/02/26 11:28:03 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/5523617694 2023/02/26 11:28:04 Could not obtain certificates: acme: Error -> One or more domains had a problem: [tevk.nl] [tevk.nl] acme: error presenting token: versio: 400: request failed: ObjectDoesNotExist|Domain not found

So it does indeed just seems to be something with versio... I'm gonna check out their API if something changed. But if somebody has any ideas; let me know!

6

Ah, I missed that, yes, the version 1 of the ACME API (pre-RFC) has been shut down for some while now. Not sure where you got that URL from, as you manually entered it into the command line parameters for Lego.

I think so too, yes. Not sure how or why. Your domain name is clearly hosted by Versio when running dig +trace tevk.nl..

7

It is 99% sure that this is an issue on versio's side. Something changed or is broken there.

I've opened an issue on the LEGO github here. for anyone interested.

8

Got a response from Versio. They confirmed their api is non-functional for (new) customers. Nothing letsencrypt or traefik can do anything about.

Hopefully they can fix everything on their side, but i get the impression that the API is the least of their troubles.

9

Can it be placed behind a proxy?

10

Probably, but that's not my intention.

The solution I now picked is use a different type of challange that does not need this api. Too bad for me: No wildcard certificates. But works fine and no longer dependent on versio.