LetsEncrypt certificate requests fail in traefik reverse-proxy on raspberrypi

Help
1

Dear support team,
running evcc car charging system and traefik reverse-proxy in docker on a raspberrypi4 - please see https://jfraundo251158.github.io.
For some months everything was working fine. Some weeks ago unfortunately there were some changes, more or less in parallel. So I have no clue whether it was probably broken by an AVM Fritzbox or raspberry OS update, or by running out of the 3 month certificate lifetime and a possible never functioning renewal?

My domain is:
7y4sbu5yhxrbxqpi.myfritz.net

I ran this command:

It produced this output: Traefik log
...
time="2023-11-21T09:59:58Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/286007434446"
time="2023-11-21T09:59:58Z" level=error msg="Unable to obtain ACME certificate for domains "7y4sbu5yhxrbxqpi.myfritz.net": unable to generate a certificate for the domains [7y4sbu5yhxrbxqpi.myfritz.net]: error: one or more domains had a problem:\n[7y4sbu5yhxrbxqpi.myfritz.net] acme: error: 400 :: urn:ietf:params:acme:error:connection :: 2a02:3100:9200:af09:e228:6dff:fe08:34ed: Error getting validation data\n" rule="Host(7y4sbu5yhxrbxqpi.myfritz.net)" routerName=https-evcc-https@file providerName=le.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
...
time="2023-11-21T10:00:00Z" level=debug msg="Domains ["7y4sbu5yhxrbxqpi.myfritz.net"] need ACME certificates generation for domains "7y4sbu5yhxrbxqpi.myfritz.net"." ACME CA="https://acme-v02.api.letsencrypt.org/directory" rule="Host(7y4sbu5yhxrbxqpi.myfritz.net)" routerName=https-evcc-https@file providerName=le.acme
time="2023-11-21T10:00:00Z" level=debug msg="Loading ACME certificates [7y4sbu5yhxrbxqpi.myfritz.net]..." rule="Host(7y4sbu5yhxrbxqpi.myfritz.net)" routerName=https-evcc-https@file providerName=le.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
time="2023-11-21T10:00:00Z" level=debug msg="legolog: [INFO] [7y4sbu5yhxrbxqpi.myfritz.net] acme: Obtaining bundled SAN certificate"
time="2023-11-21T10:00:00Z" level=error msg="Unable to obtain ACME certificate for domains "7y4sbu5yhxrbxqpi.myfritz.net": unable to generate a certificate for the domains [7y4sbu5yhxrbxqpi.myfritz.net]: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many failed authorizations recently: see Failed Validation Limit - Let's Encrypt" routerName=https-evcc-https@file providerName=le.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" rule="Host(7y4sbu5yhxrbxqpi.myfritz.net)"
time="2023-11-21T10:03:59Z" level=debug msg="http: TLS handshake error from 192.241.222.92:36192: tls: client offered only unsupported versions: "
...

My web server is (include version):

The operating system my web server runs on is (include version):
Raspi OS (Debian)

My hosting provider, if applicable, is:
System is running on my raspberrypi4 with up-to-date OS

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Traefik 2.10.5 is doing the certification requests

Any help to detect the issue would be very welcome!

Thanks,
Juergen

2

Hello @jfr251158, welcome to the Let's Encrypt community. :slightly_smiling_face:

The domain has both an IPv4 and an IPv6 Addresses; they both need to respond to Port 80
for the HTTP-01 challenge however it seems at least IPv6 is not responding on Port 80.

Best Practice - Keep Port 80 Open

Using the online tool Let's Debug yields these results https://letsdebug.net/7y4sbu5yhxrbxqpi.myfritz.net/1685307?debug=y

AAAANotWorking
ERROR
7y4sbu5yhxrbxqpi.myfritz.net has an AAAA (IPv6) record (2a02:3100:9200:af09:e228:6dff:fe08:34ed) but a test request to this address over port 80 did not succeed. Your web server must have at least one working IPv4 or IPv6 address. You should either ensure that validation requests to this domain succeed over IPv6, or remove its AAAA record.
Get "http://7y4sbu5yhxrbxqpi.myfritz.net/.well-known/acme-challenge/letsdebug-test": dial tcp [2a02:3100:9200:af09:e228:6dff:fe08:34ed]:80: connect: permission denied

Trace:
@0ms: Making a request to http://7y4sbu5yhxrbxqpi.myfritz.net/.well-known/acme-challenge/letsdebug-test (using initial IP 2a02:3100:9200:af09:e228:6dff:fe08:34ed)
@0ms: Dialing 2a02:3100:9200:af09:e228:6dff:fe08:34ed
@108ms: Experienced error: dial tcp [2a02:3100:9200:af09:e228:6dff:fe08:34ed]:80: connect: permission denied
1 Like
3

Sounds like you need to test on the test/staging environment.

2 Likes
4

Thanks for fast reply! Have enabled IPv6 too - but didn't fix it. Found now that most likely the Fritzbox is doing something wrong. Forwarding port 80 to my raspberrypi seems to be fishy and instead http://7y4sbu5yhxrbxqpi.myfritz.net brings up the Fritzbox login dialog. I'm trying to clarify this with AVM.

2 Likes
5

I don't know anything about Fritzbox but vaguely remember it needs special care :slight_smile: You might find something helpful on past threads for it

https://community.letsencrypt.org/search?expanded=true&q=fritzbox

4 Likes
6

You are right - there are a lot of similar issues. e.g. Certificate renewal on Ubuntu 16.4 failed. In there it's said "You'll also need to make sure that your Apache server that Certbot is creating a certificate for is accessible on port 80. At the moment your Fritzbox control panel appears to be on that port instead". Same for my setup for which I'm in touch with AVM. Thanks!

4 Likes