Unable to Renew Certificate (Traefik - Docker_ProviderTLS)

Hello Guys,

Two days now i'm trying to get rid of the weired behavior.
I have more than one server with same OS versions, running odoo behind a traefik reverse.
So, same configuration for docker-compose and traefik.toml. And on some of them, the minus 30 days certificate renewal fails with the bellow errors.
Logs are reporting a FW issue, but there is no FORWARD issue on port 443
Could you please provide support ?

My domains are: odoo12.imaginationsfertiles.fr and odoo12-test.imaginationsfertiles.fr

I ran this command: sudo docker-compose up -d --force-recreate

My docker-compose

version: "2.1"
services:
proxy:
image: traefik:latest
container_name: inverseproxy
environment:
TZ: "Europe/Paris"
networks:
shared:
aliases:
- odoo12.imaginationsfertiles.fr
- odoo12-test.imaginationsfertiles.fr
smtp:
private:
volumes:
- cert:/etc/traefik/acme:rw,Z
- logs:/var/log/traefik/:rw
- ./traefik.toml:/etc/traefik/traefik.toml:ro
ports:
- "80:80"
- "443:443"
depends_on:
- dockersocket
restart: unless-stopped

dockersocket:
    image: tecnativa/docker-socket-proxy
    container_name: inverseproxy_socket
    privileged: true
    userns_mode: "host"
    networks:
        private:
    volumes:
        - /var/run/docker.sock:/var/run/docker.sock
    environment:
        CONTAINERS: 1
        NETWORKS: 1
        SERVICES: 1
        SWARM: 1
        TASKS: 1
    restart: unless-stopped

networks:
shared:
driver_opts:
encrypted: 1
smtp:
internal: true
driver_opts:
encrypted: 1
private:
internal: true
driver_opts:
encrypted: 1

volumes:
cert:

My traefik

entryPoints]
[entryPoints.web]
address = ":80"
[entryPoints.web.http]
[entryPoints.web.http.redirections]
[entryPoints.web.http.redirections.entryPoint]
to = "websecure"
scheme = "https"

[entryPoints.websecure]
address = ":443"
[entryPoints.websecure.http]
middlewares = ["security-headers@file", "limit@file", "compression@file"]
[entryPoints.websecure.http.tls]
options = "default"
certResolver = "le"

[providers]
[providers.docker]
endpoint = "http://dockersocket:2375"
exposedByDefault = false
network = "inverseproxy_shared"
[providers.file]
directory = "/etc/traefik"

[log]
level = "DEBUG"

[accesslog]
filePath = "/var/log/traefik/access.log"
[accessLog.fields]
defaultMode = "keep"
[accessLog.fields.headers]
defaultMode = "keep"
[accessLog.fields.names]
StartUTC = "drop"
[accessLog.filters]
statusCodes = "400-499"

[certificatesResolvers]
[certificatesResolvers.le]
[certificatesResolvers.le.acme]
email = "mco@mail"
storage = "/etc/traefik/acme/acme.json"
[certificatesResolvers.le.acme.tlsChallenge]

[http]
[http.middlewares]
[http.middlewares.security-headers.headers]
browserXssFilter = true
contentTypeNosniff = true
referrerPolicy = "same-origin"
forceSTSHeader = true
frameDeny = true
customFrameOptionsValue = "SAMEORIGIN"
sslRedirect = true
stsIncludeSubdomains = true
stsPreload = true
stsSeconds = 15552000
[http.middlewares.security-headers.headers.customResponseHeaders]
server = "" # Removes
[http.middlewares.compression.compress]
[http.middlewares.limit.buffering]
retryExpression = "IsNetworkError() && Attempts() < 5"
[http.middlewares.auth.basicAuth]
users = ["iffil:gdfgdgdfggdggdfqzrerer]
[http.middlewares.smtp-stripprefix.stripprefix]
prefixes = ["/smtp"]
[http.middlewares.norobot-headers.headers]
[http.middlewares.norobot-headers.headers.customResponseHeaders]
X-Robots-Tag = "noindex, nofollow"

[tls]
[tls.options]
[tls.options.default]
minVersion = "VersionTLS12"
sniStrict = true
cipherSuites = [
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
"TLS_AES_256_GCM_SHA384",
"TLS_AES_128_GCM_SHA256",
"TLS_CHACHA20_POLY1305_SHA256",
"TLS_FALLBACK_SCSV" # Client is doing version fallback. See RFC 7507.
]

My web server is (include version):
Moved from traefik: v2.4 to traefik:latest

The operating system my web server runs on is (include version):
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"

I can login to a root shell on my machine (yes ):
And i get following error after runing the command#
"sudo docker-compose logs -f | grep -i error"

inverseproxy | time="2021-04-13T10:41:45+02:00" level=debug msg="Setting up buffering: request limits: 0 (mem), 0 (max), response limits: 0 (mem), 0 (max) with retry: 'IsNetworkError() && Attempts() < 5'" middlewareType=Buffer entryPointName=websecure routerName=websecure-odootest-restrict@docker middlewareName=limit@file
inverseproxy | time="2021-04-13T10:41:45+02:00" level=error msg="Error renewing certificate from LE: {odoo12-test.imaginationsfertiles.fr }, error: one or more domains had a problem:\n[odoo12-test.imaginationsfertiles.fr] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem)\n" providerName=le.acme
inverseproxy | time="2021-04-13T10:41:47+02:00" level=debug msg="Setting up buffering: request limits: 0 (mem), 0 (max), response limits: 0 (mem), 0 (max) with retry: 'IsNetworkError() && Attempts() < 5'" middlewareName=limit@file middlewareType=Buffer entryPointName=websecure routerName=websecure-odootestsmtp@docker

Is the testflume.ct.letsencrypt.org (Testing) listed under https://letsencrypt.status.io/ has any impact on certificate renewal ?

Nope, it doesn't. (I'm not familiar with docker nor Traefik, so can't help you with that, sorry.)

Thanks Osiris for your insights.
Anyone to help me tackle this issue ?
Regards

Following the test result from outside:
curl -IkL -m 15 https://odoo12.imaginationsfertiles.fr
HTTP/2 200
content-type: text/html; charset=utf-8
date: Wed, 14 Apr 2021 07:38:01 GMT
referrer-policy: same-origin
set-cookie: session_id=8eb5ec378fb4ce9b3dd77b313be5cefea1d33849; Expires=Tue, 13-Jul-2021 07:38:01 GMT; Max-Age=7776000; HttpOnly; Path=/
strict-transport-security: max-age=15552000; includeSubDomains; preload
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
content-length: 84

Hi @buju

I don't know how docker works.

But your configuration has minimal two bugs, that can't work - see https://check-your-website.server-daten.de/?q=odoo12.imaginationsfertiles.fr#url-checks

Domainname Http-Status redirect Sec. G
http://odoo12.imaginationsfertiles.fr/ 51.178.182.65 301 https://odoo12.imaginationsfertiles.fr/ Html is minified: 100,00 % 0.040 A
http://odoo12.imaginationsfertiles.fr/ 2001:41d0:404:200::5475 -14 10.030 T
Timeout - The operation has timed out
https://odoo12.imaginationsfertiles.fr/ 51.178.182.65 Inline-JavaScript (∑/total): 1/41 Inline-CSS (∑/total): 0/0 200 Html is minified: 323,08 % 2.537 B
small visible content (num chars: 0)
https://odoo12.imaginationsfertiles.fr/ 2001:41d0:404:200::5475 -14 10.023 T
Timeout - The operation has timed out
http://odoo12.imaginationsfertiles.fr/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 51.178.182.65 -14 10.020 T
Timeout - The operation has timed out
Visible Content:
http://odoo12.imaginationsfertiles.fr/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 2001:41d0:404:200::5475 -14 10.020 T
Timeout - The operation has timed out
  1. Checking your root http + ipv4 works, http + ipv6 doesn't work, timeout. That's fatal, Letsencrypt prefers ipv6, so your domain isn't visible.

  2. But http + ipv4 + / answers, http + ipv4 + /.well-known/acme-challenge/random-filename doesn't answer.

Looks like you have another, additional configuration (proxy, that handles /.well-known, firewall, that blocks). /.well-known/acme-challenge/random-filename must answer (via ipv6, if you have ipv6).

Thanks JuergenAuer for your detailed insights.
I'm a newbie in using Traefik-Lestencrypt + Docker but now that i switched my Challenge from tls to http, thing seems to work better. Certificates have been renewed from 30 May to 31 July.
Plus, the provided configuration had to work well and is still working on other servers i'm managing.
I just check if ipv6 is disabled with the following result:
imaginationsfertiles:~# sysctl -a 2>/dev/null | grep disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.br-20ecbfce02ee.disable_ipv6 = 0
net.ipv6.conf.br-6725bc40c6ec.disable_ipv6 = 0
net.ipv6.conf.br-6cf611c31e54.disable_ipv6 = 0
net.ipv6.conf.br-7a18718d6ea9.disable_ipv6 = 0
net.ipv6.conf.br-a5fd79c1b984.disable_ipv6 = 0
net.ipv6.conf.br-ccf861744af8.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.docker0.disable_ipv6 = 0
net.ipv6.conf.ens3.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.veth021bd63.disable_ipv6 = 0
net.ipv6.conf.veth2b34572.disable_ipv6 = 0
net.ipv6.conf.veth445ab16.disable_ipv6 = 0
net.ipv6.conf.veth514ca69.disable_ipv6 = 0
net.ipv6.conf.veth6794eae.disable_ipv6 = 0
net.ipv6.conf.veth76d77ac.disable_ipv6 = 0
net.ipv6.conf.veth7e8c973.disable_ipv6 = 0
net.ipv6.conf.vetha21cb84.disable_ipv6 = 0
net.ipv6.conf.vethc69ed9f.disable_ipv6 = 0
net.ipv6.conf.vethca3b2d6.disable_ipv6 = 0
net.ipv6.conf.vethda8d313.disable_ipv6 = 0
net.ipv6.conf.vethe711f18.disable_ipv6 = 0
net.ipv6.conf.vethef3b9a7.disable_ipv6 = 0

Also checked ipv6 open ports , only ssh and monitoring solutions ports are open.
I have check that no more than these ports are open on working servers .

Could i say requirement for Letsencrypt tlsChallenge include https ip6 port open ?

Could you please explain a bit regarding bugs you mentioned ?

Regards

And what about this entry

P https://51.178.182.65/ 51.178.182.65
-10
Error creating a TLS-Connection: TLSv1.3 found, but no connection via TLSv1.2 possible. Please activate TLSv1.2

Could it be my main issue ?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.