Early Renewal Traefik

Well this stinks, with very little notice. I have about 10 websites impacted, because of using Traefik's TLS challenge.

In my case, I looked at the acme.json in all my various server's traefik configs, and searched for "acct". There's an entry in the json that looks like:

"uri": "https://acme-v02.api.letsencrypt.org/acme/acct/XXXXXXXXX"

...which is where I was able to match the account IDs from the email.

Looks like I gotta figure out how to forcibly renew Traefik certs before Friday morning.

4 Likes

It's not hard, just empty the array inside your acme.json and restart Traefik it will do the rest.

4 Likes

Thanks @suderman good observation.

Here is what I did: grep for the account number. Then remove the acme.json (assuming that is the name of the traefik-generated cert/key file) and bounce traefik.

5 Likes

Hi,

I'm a Traefik user and enabled tlsChallenge so I am affected :stuck_out_tongue:

Following Manually trigger Traefik to generate letsencrypt certificate using API · Issue #3652 · traefik/traefik · GitHub, I replaced acme.json with this content (personal informations changed :wink: ):

{
  "le": {
    "Account": {
      "Email": "mail@example.com",
      "Registration": {
        "body": {
          "status": "invalid",
          "contact": [
            "mailto:mail@examplel.com"
          ]
        }
      }
    }
  }
}

I restarted traefik container and my certs were regenerated.

6 Likes

I stopped my traefik container, made a backup of my acme.json file, then deleted everything inside, and restarted the container. An entirely new certificate was generated, the account number in the acme.json file is different now, and the website still works with no warnings, and the date of validity has been extended. So while it wasn't a renewal, it solved my problem.

5 Likes

Exactly like @BamButz said. I also use Traefik with docker-compose.yml.
What I did in steps:

  1. Log on to your server and cd in the letsencrypt directory with the acme.json
  2. Rename file (just for backup): mv acme.json revoked_acme.json
  3. Create new empty file: touch acme.json
  4. Shut down all containers: docker-compose down
  5. Start all containers (detached): docker-compose up -d
  6. Now open your acme.json and you will see a new certificate with a new key.
6 Likes

Is there any other option that doesn't make you delete the acme.json and restarting Traefik? I have lots of certificates that were created using DNS-01 so they don't need to be renewed, but also lots of TLS-ALPN-01 certs that will take some time to renew, thus deleting and restarting implies that all those sites are going to be down for some time with users getting certificate errors.

1 Like

You can just remove the certificate key (the whole line) from the affected certificates objects, and restart traefik.

2 Likes

Hmm, I think I understand what needs to be done. But looks like we're dealing with some amount of downtime while Traefik restarts and waits for new certificates to arrive. I guess I'll be up tonight.

1 Like

Hello,

For people that need help to clean their acme.json file, I quickly created a simple tool to help to remove your certificates from the acme.json (Traefik v2 only).

The process:

  • run the tool with the right arguments
  • copy the content of the generated file to your acme.json file
  • restart Traefik

The readme contains examples for all the options (only 3 options).

FYI I'm a Traefik maintainer

7 Likes

I think I'll attempt this approach first. Leaves most of my acme.json intact, keeping account number, keys, etc.

1 Like

Traefik sent out a guide tonight:

After replacing the value of the certificates key with an empty array "Certificates": [ ], I bounced Traefik docker service update platform_traefik --force and everything came back with fresh certificates. On one of my busier swarms, it took a minute since there were many more domains to process, so I did see certificate errors briefly. But it cleared up before any of my monitors noticed. :slight_smile:

5 Likes

Instructions for someone not familiar with Golang?

I did this, and I found it to be easiest for me. I did all the steps except touch acme.json to create the new, blank file. It seemed to confuse Traefik, so I just made the backup "revoked_acme.json", exited the Docker container, and restarted the container. When I re-entered the container, the new "acme.json" was there, and Traefik created new certificates.

Total website downtime ~30 seconds or so.

1 Like

Hello,

I added installation instructions.

2 Likes

Awesome thanks! I ended up just deleting the acme.json file and restarting Traefik, which worked really well for me, but I will save this for future reference!

1 Like

@mccarthysean well, I wasn't sure if traffic expects the acme.json or creates a new one. Thanks for the hint, I'll update my answer.

2 Likes

Following the instructions provided we're getting the following error when trying to renew the certificates in question (quite a bunch of them).

time="2022-01-29T16:28:11Z" level=error msg="Unable to obtain ACME certificate for domains "mydomain.com": unable to generate a certificate for the domains [mydomain.com]: acme: Error -> One or more domains had a problem:\n[mydomain.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Incorrect validation certificate for tls-alpn-01 challenge. Requested mydomain.com from 188.166.200.61:443. Received 1 certificate(s), first certificate had identifiers "e415182936ebe9b8d955d8479985a8c5.89f80cd55880258102a1c2cb655a436a.traefik.default, traefik default cert"; got error dNSName does not match expected identifier, url: \n" routerName=default-my-app-name-8c772840d127f0d16fa8@kubernetescrd rule="(Host(mydomain.com)) && PathPrefix(/)" providerName=default.acme

Using Traefik 2.1.9 in Kubernetes.

What does "dNSName does not match expected identifier" really mean?

While remediating our TLS-ALPN-01 OID and TLS versions incident, we received a report about another TLS-ALPN-01 RFC non-compliance in our Boulder code. We have posted a preliminary report 1752670 - Let’s Encrypt TLS Using ALPN Allows Additional Identifiers in Challenge Certificate

We have also patched our CA software Boulder to correctly respond and error when client TLS challenge certificates are not compliant to the RFC.

If you are receiving this error, then your client has likely incorrectly implemented the TLS-ALPN-01 challenge type. Your error indicates the the TLS challenge certificate contains incorrect or unexpected information such that the TLS challenge certificate is not compliant with the RFC. You will need to use another challenge type to issue a certificate until your client is fixed. If you are able to open an issue on their source code repository, you should do so. cc @elDez

7 Likes

I think it's because your Traefik version is too old (2 years), can you update your Traefik?

@jilian do you plan to also update pebble?

2 Likes