Traefik Docker broken

Hello all,

I have been running traefik for a good year and change with very minor issues. Today I noticed the docker container is no longer running. When I tried to restart it, I got the dreaded “command traefik error: field not found, node: redirect” error.

It’s my understanding that this happens when updating traefik (I have my docker containers all set to autoupdate) and the config file contains something that is no longer available in the version being used.

So my first question is - why did this happen? I mean, I know it’s because config no longer matches the API, but I question why the config file doesn’t get updated by traefik to convert to the new calls. I’m sure I’m not the only one that had traefik break on them because of this.

Anyway, that’s more of a gripe. I’m assuming the point that the new version no longer likes is this:

# Force HTTPS
[entryPoints]
  [entryPoints.http]
  address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
  [entryPoints.https]
  address = ":443"
[entryPoints.https.tls]

How can I fix this? My SSL certs are all busted right now. Please help :frowning:

2 Likes

Same for me and I’m a total noob. I barely got Traefik running after hours of following tutorials. I wouldn’t have a clue on how to fix it. Following this thread for future info from smart people.

2 Likes

Hi. I’d like to welcome you both to join us at the Community Forum for all Traefik related questions. Thank you. https://community.containo.us

I’m struggling so far, but the answer is somewhere here, I think:

https://docs.traefik.io/v2.0/migration/v1-to-v2/#http-to-https-redirection-is-now-configured-on-routers

A whole bunch has been changed. It makes me want to revert back to 1.7 or whatever version I was using. For someone not totally familiar with creating toml files to configure the docker containters, the changes are overwhelming. And the samples appear to contain extraneous things other than a 1:1 conversion.

1 Like

If you haven’t found a solution yet, I was successful in getting redirection working on Traefik 2.0, although it’s a little more verbose and currently I have it set up on the service, not globally. Basically you need a router taking http traffic which calls a middleware to do the redirect to https, and you need a second router which routes incoming https traffic to the service. As an example, my whoami demo looks like this currently (I’m using Docker swarm mode):

version: '3.7'

services:
  whoami3:
    image: containous/whoami
    networks:
      - traefik-public
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik-public"
# Set up redirect on insecure port to https 443 (using an arbitrary middleware name of `whoami3-redirect`)
# `whoami3-web` is the router name for http to https redirection
        - "traefik.http.routers.whoami3-web.entrypoints=web"
        - "traefik.http.routers.whoami3-web.rule=Host(`whoami3.local`)"
        - "traefik.http.middlewares.whoami3-redirect.redirectscheme.scheme=https"
        - "traefik.http.middlewares.whoami3-redirect.redirectscheme.permanent=true"
        - "traefik.http.routers.whoami3-web.middlewares=whoami3-redirect@docker"
# Handle secure traffic (requires a separate frontend router for TLS)
# `whoami3-websecure` is the router name for TLS connections
        - "traefik.http.routers.whoami3-websecure.entrypoints=websecure"
        - "traefik.http.routers.whoami3-websecure.rule=Host(`whoami3.local`)"
        - "traefik.http.routers.whoami3-websecure.tls=true"
# Define which port to connect to the service on (backend)
        - "traefik.http.services.whoami3-service.loadbalancer.server.port=80"

networks:
  traefik-public:
    external: true
    name: traefik-public

It’s notable that you need to specify the tls=true on the https router, and you also need to specify the service port (last label). I’m not using toml anymore, but I imagine you would just remove the redirect line and the tls line from the toml config, and leave the entry points.

Also this:

2 Likes

Thank you. That second link is helpful. I do global http to https redirection with Let’s Encrypt SSL encryption. I think I would do the redirects via labels in each docker, though. The SSL encryption/LE stuff was mainly in a toml file. I basically followed the guide here:

with wildcards for any subdomain and then just make A records in Cloudflare. With 2.0, every single bit of that has changed and I really wish they didn’t push the upgrade upstream. I have my docker containers all set to auto update on an UnRAID box and it broke everything.

1 Like

I saw a lot of other people talking about that, using latest as the version for Traefik. Generally the best practice way with Docker is to specifically define the version you want to use, which avoids breaking changes or at least specify the major version like v1.7 so you still get security updates.

I just got global redirection working by basically putting that redirection router on the traefik container:

        - "traefik.enable=true"
        - "traefik.docker.network=traefik-public"
        - "traefik.http.routers.http-catchall.rule=HostRegexp(`{any:.*}`)"
        - "traefik.http.routers.http-catchall.entrypoints=web"
        - "traefik.http.routers.http-catchall.middlewares=redirect-to-https@docker"
        - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=websecure"
        - "traefik.http.services.noop.loadbalancer.server.port=80"

So you put every one of those labels on each container? I’d have to do that for a ton of containers. That’s not really feasible for me. I guess I’ll have to try to work out how to do it in the toml file. I checked my config on one of the other untouched containers and all I had was the traefik enable label and it did the rest. It was so much easier.

As far as using “latest” I just used the default template from the repo (which I guess defaults to latest). I stand by my point that any revision changes that are going to completely break everything should never be pushed upstream. The only thing I’ve done “differently” is enabled auto-update recently on UnRAID. I guess that’s my fault. Still, it’s like expecting security patches and getting a full version upgrade instead. I’m considering going back to 1.7 just to fix everything.

Thank you, Patricia. I may jump over there and ask the same question. Ugh, this is so frustrating. Everything worked fine until the auto update to 2.0.

No, actually you can put those labels right on the Traefik container (basically setting up one router + one middleware which redirects all traffic from http to https). Then you only need to define the https router per service (pretty much what you're already doing).
You can use toml to define this (as outlined in that post on the containous forums), it's user preference.

I understand the frustration with the upgrade breaking everything without notice, but that's a result of using the latest tag. They have other tags that behave as you were probably expecting, for example the v1.7 or maroilles tag which would allow automatic updates for minor versions but prevent breaking changes. Unfortunately whoever created the template you used should not have defaulted to latest which is considered bad practice.

Oh that makes things a bit easier, then.

I checked - and it does have the latest tag. It’s also from the official Traefik hub

https://hub.docker.com/_/traefik/

I think all repositories on the hub have a latest tag, but they just don’t recommend using it for production. I don’t see anywhere on that page where they include that in a template or example though.

If you look:

docker pull traefik

That will always pull the latest. And the UnRAID templates all just use that docker pull command.

I should clarify - if you have the docker containers set to autoupdate (which basically does a pull)

Well I think we already established the importance of pinning the version, so if UnRAID is encouraging people to deploy with auto updates using latest - they’re doing it wrong. They should specify a version or at least include a comment about using the version.

Anyway I think this discussion devolved beyond the original question, but let me know if you still need help with the new configuration assuming you are moving forward with 2.0.

I don’t disagree with you, mind you. Just saying these are basically the “out of the box” options and people who aren’t docker savvy are going to run into these very same issues I have. In fact, UnRAID doesn’t even support docker compose out of the box. It’s all CLI (which is why the templates.) The template given by the official Traefik hub is for latest. I guess if you do not have auto-update enabled, it won’t jump from 1.7 to 2.0 and it is disabled by default, so there is that.

But yes, we are digressing. So I tried to start fresh. I figured things are broken right now, so I can’t break them any worse haha. I used the sample toml file from the docs:

https://raw.githubusercontent.com/benderstwin/Portainer-Templates/master/traefik.toml

I changed nothing, though I added an .htpasswd file as it instructs.

I start the docker and immediately get:

2019/09/24 16:04:11 command traefik error: field not found, node: address

Are you using Traefik 2.0 (latest)? That will definitely not work because some of the options in that file are incompatible with version 2.0. You would need to remove the redirect stuff, TLS stuff, and honestly I’m not sure how the dashboard authentication works yet so that may also be a little different.

I mean I tried using Traffic 2.0 and using their example config and got that error. I’ve managed to get past that since yesterday, though.

Now, I get no errors. But nothing is working yet. The thing that makes this the most difficult is that the docs have everything split over multiple pages and even in the example configs for say, tls, the TOML files have other unrelated things in them, so it’s hard to parse what I need and what I don’t. :frowning:

It’s hard to say what’s wrong without more details at this point. If you could post your docker-compose.yml file and traefik.toml file that would be a good start. When you say nothing is working, can you access the dashboard at all?

I don’t/can’t use docker-compose (due to being on UnRAID) so it’s all CLI launched. I can post my traefik.toml file, though. Should I redact my domain and whatnot?

And correct - no dashboard, no nothing. It’s also not creating the acme.json or even the folder - yet strangely enough, when I picked the same root path for the logfiles, it did create those.