Challenge failed with wrong domain

Hi all
I have several domains managed by letsencrypt (*.levillain.ml, eklectik-mood.ml) and I recently created a new domain named levillain.tk that I want to use for testing moving some of my stuff (email server, nextcloud…) to docker.
All these domains use the same IP address (actually, there all running on the same rapsberrypi)

So I created a docker container using the mailu/mailu image which manages letsencrypt automatically.
I setup the container to manage the mail domain mail.levillain.tk and I ran it.

From the mailu sources, I see it’s running the following python code to configure letsencrypt:

command = [
    "certbot",
    "-n", "--agree-tos", # non-interactive
    "-d", os.environ["HOSTNAMES"],
    "-m", "{}@{}".format(os.environ["POSTMASTER"], os.environ["DOMAIN"]),
    "certonly", "--standalone",
    "--cert-name", "mailu",
    "--preferred-challenges", "http", "--http-01-port", "8008",
    "--keep-until-expiring",
    "--rsa-key-size", "4096",
    "--config-dir", "/certs/letsencrypt",
    "--post-hook", "/config.py"
]

Looking at the logs, I can see the following:

2020-05-09 08:12:17,140:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/4466700232 HTTP/1.1" 200 1551
2020-05-09 08:12:17,141:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 09 May 2020 08:12:16 GMT
Content-Type: application/json
Content-Length: 1551
Connection: keep-alive
Boulder-Requester: 85626288
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0002lfRVqtvEt9HD5keN5YzCrkomLLtQP-Ebj48CmNvn7c8
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "mail.levillain.tk"
  },
  "status": "invalid",
  "expires": "2020-05-16T08:12:06Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from https://www.levillain.ml/.well-known/acme-challenge/WQVdC9UkkLusfqx_LCsGjSvB7T-brnAO7OniPy4sYx0 [78.203.12.245]: \"\u003c!DOCT$        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/4466700232/kSphaA",
      "token": "WQVdC9UkkLusfqx_LCsGjSvB7T-brnAO7OniPy4sYx0",
      "validationRecord": [
        {
          "url": "http://mail.levillain.tk/.well-known/acme-challenge/WQVdC9UkkLusfqx_LCsGjSvB7T-brnAO7OniPy4sYx0",
          "hostname": "mail.levillain.tk",
          "port": "80",
          "addressesResolved": [
            "78.203.12.245"
          ],
          "addressUsed": "78.203.12.245"
       },
        {
          "url": "https://www.levillain.ml/.well-known/acme-challenge/WQVdC9UkkLusfqx_LCsGjSvB7T-brnAO7OniPy4sYx0",
          "hostname": "www.levillain.ml",
          "port": "443",
          "addressesResolved": [
            "78.203.12.245"
          ],
          "addressUsed": "78.203.12.245"
        }
      ]
    }
  ]
}
2020-05-09 08:12:17,142:DEBUG:acme.client:Storing nonce: 0002lfRVqtvEt9HD5keN5YzCrkomLLtQP-Ebj48CmNvn7c8
2020-05-09 08:12:17,143:WARNING:certbot.auth_handler:Challenge failed for domain mail.levillain.tk
2020-05-09 08:12:17,144:INFO:certbot.auth_handler:http-01 challenge for mail.levillain.tk

So, as you can see, it tried to create certificate for mail.levillain.tk but asked the challenge from https://www.levillain.ml/.well-known/… instead of https://www.levillain.tk/.well-known/…

As levillain.tk is running in the docker container, the 2 domains do not share anything…

My web server is (include version): nginx 1.16.1 in docker container

The operating system my web server runs on is (include version): alpine 3.10.5 in docker container

I can login to a root shell on my machine (yes or no, or I don’t know): I didn’t find how to login as root yet in the docker container but I can execute commands one by one as root.

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): docker exec mailu_front_1 certbot --version => 0.35.1

1 Like
$ curl -I mail.levillain.tk/.well-known/acme-challenge/test
HTTP/1.1 302 Found
Date: Sat, 09 May 2020 09:15:48 GMT
Server: Apache
Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
Referrer-Policy: no-referrer
Location: https://www.levillain.ml/.well-known/acme-challenge/test
Content-Type: text/html; charset=iso-8859-1

Look at the Location header sent by your server ^.

I’d guess that the issue is that the HTTP-to-HTTPS rule sitting in your Apache config, in front of your Docker containers, is preventing the port 80 requests from reaching the mailu container.

Maybe you could add a port 80 virtualhost in Apache for the mailu domain, which proxies the requests to the mailu container, without any redirect?

1 Like

Thanks @_az
You’re totally right and actually, I’m a little bit dumb :upside_down_face:
There’s only one port 80 per IP adress and I forgot I wanted to use the port 81 for HTTP on levillain.tk as it’s only for testing and I have already “production” sites running on the other domains…
Is there a way to tell LE to use port 81 (or any other port) for the challenge?

1 Like

The only permitted port for the HTTP challenge is 80.

I’m not sure why you can’t use the one Apache server for this task, though. With name-based virtual hosting, you just need to tell Apache to proxy requests for mail.levillain.tk to whatever the ip:port of your Docker container is, and your other sites will be unaffected.

2 Likes

Actually, I’m a bit puzzled (probably because I don’t know docker networking enough) and also probably, it’s not even a letsencrypt troubleshooting problem anymore :thinking:

I managed to write a vhost that proxies my requests to the container.
The vhost looks like this:

<VirtualHost *:80>
  ServerName levillain.tk
  ServerAlias www.levillain.tk
  ServerAlias mail.levillain.tk
  ProxyPass "/" "http://localhost:8080/"
  ProxyPassReverse "/" "http://localhost:8080/"
</VirtualHost>

8080 is mapped onto the 80 port inside my container… (changed from 81 to 8080 to be more “standard”):

/var/lib/mailu $ docker port mailu_front_1
110/tcp -> 0.0.0.0:8110
143/tcp -> 0.0.0.0:8143
25/tcp -> 0.0.0.0:8025
465/tcp -> 0.0.0.0:8466
80/tcp -> 0.0.0.0:8080
995/tcp -> 0.0.0.0:8995
993/tcp -> 0.0.0.0:8993
443/tcp -> 0.0.0.0:8443
587/tcp -> 0.0.0.0:8587

But now, I have:

  • a 503 error when accessing to http://mail.levillain.tk (or http://levillain.tk)
  • a 403 error when accessing directly to http://myrasberrypi:81 (showing that nginx is working by the way).
    I assume it is because the nginx server disallow the direct connection in http mode. (except for certbot which is running in standalone mode)
    But still, the letsencrypt log also show this 503 error:

Domain: mail.levillain.tk
Type: unauthorized
Detail: Invalid response from http://mail.levillain.tk/.well-known/acme-challenge/HOGF39FH0gkP9GoQU3BzmcHc7pqSZYmEWsvO7hVDa9Y [78.203.12.245]: “\n
\n503 Service Unavailable\n\n

Service”

Also, I noticed that certbot used by mailu was using the 8008 port (–http-01-port 8008) so I mapped it too but as far as I understand, it’s used internally only…

Still searching…

1 Like

Seems like it should work, assuming Apache and Docker are on the same host. What’s the Apache error_log show when you encounter that 503?

1 Like

I get this in the apache error.log:

[Sun May 10 09:46:11.305869 2020] [proxy:error] [pid 24789:tid 2592027680] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[Sun May 10 09:46:11.306039 2020] [proxy_http:error] [pid 24789:tid 2592027680] [client 18.196.96.172:58328] AH01114: HTTP: failed to make connection to backend: localhost
[Sun May 10 09:46:11.945691 2020] [proxy:error] [pid 24788:tid 2703209504] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[Sun May 10 09:46:11.945795 2020] [proxy:error] [pid 24788:tid 2585711648] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[Sun May 10 09:46:11.945786 2020] [proxy_http:error] [pid 24788:tid 2703209504] [client 34.209.232.166:35534] AH01114: HTTP: failed to make connection to backend: localhost
[Sun May 10 09:46:11.945866 2020] [proxy_http:error] [pid 24788:tid 2585711648] [client 52.15.254.228:38708] AH01114: HTTP: failed to make connection to backend: localhost
[Sun May 10 09:46:11.946436 2020] [proxy:error] [pid 24788:tid 2566906912] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
[Sun May 10 09:46:11.946476 2020] [proxy_http:error] [pid 24788:tid 2566906912] [client 66.133.109.36:19976] AH01114: HTTP: failed to make connection to backend: localhost

So, it seems Apache cannot connect to localhost:8080 which should be redirected to the 80 port in my container which itself should run the standalone certbot http server… :pensive:

Actually, I’m puzzled by 2 things…

  1. The letsencrypt log says:
{
   "identifier": {
    "type": "dns",
    "value": "mail.levillain.tk"
  },
  "status": "invalid",
  "expires": "2020-05-17T07:45:38Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://mail.levillain.tk/.well-known/acme-challenge/4lSF0cLfAiSXdegGobWl1VRdq4MeFgPkpoiqTlstmVk [78.203.12.245]: \"\u003c!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\"\u003e\\n\u003chtml\u003e\u003chead\u003e\\n\u003ctitle\u003e503 Service Unavailable\u003c/title\u003e\\n\u003c/head\u003e\u003cbody\u003e\\n\u003ch1\u003eService\"",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/4487553073/SGjhag",
      "token": "4lSF0cLfAiSXdegGobWl1VRdq4MeFgPkpoiqTlstmVk",
      "validationRecord": [
        {
          "url": "http://mail.levillain.tk/.well-known/acme-challenge/4lSF0cLfAiSXdegGobWl1VRdq4MeFgPkpoiqTlstmVk",
          "hostname": "mail.levillain.tk",
          "port": "80",
          "addressesResolved": [
            "78.203.12.245"
          ],
          "addressUsed": "78.203.12.245"
        }
      ]
    }
  ]
}

So, you can see that if apache returns a 503 error in the message, the actual status is 403 : how can I get a 403 error from the certbot standalone server?

  1. the http-01-port 8008 parameter: when I grep 8008 in the LE logs, I get:

$ docker exec mailu_front_1 grep 8008 /var/log/letsencrypt/letsencrypt.log
2020-05-10 07:45:22,120:DEBUG:certbot.main:Arguments: [’-n’, ‘–agree-tos’, ‘-d’, ‘mail.levillain.tk’, ‘-m’, ‘admin@levillain.tk’, ‘–standalone’, ‘–cert-name’, ‘mailu’, ‘–preferred-challenges’, ‘http’, ‘–h
ttp-01-port’, ‘8008’, ‘–keep-until-expiring’, ‘–rsa-key-size’, ‘4096’, ‘–config-dir’, ‘/certs/letsencrypt’, ‘–post-hook’, ‘/config.py’]
2020-05-10 07:45:38,935:DEBUG:acme.standalone:Successfully bound to :8008 using IPv6
2020-05-10 07:45:38,936:DEBUG:acme.standalone:Certbot wasn’t able to bind to :8008 using IPv4, this is often expected due to the dual stack nature of IPv6 socket implementations.
2020-05-10 07:46:14,213:DEBUG:certbot.plugins.standalone:Stopping server at :::8008…

==> I don’t understand what is the meaning of this parameter. The logs seem to say that the 80 port is used…

The 403 that Let’s Encrypt reports is very misleading.

You might think it’s the status code that your Apache server generated, but it isn’t. It’s actually just a generic “Unauthorized” code that indicates the challenge failed.

Let’s Encrypt did observe the 503. Look in the detail key - you’ll find the 503 Service Unavailable text in there.

The 8008 stuff is just an internal detail of the mailu Docker image. If you look at how the nginx server is configured, it proxies the acme-challenge requests to 8008 internally:

So you can ignore that this part exists - it’s not the reason things aren’t working.

Yes, that’s the real problem to solve.

The Docker daemon and Apache are running in the same server and same operating system, right? Apache is not in a container?

On the Apache server, can you run:

ss -tlnp | grep ":8080"
curl -X GET -I localhost:8080/

Thanks for you help @_az
Yes, Apache and the Docker container are running on the same server (a pi4 under Raspbian buster). As far as I can see, mailu itself is running on Alpine in the docker container…

Here are the results of the commands (192.168.0.8 is my Pi internal IP address as shown by ifconfig)

$ ss -tlnp | grep ":8080"
LISTEN    0         128            192.168.0.8:8080             0.0.0.0:*
pi@raspberrypi ~ $ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.8  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::2192:a65a:a0bd:38b6  prefixlen 64  scopeid 0x20<link>
        ether dc:a6:32:2a:d2:0f  txqueuelen 1000  (Ethernet)
        RX packets 3578823  bytes 3370013817 (3.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3221609  bytes 1102551860 (1.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

pi@raspberrypi ~ $ curl -X GET -I localhost:8080/
curl: (7) Failed to connect to localhost port 8080: Connexion terminée par expiration du délai d'attente

(==> Connection terminated by timeout)

If I get it properly, that’s normal as the server on the port 8080 (80 in the container) is only the certbot standalone server and it terminates at the end of the certbot command, right?

I think :8080 should map to the nginx server running on the container. The Certbot standalone server doesn’t get a port exposed because nginx reverse proxies to it (internally in the container).

Out of curiosity, what about:

curl -X GET -I 192.168.0.8:8080

Does that show the nginx server?

Actually, I forgot it but as said yesterday nginx is running and returns a 403 error:

$ curl -X GET 192.168.0.8:8080
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

Right … so the mistake we made is that we assumed Docker is binding to all addresses (0.0.0.0), when it’s actually only binding your 192/8 address. :man_shrugging:.

Update your Apache config to use this address instead of the localhost:8080 one, and I think everything should start working.

1 Like

Halleluya, it worked, I got a certificate !!!
Thank you so much for your help @_az :clap: :clap: :clap:

No, I just need to figure the next steps. Hopefully, I’ll get less headbanging :slight_smile:

Ouch…
Still headbanging…

Firefox ne fait pas confiance à ce site, car il utilise un certificat qui n’est pas valide pour mail.levillain.tk. Le certificat est seulement valide pour les noms suivants : *.levillain.ml, levillain.ml

  • When I access https://192.168.0.8:8443/ (port 8443 mapped to 443 in the container), I get the message that the certificate is valid only for mail.levillain.tk !

Firefox ne fait pas confiance à ce site, car il utilise un certificat qui n’est pas valide pour 192.168.0.8:8443. Le certificat n’est valide que pour mail.levillain.tk.

How can I also “proxy” the certificate through the 443 port?

EDIT 2: my vhosts proxying is as follows:

#Define mailu_host levillain.tk
Define mailu_host localhost
#Define mailu_host 192.168.0.8
Define mailu_port 8080
Define mailu_ports 8443

<VirtualHost *:80>
  ServerName levillain.tk
  ServerAlias www.levillain.tk
  ServerAlias mail.levillain.tk
  ProxyPass "/" "http://${mailu_host}:${mailu_port}/"
  ProxyPassReverse "/"  "http://${mailu_host}:${mailu_port}/"
</VirtualHost>

# virtualhost https
<VirtualHost *:443>
  ServerName levillain.tk
  ServerAlias www.levillain.tk
  ServerAlias mail.levillain.tk
  # also tried with mailu_host = mail.levillain.tk, same result 
  ProxyPass "/" "https://${mailu_host}:${mailu_ports}/"
  ProxyPassReverse "/"  "https://${mailu_host}:${mailu_ports}/"
</VirtualHost>

It’s not possible with Apache, as you’d need to do something like SSL prereading + TCP proxying.

To try summarize your needs:

  • You only have 1 IP address available.
  • The mailu container needs a certificate so it can secure the mail ports
  • The Docker host needs a certificate so it can secure the reverse proxy to the mailu HTTP server

I think the efficient thing to do would be to have the Docker host manage the certificate (with e.g. Certbo), using that certificate with Apache, and also passing that certificate into the container. According to the mailu docs, that would be the cert TLS flavor.

Ok, thanks @_az.
For now, I’ll continue my tests with the explicit port 8443.
I’ll see the certs mode if I can’t avoid it, I don’t want to run again in the same kind of trouble…
BTW, you say it’s not possible with Apache and you point an Nginx module, would it mean that it could work with nginx replacing Apache?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.