Caddy auto https error, 404 Timeout after connect (your server may be slow or overloaded)

ibb2 · November 13, 2020, 2:01am

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

My project is containerized along side caddy (version 2.2.1-alpine) and is orchestrated with docker swarm. It is running on a remote server from linode, using the docker app from the marketplace which runs debain 9. When I issue the docker deploy command, everything is working. However, inspecting the logs of caddy shows that it is not able to complete any of the challenges ([https://pastebin.com/eBHkn8DE](https://Link to full error log form docker service logs).

From inside of my remote server, I have tried to curl different letsencrypt urls and all of them are functional. From outside the server, I am able to see that both port 80 and 443 are open (When I am running my server) when I run nmap 172.105.179.254 or nmap mediacomune.com

Starting Nmap 7.80 ( https://nmap.org ) at 2020-11-13 12:03 AEDT
Nmap scan report for mediacomune.com (172.105.179.254)
Host is up (0.027s latency).
Not shown: 997 closed ports
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https

So to me it seems that both ports are reachable from outside of the server and lets encrypt should be able to reach it. Lets debug results of my my domain name however, shows that AAAA and A are not working through either port.

When I checked my DNA and AAAA records on mxtoolbook it also showed the corrects ip address, so it seems that my A/AAAA records are functional. So I am lost as to why caddy cant issue the certificates. And why letsencrypt cant reach my sever.

rg305 · November 13, 2020, 2:07am

Your nmap test is obviously coming from a trusted network:

curl -Iki 172.105.179.254
curl: (7) Failed to connect to 172.105.179.254 port 80: Connection refused

rg305 · November 13, 2020, 2:13am

Additional tidbit:

Name:    mediacomune.com
Addresses:  2400:8907::f03c:92ff:fe15:b193
          172.105.179.254

curl -Iki [2400:8907::f03c:92ff:fe15:b193]
curl: (7) Failed to connect to 2400:8907::f03c:92ff:fe15:b193 port 80: Connection refused

ibb2 · November 13, 2020, 3:54am

Its probably just due to the fact that caddy is spun down. If you try again it should show both ports open.
And if it is due to me using a trusted network, what can I do to allow untrusted networks to access my sever. As ufw on my linode has default of allowing all incoming and outgoing.

rg305 · November 13, 2020, 4:29am

Better.
But still a problem:

curl -6Iki mediacomune.com/.well-known/acme-challenge/test-1234
HTTP/1.1 403 Forbidden
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: frame-ancestors 'self'
Content-Type: text/html; charset="utf-8"
Content-Length: 2969
Connection: Close

curl -4Iki mediacomune.com/.well-known/acme-challenge/test-1234
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://mediacomune.com/.well-known/acme-challenge/test-1234
Server: Caddy
Date: Fri, 13 Nov 2020 04:01:47 GMT

As LE prefers IPv6 when present, that will cause this to fail.
Remove IPv6 or test/correct the IPv6 address.

griffin · November 13, 2020, 4:15am

Well you don't see that every day.

rg305 · November 13, 2020, 4:17am

It's the last chance warning.
If I have to tell you again...
Stop knocking on my door - that guy doesn't live here anymore!

ibb2 · November 13, 2020, 4:28am

Im pretty sure its a configuration in my caddyfile, do redirect to https.

ibb2 · November 13, 2020, 4:29am

Removing the AAAA records from linodes dns manager fixed the problem. Im curious though I let linode auto fill the ipv6 and added it to etc/hosts like the ipv4, so why would it cause issues?

griffin · November 13, 2020, 4:29am

It's a 308 though instead of the usual 301. Nothing wrong with it (unless you want to change the HTTP request type).

rg305 · November 13, 2020, 4:30am

The IPv6 wasn't being handled properly by the web service.

rg305 · November 13, 2020, 4:33am

308 was a red herring.
The 403 on IPv6 was the problem.

griffin · November 13, 2020, 4:43am

The 308 was never a problem. Just a curious observation.

308 Permanent Redirect (RFC 7538)

The request and all future requests should be repeated using another URI. 307 and 308 parallel the behaviors of 302 and 301, but do not allow the HTTP method to change. So, for example, submitting a form to a permanently redirected resource may continue smoothly.