Certbot redirect loop

Spansh · June 30, 2022, 10:18am

So I have several servers configured similarly. All of them redirect http directly to https. On the main server the certbot renewal process has been running for years without a problem, upon renewal I see a request to the http server, it gets a 301 response and I then see a request to the https server and things work as normal.

On the server I'm attempting to refresh I see the initial request to the http server and the 301 response. I then do not see another request and I simply see a redirect loop detected error. Attempting this using curl I see appropriate responses and they work as expected:
curl http://kibana.spansh.co.uk/.well-known/acme-challenge/kOonsBWq4xZeV7RLzKn1bwVNM2i9s0vkOFulVfAUTsM -v
responds with the following header
Location: https://kibana.spansh.co.uk/.well-known/acme-challenge/kOonsBWq4xZeV7RLzKn1bwVNM2i9s0vkOFulVfAUTsM

My domain is: kibana.spansh.co.uk

I ran this command: certbot certonly --webroot --webroot-path /var/www/ssl-proof/ --force-renewal --post-hook '/etc/init.d/nginx reload' --dry-run --cert-name kibana.spansh.co.uk --debug-challenges -v

It produced this output:

Challenge failed for domain kibana.spansh.co.uk
http-01 challenge for kibana.spansh.co.uk

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: kibana.spansh.co.uk
  Type:   connection
  Detail: 136.243.20.72: Fetching https://kibana.spansh.co.uk/.well-known/acme-challenge/ZnvEukHLrVvoBJL5V-VHMyXIZZTYkswH4eWcrCXhj9Q: Redirect loop detected

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

My web server is (include version): nginx 1.22.0

The operating system my web server runs on is (include version): Linux Ubuntu bullseye

My hosting provider, if applicable, is: Hetzner

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.28.0

Osiris · June 30, 2022, 10:28am

Currently, I'm not getting a redirect loop, but a 404 file not found when trying the challenge. Probably due to the fact Certbot isn't running any longer. Which makes this kinda difficult to debug.

First thing I'd think is rogue .htaccess files redirecting or perhaps misconfigured Wordpress.

Spansh · June 30, 2022, 10:29am

I've currently paused the challenge before sent to the CA at http://kibana.spansh.co.uk/.well-known/acme-challenge/CRnhQDdm1gPWsnkM5zjKngQrI9TG-frvrD4T-tdA7k8

It's not running Wordpress, in fact there's not really much running on the rest of the site at the moment. The well-known direction (location) is specifically handled as a separate docroot, so it's purely nginx, There isn't a htaccess, though the rest of the site (not under .well-known) is behind basic http auth.

Osiris · June 30, 2022, 10:31am

Weird, I can just reach that file perfectly:

osiris@erazer ~ $ curl -Lv http://kibana.spansh.co.uk/.well-known/acme-challenge/CRnhQDdm1gPWsnkM5zjKngQrI9TG-frvrD4T-tdA7k8
*   Trying 2a01:4f8:211:1147::2:80...
* Connected to kibana.spansh.co.uk (2a01:4f8:211:1147::2) port 80 (#0)
> GET /.well-known/acme-challenge/CRnhQDdm1gPWsnkM5zjKngQrI9TG-frvrD4T-tdA7k8 HTTP/1.1
> Host: kibana.spansh.co.uk
> User-Agent: curl/7.83.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.22.0
< Date: Thu, 30 Jun 2022 10:29:54 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
< Location: https://kibana.spansh.co.uk/.well-known/acme-challenge/CRnhQDdm1gPWsnkM5zjKngQrI9TG-frvrD4T-tdA7k8
< 
* Ignoring the response-body
* Connection #0 to host kibana.spansh.co.uk left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://kibana.spansh.co.uk/.well-known/acme-challenge/CRnhQDdm1gPWsnkM5zjKngQrI9TG-frvrD4T-tdA7k8'
*   Trying 2a01:4f8:211:1147::2:443...
* connect to 2a01:4f8:211:1147::2 port 443 failed: Connection refused
*   Trying 136.243.20.72:443...
* Connected to kibana.spansh.co.uk (136.243.20.72) port 443 (#1)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=kibana.spansh.co.uk
*  start date: Apr 18 09:06:04 2022 GMT
*  expire date: Jul 17 09:06:03 2022 GMT
*  subjectAltName: host "kibana.spansh.co.uk" matched cert's "kibana.spansh.co.uk"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /.well-known/acme-challenge/CRnhQDdm1gPWsnkM5zjKngQrI9TG-frvrD4T-tdA7k8]
* h2h3 [:scheme: https]
* h2h3 [:authority: kibana.spansh.co.uk]
* h2h3 [user-agent: curl/7.83.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x560767204220)
> GET /.well-known/acme-challenge/CRnhQDdm1gPWsnkM5zjKngQrI9TG-frvrD4T-tdA7k8 HTTP/2
> Host: kibana.spansh.co.uk
> user-agent: curl/7.83.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx/1.22.0
< date: Thu, 30 Jun 2022 10:29:54 GMT
< content-type: application/octet-stream
< content-length: 87
< last-modified: Thu, 30 Jun 2022 10:28:28 GMT
< etag: "62bd7acc-57"
< strict-transport-security: max-age=15768000
< x-frame-options: SAMEORIGIN
< accept-ranges: bytes
< 
* Connection #1 to host kibana.spansh.co.uk left intact
CRnhQDdm1gPWsnkM5zjKngQrI9TG-frvrD4T-tdA7k8.wZUQxCp8nlYOVuACXDIbnCqAGnv5WOcqMyWQhck3eusosiris@erazer ~ $

The thing I notice is your webserver not listening on IPv6 port 443, but I don't think that leads to a redirect loop (which I'm also not seeing).

Spansh · June 30, 2022, 10:41am

Ok, after your IPv6 prod I've checked it and I'd forgotten to add a listen [::]:443 ssl http2; to my https side.

After I fixed that it has allowed me to renew fine. I suspect that the error reporting on the certbot/letsencrypt side was flawed. It saw it could request on IPv6 on port 80 and the failure on port 443 caused it to think it was a redirect loop.

Osiris · June 30, 2022, 10:43am

Not sure if IPv6 was the issue. I could get the challenge fine without your fix and Boulder, the software doing the Let's Encrypt validation, would definitely give a different error if there was something else than an actual redirect loop.

But I guess it's a good thing it worked now? Although I'm not sure you're definitely out of the woods for future renewals, keep a close eye I'd say

Nummer378 · June 30, 2022, 10:44am

Let's Encrypt does not implement a fallback to IPv4 when following redirects, so that was probably what went wrong.

Osiris · June 30, 2022, 10:44am

Hm, but how would that give a redirect loop? It would just plain fail, right?

Spansh · June 30, 2022, 11:02am

I suspect perhaps a logic error on the Lets Encrypt side, http worked and redirected, https failed due to no connection and it reported redirect loop when it potentially should have been caught another way.

On an unrelated note, my god the amount of attempted scans/hacks/nmaps/probes on that server since I posted here (it always had some, I just saw an increase of a few thousand percent).

Osiris · June 30, 2022, 11:13am

Threads on this site are also indexed by search engines VERY quickly, so I'm not surprised.

Also note that all certificates are logged to Certificate Transparancy Logs and hackers can easily monitor those logs for new hostnames and try out stuff. For example, if your Wordpress site isn't secure and "wide open", but you get a public certificate anyway, this can lead to trouble.

Which emphasises the importance of having your server and public applications secure FIRST before you put it out there on the internet.

system · July 30, 2022, 11:14am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.