Apparently spurious "Redirect loop detected" on host with both IPv4 & IPv6 addresses

fil · March 27, 2023, 2:42pm

My domain is: id.hands.com & fast.hands.com (among others that have shown this behavior in the past)

I ran this command: dehydrated -c

It produced this output:

[deleted: previous successful renewals and/or skipping up-to-date certs for other domains]
...
Processing fast.hands.com
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Apr 25 03:33:26 2023 GMT (Less than 30 days). Renewing!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 1 authorizations URLs from the CA
 + Handling authorization for fast.hands.com
 + 1 pending challenge(s)
 + Deploying challenge tokens...
 + Responding to challenge for fast.hands.com authorization...
 + Cleaning challenge tokens...
 + Challenge validation has failed :(
ERROR: Challenge is invalid! (returned: invalid) (result: ["type"]	"http-01"
["status"]	"invalid"
["error","type"]	"urn:ietf:params:acme:error:connection"
["error","detail"]	"78.129.164.123: Fetching https://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ:
Redirect loop detected"
["error","status"]	400
["error"]	{"type":"urn:ietf:params:acme:error:connection","detail":"78.129.164.123: Fetching
https://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ: Redirect loop detected","status":400}
["url"]	"https://acme-v02.api.letsencrypt.org/acme/chall-v3/214324257637/UsGFqg"
["token"]	"YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ"
["validationRecord",0,"url"]	"http://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ"
["validationRecord",0,"hostname"]	"fast.hands.com"
["validationRecord",0,"port"]	"80"
["validationRecord",0,"addressesResolved",0]	"78.129.164.123"
["validationRecord",0,"addressesResolved",1]	"2001:1b40:5600:ff80:f8ee::1"
["validationRecord",0,"addressesResolved"]	["78.129.164.123","2001:1b40:5600:ff80:f8ee::1"]
["validationRecord",0,"addressUsed"]	"2001:1b40:5600:ff80:f8ee::1"
["validationRecord",0]	{"url":"http://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ","hostname":"fast.hands.com","port":"80","addressesResolved":["78.129.164.123","2001:1b40:5600:ff80:f8ee::1"],"addressUsed":"2001:1b40:5600:ff80:f8ee::1"}
["validationRecord",1,"url"]	"https://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ"
["validationRecord",1,"hostname"]	"fast.hands.com"
["validationRecord",1,"port"]	"443"
["validationRecord",1,"addressesResolved",0]	"78.129.164.123"
["validationRecord",1,"addressesResolved",1]	"2001:1b40:5600:ff80:f8ee::1"
["validationRecord",1,"addressesResolved"]	["78.129.164.123","2001:1b40:5600:ff80:f8ee::1"]
["validationRecord",1,"addressUsed"]	"2001:1b40:5600:ff80:f8ee::1"
["validationRecord",1]	{"url":"https://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ","hostname":"fast.hands.com","port":"443","addressesResolved":["78.129.164.123","2001:1b40:5600:ff80:f8ee::1"],"addressUsed":"2001:1b40:5600:ff80:f8ee::1"}
["validationRecord",2,"url"]	"http://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ"
["validationRecord",2,"hostname"]	"fast.hands.com"
["validationRecord",2,"port"]	"80"
["validationRecord",2,"addressesResolved",0]	"78.129.164.123"
["validationRecord",2,"addressesResolved",1]	"2001:1b40:5600:ff80:f8ee::1"
["validationRecord",2,"addressesResolved"]	["78.129.164.123","2001:1b40:5600:ff80:f8ee::1"]
["validationRecord",2,"addressUsed"]	"78.129.164.123"
["validationRecord",2]	{"url":"http://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ","hostname":"fast.hands.com","port":"80","addressesResolved":["78.129.164.123","2001:1b40:5600:ff80:f8ee::1"],"addressUsed":"78.129.164.123"}
["validationRecord"]	[{"url":"http://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ","hostname":"fast.hands.com","port":"80","addressesResolved":["78.129.164.123","2001:1b40:5600:ff80:f8ee::1"],"addressUsed":"2001:1b40:5600:ff80:f8ee::1"},{"url":"https://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ","hostname":"fast.hands.com","port":"443","addressesResolved":["78.129.164.123","2001:1b40:5600:ff80:f8ee::1"],"addressUsed":"2001:1b40:5600:ff80:f8ee::1"},{"url":"http://fast.hands.com/.well-known/acme-challenge/YPtjiruRqXbrdWU5o6K5LttHd3PkQLmaW7yZZCxNFqQ","hostname":"fast.hands.com","port":"80","addressesResolved":["78.129.164.123","2001:1b40:5600:ff80:f8ee::1"],"addressUsed":"78.129.164.123"}]
["validated"]	"2023-03-27T03:33:30Z")

My web server is (include version): nginx 1.18.0-6.1+deb11u3

The operating system my web server runs on is (include version): Debian GNU/Linux 11.6

My hosting provider, if applicable, is: (co-lo server)

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

Dehydrated version: 0.7.0
GIT-Revision: unknown

OS: Debian GNU/Linux 11 (bullseye)
Used software:
 bash: 5.1.4(1)-release
 curl: 7.74.0
 awk: GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)
 sed: sed (GNU sed) 4.7
 mktemp: mktemp (GNU coreutils) 8.32
 grep: grep (GNU grep) 3.6
 diff: diff (GNU diffutils) 3.7
 openssl: OpenSSL 1.1.1n  15 Mar 2022

I don't actually need help with these domains now, because I've managed to get them all updated, but thought I ought to mention what worked, as it may point towards a bug in the validation server.

While trying various minor modifications to the server setup, and retrying, the certificate for fast.hands.com was issued -- I'm pretty sure that nothing important changed to cause that, since it then failed in exactly the same way with id.hands.com and making the same changes there had no effect.

I noted that what appeared to be going on is that the validation server connects to my server's IPv6 address, using HTTP on port 80, and is redirected (301) to HTTPS on 443, which appears to work. Then it connects again, this time to port 80 on my IPv4 address, and is again redirected (301) to HTTPS, but this time claims there's a Redirect loop and gives up.

I'm pretty sure that there's not really an issue at my end, because I persuaded it to work for id.hands.com by changing its DNS entry to only list IPv4 -- at which point it worked as soon as the DNS update had propagated (having failed just before when I tried it too early, failing in the IPv4 attempt).

I'm happy to report this as a bug if someone can point me at the relevant bug tracker, and if there's anything I can do to help diagnose it at my end, just ask.

Bruce5051 · March 27, 2023, 3:03pm

Hello @fil, welcome to the Let's Encrypt community.

From the log you provided I would say you have an issue with redirects; all 3 of the IPv4 Address must give the same responses.

And here is are the DNS Records DNS Spy report for sight--care.com

fil · March 27, 2023, 3:09pm

Hi @Bruce5051,

Did you read the paragraphs at the bottom?

... in which I say how I fixed it, which didn't involve removing any redirects, but rather just removing the IPv6 address, at which point the "Redirect loop" it was apparently complaining about seems to have evaporated -- suggesting to me that it didn't exist in the first place -- hence my post.

Cheers, Phil.

Bruce5051 · March 27, 2023, 3:12pm

It seems the web server configuration is not properly accounting for both IPv4 and IPv6 Address together.

rg305 · March 27, 2023, 3:17pm

You removed the IPv6 address...
Then the "Redirection loop" evaporated...
From which you then understood that it never existed in the first place?
If so, then you can put back the IPv6 address; As it won't cause any such loop.
OR will it?

MikeMcQ · March 27, 2023, 3:31pm

Right now I don't see a redirect loop for fast.hands.com or id.hands.com. fast has both IPv4/6 via a CNAME but id just IPv4 via A record. Does the problem persist on both?

That redirect loop may be coming from dehydrated itself. I think it does a pre-check for HTTP Challenges. Can you disable that and see if Let's Encrypt servers issue the same error.

Also, your DNS server ns1.hands.com has some responsiveness issues on TCP. See:
https://dnsviz.net/d/fast.hands.com/dnssec/

MikeMcQ · March 27, 2023, 3:44pm

The Let's Debug test site results show a comms problem with IPv6. It issues a test HTTP request to a URL like an HTTP Challenge. The request to the IPv6 for fast times out while IPv4 is fine.

Let's Debug also uses the Let's Encrypt Staging System to try to issue a cert. It is expected to fail the challenge but it does not fail with the IPv6 comms. It fails with the expected 404 because the challenge file token does not exist (because it cant place a file on your server of course).

This looks like either a firewall rule for IPv6 or even possibly a comms routing problem from Let's Debug to your server. L/D uses AWS but my own test from an AWS system in same AWS region works fine.

That said, since the actual Let's Encrypt servers reached you fine on IPV6 you still may be able to get a cert. But, that Let's Debug can't reach you points to some sort of general comms issue.

fil · March 27, 2023, 4:15pm

AFAIK the only redirection on that server is for some sites from http to https -- once on https it just serves up the content, so I really don't think there are any loops, in fact (the sites do work generally).

BTW Is it normal behaviour for it to first do the challenge via IPv6, and then IPv4, or is the fact that it's continuing to IPv4 an indication that it wasn't happy about what happened during the IPv6 challenge?

I just had a look at the dehydrated source, and there's no occurrence of the word "Redirect", so I doubt the message is coming from there (unless it's being generated by curl, say) but my impression was that that is being generated in the server.

This seems the most likely origin of that message:

github.com

letsencrypt/boulder/blob/46323d25beef3e16c92d6175a1cdd98de0e53b18/va/http.go#L555


      
          		// those in the redirect path
          		redirQuery := ""
          		if req.URL.RawQuery != "" {
          			redirQuery = req.URL.RawQuery
          		}
          
          
		// Check for a redirect loop. If any URL is found twice before the
          		// redirect limit, return error.
          		for _, record := range records {
          			if req.URL.String() == record.URL {
          				return berrors.ConnectionFailureError("Redirect loop detected")
          			}
          		}
          
          
		// Create a validation target for the redirect host. This will resolve IP
          		// addresses for the host explicitly.
          		redirTarget, err := va.newHTTPValidationTarget(ctx, redirHost, redirPort, redirPath, redirQuery)
          		if err != nil {
          			return err
          		}

Are you really trying to say that a 130ms DNS response time is relevant? (that server averages about 280 Mbit/s, so it's not exactly idle)

fil · March 27, 2023, 4:17pm

OK, thanks for the pointer @MikeMcQ -- Let's Debug looks like it'll help track this down -- I'll see what I can discover...

MikeMcQ · March 27, 2023, 4:26pm

It was not responding at all when I looked but seems fine now.

Agree the error looks probably from Boulder given your findings.

IPv6, if AAAA present, is tried first by Let's Encrypt Servers. Certain IPv6 failures will have it retry with IPv4. The retriable failures generally are clean timeouts.

Let's Encrypt servers use the authoritive DNS servers so you don't have to wait for world-wide propagation (just between your auth servers)

Good idea to review with Let's Debug to see if can help isolate. Does the failed redirect persist? I can't reproduce it.

fil · March 27, 2023, 4:55pm

I cannot reproduce it from elsewhere, but Let's Debug is still showing it -- I'm a little suspicious that the logging seems a bit odd, as it seems that some of the time the messages are going to the general log file, rather than the virtual-host-specific one, so it's possible that there's something about the config that's not quite right... [that was a red-herring - I'd simply failed to add the log settings to the http stanza for that vhost]

MikeMcQ · March 27, 2023, 5:09pm

I don't see any evidence this is wrong but have you made certain that each server block has a listen for both IPv4 and IPv6?

Bruce5051 · March 27, 2023, 6:40pm

Yes I did.

fil · March 28, 2023, 12:59pm

Thanks again @MikeMcQ for pointing out letsdebug.net, that allowed me to work out what was going on in the end.

The problem seems to be that the server in question is a VM, and in addition to the Internet facing interface, it has another interface that is attached to a virtual bridge on the physical host (that's just used for intra-VM communication, and has no route to the Internet).

It seems that systemd-networkd defaults to bringing up IPv6 on interfaces, and that for some reason some small portion of the IPv6 traffic was being routed to the not-on-the-internet interface. This seems to relate to specific remote addresses, since IPv6 was mostly working, but there's some randomness about which remote addresses were affected, since at one point everything was working fine from a machine from which I was testing, and then later it wasn't.

Anyway, I ended up telling networkd to leave IPv6 unconfigured on the internal interface, which is what I actually wanted anyway, because there's nothing configured to use IPv6 on that bridge, and now it all seems to be working fine.

In case anyone's interested, the incantation required to disable IPv6 is to add the following to the [Network] section in the relevant interface's config under /etc/systemd/network:

LinkLocalAddressing=no
IPv6AcceptRA=no

Having said all that, the fact that boulder tells one that there's a Redirect loop, when what is in fact going on is that it's not getting any replies to its IPv6 requests, strikes me as a bug.
It's completely misleading -- there never was a Redirect loop.

P.S. on reflection, I'm not 100% sure that I confirmed that all reply packets were going to the wrong interface, so if it were somehow possible for only the HTTPS replies to be affected by this (I don't really see how that would occur) then it could have been that the redirects in HTTP were getting through, but the subsequent HTTPS replies were not, and I don't think I'd have noticed that subtlety.

system · April 27, 2023, 1:00pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.