My domain is:
forum.melonland.net
letters.melonland.net
toot.melonland.net
etc
My web server is (include version): nginx/1.27.1
The operating system my web server runs on is (include version): Ubuntu 24
I can login to a root shell on my machine (yes or no, or I don't know): yes
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.9.0
I have a single cert that was originally created for one subdomian (toot.melonland.net) and later expanded to include many more subdomains and separate domains after the majority of my sites were consolidated on a single machine. In total around 15 subdomains and domains are hosted via nginx, a mix of direct hosts and reverse proxies. They all share the same cert that was generated and expanded as certbot suggested.
e.g.
server {
server_name letters.melonland.net;
xxxxx
listen 443 ssl; # managed by Certbot
listen [::]:443 ssl;
ssl_certificate /etc/letsencrypt/live/toot.melonland.net/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/toot.melonland.net/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
Since the server move I have started seeing a "421 Misdirected Request" error coming up when moving from one subdomain to another. eg going from a link on subdomain A to subdomain B - the error goes away if nginx is restarted or if I use another browser.
I've researched the issue widely and received a mix of replies suggesting its caused by some confusion of the hostname not matching the certificate correctly.
The two key solutions suggested were to manually set the SSL name and header in nginx; for example:
However these solutions seem to have no effect on the issue at all.
At this point I'm at a loss about how to deal with this issue and Im unsure if I'm using certbot in the wrong way or have setup this certificate incorrectly for the number of hosts my cert needs to cover.
It would be helpful to know if there is a better way I can configure my certificates to avoid this issue, or if the issue is related to the certificate configuration at all (verses general nginx configuration)
I cannot always intentionally replicate it myself, sometimes it happens very frequently and other times it vanishes for days and appears resolved before returning.
That said; it seems to happen most frequently when a page on one subdomain contains an iframe that loads a page on a separate subdomain.
I tried to setup a worst case situation test.php that loads multiple APIs and webpages simultaneously across 3 subdomains, but I cant get it to trigger the error on demand. https://forum.melonland.net/test.php
Three domains are loaded in iframes, two are PHP. One PHP site is loading the SSI file locally, while the other is also loading it from a separate subdomain. The third is a NodeJS site running Etherpad.
Outside of a synthetic test this kind of cross site loading seems to be what causes the 421 error most frequently, but as I said I cant replicate it here.
To follow this up: last night I observed the 421 error on an image caching page that has no programatic connection to the rest of my sites and lives inside docker. This makes me think it's not related to the code itself.
I also have a suspicion that the error moves; as in, it will affect site A, then move to site B, then move to site C. Like a rolling blackout; I'm not sure if there is any reality in that, but it does seem possible that the fault may only be effecting one domain at any given time.
As in does it go away with a browser reload or a browser cache clear? No it does not; it continues to return a 421 error; usually for anywhere between 10 minutes to an hour.
Also in relation to the 421 error on the image caching page; it was not just one image, it was every image hosted in the cache that returned a 421 (each using a separate query string) - they all broke simultaneously and all came back simultaneously when I checked again later.
Thats why I came to this forum since it seems to be something to do with the certificate or how the browser is interacting with the certificate that is unique to each browser. Since browser cache clears don't fix it but switching the entire browser does.
I also recall that some of the StackOverflow topics related to this error are focused on Chrome or Webkit browsers; I'm seeing it in Safari, however I don't recall ever seeing it in Firefox.
I don't see how the certificate could be connected to the HTTP 421.
A server that is mis-configured? Sure. But not the cert itself. I didn't see anything obvious in the nginx -T output but I only skimmed it. A fair number of moving parts and mix of IPv4 and IPv6 listens.
Do the nginx error logs show anything at the time of the 421? Seems they probably would. You may need to lower the level of the error_log (not sure).
What if you watch the browser's developer tool and check the URLs that are being sent that get the 421. Maybe they are not as you expect.
There were no errors on a standard log level, I lowered it down to "info" and I am seeing some results, I'm not sure if they are directly related.
2025/02/01 19:01:19 [error] 530192#530192: *873 connect() failed (111: Connection refused) while connecting to upstream, client: 2a01:a8:dc3:2005::322, server: pad.melonland.net, request: "GET /api/1.2.13/createGroupPad?apikey=xxx&groupID=g.j8Z4yMg17Suep3b4&padName=index HTTP/1.1", upstream: "http://[::1]:9001/api/1.2.13/createGroupPad?apikey=xxx&groupID=g.j8Z4yMg17Suep3b4&padName=index", host: "pad.melonland.net"
2025/02/01 19:01:19 [warn] 530192#530192: *873 upstream server temporarily disabled while connecting to upstream, client: 2a01:a8:dc3:2005::322, server: pad.melonland.net, request: "GET /api/1.2.13/createGroupPad?apikey=xxx&groupID=g.j8Z4yMg17Suep3b4&padName=index HTTP/1.1", upstream: "http://[::1]:9001/api/1.2.13/createGroupPad?apikey=xxx&groupID=g.j8Z4yMg17Suep3b4&padName=index", host: "pad.melonland.net"
2025/02/01 19:01:19 [info] 530190#530190: *271 client attempted to request the server name for which the negotiated protocol is disabled while reading client request headers, client: 2a02:8084:90c0:e900:dd00:5d19:69a2:542a, server: pad.melonland.net, host: "pad.melonland.net", referrer: "https://forum.melonland.net/"
I was also able to screenshot the browser inspector on a 421-ing page and it seems normal; I would note that the certificate is titled "loom.cafe" which is not the domain I would have expected as the certificate title domain, but otherwise it seems normal.
In this situation pad.php is attempting to load pad.melonland (the failing page/subdomain) in an iframe:
One thing I did notice is that the pad.melonland.net has an IP6 address whereas its being loaded in an iframe from an IP4 site (forum.melonland.net is always IP4)
Why not? That is the CN for the cert issued Jan19. You could verify using https://crt.sh but it seems down at the moment.
The Certificate Name used by Certbot is a local file name. As you change the domains in the cert it may preserve that local file while the CN name chosen by Let's Encrypt may then change.
From a client a mix of IPv4 and v6 is fine. You just need to make sure all your flows on the server are right.
This is all fairly outside our normal scope. Someone may help anyway. Just want to set your expectations. A server admin forum probably best.
Thanks, that does at least narrow down potential causes! My initial reaction was to consider separating out each domain to its own certificate; however since you have not suggested this, I will not attempt it as a solution for now.
It does occur to me that this issue started after I added IPv6 support to most of the sites; you say its not an issue, but perhaps its playing some role. I may try removing IPv6 support.
That every inbound (IPv4 and IPv6) reaches the correct server block and for any proxy it goes to the right place. A common mistake is having IPv6 for a domain but not having a corresponding listen statement for it. Then, nginx uses a default IPv6 server block for that.
I don't know that this is your problem. I am just describing what I meant by "flows"
One odd thing I see here Permanent link to this check report is most locations from around the world have Results of 46.22.135.223 yet a few have 46.22.135.223, 2a01:a8:dc3:2005::322
I just dropped IPv6 routing from the DNS table; so I imagine it's gradually disappearing now. I'm going to give it a day or two to see if it has any effect on the frequency of 421 errors before adding it back in.
Dropping IPv6 is more of a patch than a real solution; but it does at least clarify if the issue is related to IPv's - I'll report back here if it has any noticeable impact!
Reporting back with an update for anyone encountering this thread; the issue is not related to IPv6.
However I believe it has been solved! (At least it has not appeared in days)
It seems to have been caused by http2 being enabled on some sites and disabled on others. I initially thought it was caused by IPv6 because it started after I moved host and added IPv6 to my sites; however that move also meant nginx jumped to a new version that my old host did not support. In older versions of nginx, if one site had http2 enabled it was forcibly enabled on all sites. Newer versions of nginx allow for per-site http2 settings - this meant that sites where it was explicitly enabled used it, while others did not. Enabling http2 on all sites seems to have (so far) resolved the issue