The arch wiki claims that the "master process" runs as root and whatever else runs as user "http"
I am considering to just switch to apache.
You'd need input from someone who's more of a nginx + arch expert but from the point of view of your certificate and it's renewal it's working ok, it's you're web server config or permissions reading the cert/key files that's broken. Check the symlink for the privkey file as well and check your config. Have a look at the contents of /etc/letsencrypt/options-ssl-nginx.conf in case there's something in there.
The thing where you are seeing a self signed certificate is probably from something like your router. Regarding outside connectivity for https, your system does not allow public internet users to connect to port 443 (https) on that domain. If it works for you you're either connecting to something else (your router) or you have allowed specific geo/ip ranges. I know because I can't telnet to port 443 (which is a one of the simplest connectivity tests you can do).
If you are interested in switching web server also look at Caddy because that handles certs for you. The various web servers (including Apache and nginx) also now have modules and variants that have automated ACME support as well, so you don't strictly need a separate ACME client like Certbot.
privkey exists, i already posted a listing of the directory where it exists.
I have experimented with setting file permissions to all kinds of things, it's not making a difference (nothing positive nor negative). I never changed my web server config since the day I brought this machine up a couple months ago, and it was working until sometime in the past few days.
/etc/letsencrypt/options-ssl-nginx.conf seems not to be interesting, just a warning not to modify it and a list of cyphers.
I am suspicious of caddy. it has an overpolished website with more sponsorship stickers than a race car, but very little information. I will have to do my homework on it.
Sure, everything has a source of sustainability, or it's not just sustained:
Apache (including the web server): Our Sponsors | Apache Software Foundations
nginx was originally a russian open source project and is now a corporate development (still open source) via F5, a billion dollar security tech company.
The source of money is not the problem, the very little information is the problem. I don't want what seems like some automagic "just install it and it works" https server if it also silently and unstoppably prevents the "legacy feature" of plain html. that would be a nightmare scenario for me.
I arrived to sort out nginx but see you've switched to Apache
Yet, the HTTPS URL still times out but the HTTP URL works. Do you want to continue with debugging Apache?
The cert info you posted when connecting using HTTPS did look like a cert used by a router admin panel or similar. Maybe even a firewall doing "HTTPS Inspection" type thing. Perhaps that is interfering with port 443 requests from the public internet reaching your server. If your router has "grabbed" port 443 it may not be forwarding the requests to your server.
I can see from https://crt.sh that you have gotten 6 certs in the last day or so. Being able to get certs is not the issue but configuring your system to reply to HTTPS and use them is. I know this isn't a new opinion than what other have shared. I'm recapping a bit. You shouldn't keep requesting certs as you are already rate limited and now can only get a new request every 34 hours. See: Rate Limits - Let's Encrypt
I assume this is a residential setup. Yes? (Your IP is assigned to Vodafone I see)
Your recent certs
Responses from my own test server
curl -i http://winged-horse.fyi
HTTP/1.1 200 OK
Date: Mon, 17 Nov 2025 03:26:53 GMT
Server: Apache/2.4.65 (Unix)
curl -i -m7 https://winged-horse.fyi
curl: (28) Connection timed out after 7001 milliseconds
I considered switching to apache, I briefly tested it but certbot shat itself and exploded "SSLCertificateFile: file '/etc/httpd/conf/server.crt' does not exist or is empty"
So I would rather stick with nginx.
I do not believe https not working is caused by my router, given that nginx is complaining about not being able to open the certs.
nginx did not complain about that when you ran sudo nginx -t
Which indicates nginx may not have been started correctly. If Apache is out of the picture we should check what nginx looks like. What do these show?
sudo ss -pant | grep -Ei 'nginx|:80|:443'
sudo systemctl status --no-pager -l nginx
sudo ps -eF | grep nginx | grep -v grep
While sudoing "nginx -t" doesn't error trying to read the certs, starting nginx with sudo still doesn't result in ssl access.
Here's the output:
[kyle@Swiftwind nginx]$ sudo ss -pant | grep -Ei 'nginx|:80|:443'
LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=10243,fd=5),("nginx",pid=10242,fd=5),("nginx",pid=10241,fd=5),("nginx",pid=10239,fd=5),("nginx",pid=10238,fd=5),("nginx",pid=10237,fd=5),("nginx",pid=10236,fd=5),("nginx",pid=10235,fd=5),("nginx",pid=10234,fd=5))
LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=10243,fd=6),("nginx",pid=10242,fd=6),("nginx",pid=10241,fd=6),("nginx",pid=10239,fd=6),("nginx",pid=10238,fd=6),("nginx",pid=10237,fd=6),("nginx",pid=10236,fd=6),("nginx",pid=10235,fd=6),("nginx",pid=10234,fd=6))
ESTAB 0 0 192.168.1.2:80 192.168.1.1:49898 users:(("nginx",pid=10235,fd=18))
ESTAB 0 0 192.168.1.2:45228 172.67.188.91:443 users:(("transmission-da",pid=900,fd=91))
TIME-WAIT 0 0 192.168.1.2:80 91.231.89.17:47421
ESTAB 0 0 [2a0a:ef40:15fc:4601:c3c3:486f:4299:938e]:46558 [2606:4700:3034::6815:30ec]:443 users:(("transmission-da",pid=900,fd=97))
● nginx.service - nginx web server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; preset: disabled)
Active: active (running) since Mon 2025-11-17 09:26:37 GMT; 3min 6s ago
Invocation: df7609b85fe34b01968236e2b324436a
Process: 10233 ExecStart=/usr/bin/nginx (code=exited, status=0/SUCCESS)
Main PID: 10234 (nginx)
Tasks: 9 (limit: 16538)
Memory: 9M (peak: 9.5M)
CPU: 42ms
CGroup: /system.slice/nginx.service
├─10234 "nginx: master process /usr/bin/nginx"
├─10235 "nginx: worker process"
├─10236 "nginx: worker process"
├─10237 "nginx: worker process"
├─10238 "nginx: worker process"
├─10239 "nginx: worker process"
├─10241 "nginx: worker process"
├─10242 "nginx: worker process"
└─10243 "nginx: worker process"
Nov 17 09:26:37 Swiftwind systemd[1]: Starting nginx web server...
Nov 17 09:26:37 Swiftwind nginx[10233]: 2025/11/17 09:26:37 [warn] 10233#10233: could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size
Nov 17 09:26:37 Swiftwind nginx[10233]: 2025/11/17 09:26:37 [warn] 10233#10233: conflicting server name "winged-horse.fyi" on 0.0.0.0:80, ignored
Nov 17 09:26:37 Swiftwind systemd[1]: Started nginx web server.
root 10234 1 0 6399 3056 2 09:26 ? 00:00:00 nginx: master process /usr/bin/nginx
http 10235 10234 0 6494 5192 3 09:26 ? 00:00:00 nginx: worker process
http 10236 10234 0 6494 4740 7 09:26 ? 00:00:00 nginx: worker process
http 10237 10234 0 6494 5192 4 09:26 ? 00:00:00 nginx: worker process
http 10238 10234 0 6494 4740 0 09:26 ? 00:00:00 nginx: worker process
http 10239 10234 0 6494 4740 4 09:26 ? 00:00:00 nginx: worker process
http 10241 10234 0 6494 4740 7 09:26 ? 00:00:00 nginx: worker process
http 10242 10234 0 6494 4740 1 09:26 ? 00:00:00 nginx: worker process
http 10243 10234 0 6494 4676 3 09:26 ? 00:00:00 nginx: worker process
I was expecting a Let's Encrypt certificate or maybe a self-signed certificate generated by certbot. This C=BE,ST=Antwerp,L=Edegem,O=Technicolor you see in your browser is very peculiar.
So I suspect this is a router firewall/NAT issue.
That would explain why we can't connect and why you're seeing this weird cert (the router is sending :443 traffic to a different device than you expect).
A simple test to verify this theory would be to shutdown nginx and see if you still get a certificate error or if it changes to a connection refused/timeout.
with nginx running:
inside my lan I get that "technicolor" cert. Outside my lan, no connection to 443
without nginx running: inside my lan I get that "technicolor" cert. Outside my lan, no connection to 443 (also no connection to port 80, but we expect that when no http server is there)
so I think you are right that my shitty consumer router has now decided on it's own to intercept port 443. I checked even the usual 192.168.1.1 config page with https and it provides the same bogus self signed cert.
Some time recently I think they forced a firmware update (which annoyed me massively as it killed my home internet for an entire hour), I think they have botched it. my port forwarding rules are all still correct.
however, that does not explain the nginx -t key file permission error. is that just a red herring caused by invoking it as my own user vs invoking it as root?
Yep. The nginx map page says the following:
-tDo not run, just test the configuration file. nginx checks the configuration file syntax and then tries to open files referenced in the configuration
file.
It probably wants to open the referenced files to check their syntax as well.
Nginx also gave this warning when you ran -t without sudo, so it's taking the current user into account when loading the configuration even for testing:
the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
Well, I just went to the internal ip of the webserver and firefox found the correct lets encrypt ssl cert (and then complained because it wasn't coming through my domain name, hah!)
so I think it's 100% confirmed to be a router issue, thank you for helping me figure that out. also (mikemcq, webprofusion). As terrible freebie routers handed out by british ISPs is not an SSL cert provider's responsibility I'll take up no more of your time. Now I'll go fight my ISP.
Enjoy the rest of your day.
Glad @dextercd was able to help identify the culprit.
And, good try on a local IP Firefox test. I was going to suggest a similar test. Just for your future reference you can try, with nginx running:
echo | openssl s_client -connect 127.0.0.1:443 --servername winged-horse.fyi
It should connect and show what cert nginx uses on port 443
For your Firefox test you could make an entry in your hosts file for your domain name with your local IP address. Then using the domain name in Firefox won't issue that mis-matched name error.
I think it likely an admin panel on your router is using port 443. Getting that modified is best of course. But, you could change nginx to use a different port. Then access it using something like: https://winged-horse.fyi:8443 Yeah, it's unpleasant but another way to test if nothing else
It turned out that the router fell victim to the ISP remotely flashing it's firmware. Whatever they did caused it to ignore the port routing that were set in it's admin panel, steal anything trying to come in to 443 from the outside world, and also trash it's own certs.
A factory reset and then update to even newer firmware got it going again.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
