Certificate is installed but suddenly stopped working

The arch wiki claims that the "master process" runs as root and whatever else runs as user "http"

1 Like

I am considering to just switch to apache.

You'd need input from someone who's more of a nginx + arch expert but from the point of view of your certificate and it's renewal it's working ok, it's you're web server config or permissions reading the cert/key files that's broken. Check the symlink for the privkey file as well and check your config. Have a look at the contents of /etc/letsencrypt/options-ssl-nginx.conf in case there's something in there.

The thing where you are seeing a self signed certificate is probably from something like your router. Regarding outside connectivity for https, your system does not allow public internet users to connect to port 443 (https) on that domain. If it works for you you're either connecting to something else (your router) or you have allowed specific geo/ip ranges. I know because I can't telnet to port 443 (which is a one of the simplest connectivity tests you can do).

If you are interested in switching web server also look at Caddy because that handles certs for you. The various web servers (including Apache and nginx) also now have modules and variants that have automated ACME support as well, so you don't strictly need a separate ACME client like Certbot.

3 Likes

privkey exists, i already posted a listing of the directory where it exists.

I have experimented with setting file permissions to all kinds of things, it's not making a difference (nothing positive nor negative). I never changed my web server config since the day I brought this machine up a couple months ago, and it was working until sometime in the past few days.

/etc/letsencrypt/options-ssl-nginx.conf seems not to be interesting, just a warning not to modify it and a list of cyphers.

I am suspicious of caddy. it has an overpolished website with more sponsorship stickers than a race car, but very little information. I will have to do my homework on it.

Sure, everything has a source of sustainability, or it's not just sustained:

Apache (including the web server): Our Sponsors | Apache Software Foundations

nginx was originally a russian open source project and is now a corporate development (still open source) via F5, a billion dollar security tech company.

1 Like

The source of money is not the problem, the very little information is the problem. I don't want what seems like some automagic "just install it and it works" https server if it also silently and unstoppably prevents the "legacy feature" of plain html. that would be a nightmare scenario for me.

I arrived to sort out nginx but see you've switched to Apache :slight_smile: Yet, the HTTPS URL still times out but the HTTP URL works. Do you want to continue with debugging Apache?

The cert info you posted when connecting using HTTPS did look like a cert used by a router admin panel or similar. Maybe even a firewall doing "HTTPS Inspection" type thing. Perhaps that is interfering with port 443 requests from the public internet reaching your server. If your router has "grabbed" port 443 it may not be forwarding the requests to your server.

I can see from https://crt.sh that you have gotten 6 certs in the last day or so. Being able to get certs is not the issue but configuring your system to reply to HTTPS and use them is. I know this isn't a new opinion than what other have shared. I'm recapping a bit. You shouldn't keep requesting certs as you are already rate limited and now can only get a new request every 34 hours. See: Rate Limits - Let's Encrypt

I assume this is a residential setup. Yes? (Your IP is assigned to Vodafone I see)

Your recent certs

Responses from my own test server

curl -i http://winged-horse.fyi
HTTP/1.1 200 OK
Date: Mon, 17 Nov 2025 03:26:53 GMT
Server: Apache/2.4.65 (Unix)

curl -i -m7 https://winged-horse.fyi
curl: (28) Connection timed out after 7001 milliseconds
2 Likes

I considered switching to apache, I briefly tested it but certbot shat itself and exploded "SSLCertificateFile: file '/etc/httpd/conf/server.crt' does not exist or is empty"

So I would rather stick with nginx.

I do not believe https not working is caused by my router, given that nginx is complaining about not being able to open the certs.

nginx did not complain about that when you ran sudo nginx -t

Which indicates nginx may not have been started correctly. If Apache is out of the picture we should check what nginx looks like. What do these show?

sudo ss -pant | grep -Ei 'nginx|:80|:443'

sudo systemctl status --no-pager -l nginx

sudo ps -eF | grep nginx | grep -v grep
1 Like

While sudoing "nginx -t" doesn't error trying to read the certs, starting nginx with sudo still doesn't result in ssl access.

Here's the output:

[kyle@Swiftwind nginx]$ sudo ss -pant | grep -Ei 'nginx|:80|:443'
LISTEN    0      511                                      0.0.0.0:443                       0.0.0.0:*     users:(("nginx",pid=10243,fd=5),("nginx",pid=10242,fd=5),("nginx",pid=10241,fd=5),("nginx",pid=10239,fd=5),("nginx",pid=10238,fd=5),("nginx",pid=10237,fd=5),("nginx",pid=10236,fd=5),("nginx",pid=10235,fd=5),("nginx",pid=10234,fd=5))
LISTEN    0      511                                      0.0.0.0:80                        0.0.0.0:*     users:(("nginx",pid=10243,fd=6),("nginx",pid=10242,fd=6),("nginx",pid=10241,fd=6),("nginx",pid=10239,fd=6),("nginx",pid=10238,fd=6),("nginx",pid=10237,fd=6),("nginx",pid=10236,fd=6),("nginx",pid=10235,fd=6),("nginx",pid=10234,fd=6))
ESTAB     0      0                                    192.168.1.2:80                    192.168.1.1:49898 users:(("nginx",pid=10235,fd=18))                                                                                                                                                                                                       
ESTAB     0      0                                    192.168.1.2:45228               172.67.188.91:443   users:(("transmission-da",pid=900,fd=91))                                                                                                                                                                                               
TIME-WAIT 0      0                                    192.168.1.2:80                   91.231.89.17:47421                                                                                                                                                                                                                                         
ESTAB     0      0      [2a0a:ef40:15fc:4601:c3c3:486f:4299:938e]:46558 [2606:4700:3034::6815:30ec]:443   users:(("transmission-da",pid=900,fd=97))   
● nginx.service - nginx web server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; preset: disabled)
     Active: active (running) since Mon 2025-11-17 09:26:37 GMT; 3min 6s ago
 Invocation: df7609b85fe34b01968236e2b324436a
    Process: 10233 ExecStart=/usr/bin/nginx (code=exited, status=0/SUCCESS)
   Main PID: 10234 (nginx)
      Tasks: 9 (limit: 16538)
     Memory: 9M (peak: 9.5M)
        CPU: 42ms
     CGroup: /system.slice/nginx.service
             ├─10234 "nginx: master process /usr/bin/nginx"
             ├─10235 "nginx: worker process"
             ├─10236 "nginx: worker process"
             ├─10237 "nginx: worker process"
             ├─10238 "nginx: worker process"
             ├─10239 "nginx: worker process"
             ├─10241 "nginx: worker process"
             ├─10242 "nginx: worker process"
             └─10243 "nginx: worker process"

Nov 17 09:26:37 Swiftwind systemd[1]: Starting nginx web server...
Nov 17 09:26:37 Swiftwind nginx[10233]: 2025/11/17 09:26:37 [warn] 10233#10233: could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size
Nov 17 09:26:37 Swiftwind nginx[10233]: 2025/11/17 09:26:37 [warn] 10233#10233: conflicting server name "winged-horse.fyi" on 0.0.0.0:80, ignored
Nov 17 09:26:37 Swiftwind systemd[1]: Started nginx web server.

root       10234       1  0  6399  3056   2 09:26 ?        00:00:00 nginx: master process /usr/bin/nginx
http       10235   10234  0  6494  5192   3 09:26 ?        00:00:00 nginx: worker process
http       10236   10234  0  6494  4740   7 09:26 ?        00:00:00 nginx: worker process
http       10237   10234  0  6494  5192   4 09:26 ?        00:00:00 nginx: worker process
http       10238   10234  0  6494  4740   0 09:26 ?        00:00:00 nginx: worker process
http       10239   10234  0  6494  4740   4 09:26 ?        00:00:00 nginx: worker process
http       10241   10234  0  6494  4740   7 09:26 ?        00:00:00 nginx: worker process
http       10242   10234  0  6494  4740   1 09:26 ?        00:00:00 nginx: worker process
http       10243   10234  0  6494  4676   3 09:26 ?        00:00:00 nginx: worker process

1 Like

I was expecting a Let's Encrypt certificate or maybe a self-signed certificate generated by certbot. This C=BE,ST=Antwerp,L=Edegem,O=Technicolor you see in your browser is very peculiar.

So I suspect this is a router firewall/NAT issue.

That would explain why we can't connect and why you're seeing this weird cert (the router is sending :443 traffic to a different device than you expect).

A simple test to verify this theory would be to shutdown nginx and see if you still get a certificate error or if it changes to a connection refused/timeout.

2 Likes

with nginx running:

inside my lan I get that "technicolor" cert. Outside my lan, no connection to 443

without nginx running: inside my lan I get that "technicolor" cert. Outside my lan, no connection to 443 (also no connection to port 80, but we expect that when no http server is there)

so I think you are right that my shitty consumer router has now decided on it's own to intercept port 443. I checked even the usual 192.168.1.1 config page with https and it provides the same bogus self signed cert.

Some time recently I think they forced a firmware update (which annoyed me massively as it killed my home internet for an entire hour), I think they have botched it. my port forwarding rules are all still correct.

however, that does not explain the nginx -t key file permission error. is that just a red herring caused by invoking it as my own user vs invoking it as root?

1 Like

Yep. The nginx map page says the following:

-t Do not run, just test the configuration file. nginx checks the configuration file syntax and then tries to open files referenced in the configuration
file.

It probably wants to open the referenced files to check their syntax as well.

Nginx also gave this warning when you ran -t without sudo, so it's taking the current user into account when loading the configuration even for testing:

the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
2 Likes

Well, I just went to the internal ip of the webserver and firefox found the correct lets encrypt ssl cert (and then complained because it wasn't coming through my domain name, hah!)

so I think it's 100% confirmed to be a router issue, thank you for helping me figure that out. also (mikemcq, webprofusion). As terrible freebie routers handed out by british ISPs is not an SSL cert provider's responsibility I'll take up no more of your time. Now I'll go fight my ISP.

Enjoy the rest of your day.

1 Like

Glad @dextercd was able to help identify the culprit.

And, good try on a local IP Firefox test. I was going to suggest a similar test. Just for your future reference you can try, with nginx running:

echo | openssl s_client -connect 127.0.0.1:443 --servername winged-horse.fyi

It should connect and show what cert nginx uses on port 443

For your Firefox test you could make an entry in your hosts file for your domain name with your local IP address. Then using the domain name in Firefox won't issue that mis-matched name error.

I think it likely an admin panel on your router is using port 443. Getting that modified is best of course. But, you could change nginx to use a different port. Then access it using something like: https://winged-horse.fyi:8443 Yeah, it's unpleasant but another way to test if nothing else

1 Like

It turned out that the router fell victim to the ISP remotely flashing it's firmware. Whatever they did caused it to ignore the port routing that were set in it's admin panel, steal anything trying to come in to 443 from the outside world, and also trash it's own certs.

A factory reset and then update to even newer firmware got it going again.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.