Can't renew certificates for various domains across servers

Help
1

Hi, all. I'm a technician working on a number of sites hosted by a small outfit. I've been struggling with certbot for several hours. We have several domains across several servers, all using certbot. They have all been fine for the last 4 or 5 years. All servers are now running ubuntu 20.04 and all sites run on nginx. Some are php sites and some are ruby on rails.

I had a certificate expiry notice on my email client. This usually doesn't happen because there are cron jobs that update all certificates. The domain Burtman.net points at a website on [let's call it] server 1. Certificates on that server are fine. The domain mail.burtman.net is the mail domain running on server 2, and has expired, affecting mail for all domains.

The error message shows that the certificate is invalid because it is only for burtman.net, not mail.burtman.net. An error to the same effect shows for the web domains.

I can't renew or expand the certificates; certbot won't let me add mail. to the burtman.net cert because it isn't on the same server. Mail.burtman.net won't renew on the correct server either because there is no website associated with it (but it was fine for the last few years without a website associated).

Further, I have renewed the certificate for fusiondrivingtuition.co.uk and it shows 89 days validity on the server when I run certbot certificates. But in the browser, it no longer loads the website. It just downloads a small file (with a random name on the computer and downloadfile.bin on the phone).

I have tried renewing, expanding and removing and reinstalling the certificates. Nothing is working. My computer has been restarted and the cache and cookies have been cleared. I have restarted nginx and even the server (it was during a scheduled patching window but the patching was canceled because of this).

The domains are:
Burtman.net (one server 1)
Mail.burtman.net (on server 2)
Fusiondrivingtuition.co.uk (on server 2)

I don't understand the following:

  1. Why this behavior is happening now, as nothing has been changed on either server in a long time.

  2. Why the certificates are valid on the servers and show the correct domains but in firefox and mail apps, they show as being for burtman.net (and thus not valid for the actual domains).

I ran this command:
certbot certificates

It shows all the certificates. All valid and correctly assigned to their respective domains. No warnings, errors, exceptions or notices.

Web server: Nginx 1.18.0

Server OS: Ubuntu 20.04 - Kernel 5.4.0-173

Hosting provider: Digital ocean

I have root on all servers via ssh.

Certbot version: 0.40.0

I have inspected the .conf files in /etc/nginx/sites-available and compared them to those on other servers. I can't see anything wrong there.

Any help would be very much appreciated (it's 4am and I need to call it a day).

2

As for fusion is this the response I should see? In Firefox on Ubuntu I just get a prompt to download a 49 byte file with a random name (like TrfPDQrB)

I get a valid HTTPS connect (both curl and firefox) so that's different than you. Your server just doesn't look like it is processing the php file and instead sending it to me as a file. That's a server misconfiguration for php.

curl -i https://fusiondrivingtuition.co.uk
HTTP/2 200
server: nginx
date: Fri, 08 Mar 2024 03:51:29 GMT
content-type: application/octet-stream
content-length: 49
last-modified: Mon, 13 Dec 2021 10:45:08 GMT
etag: "61b72434-31"
x-frame-options: sameorigin
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-download-options: noopen
x-permitted-cross-domain-policies: none
content-security-policy: default-src https: data: 'unsafe-inline' 'unsafe-eval'
referrer-policy: strict-origin
accept-ranges: bytes

<?php
        header("Location: intro.php");
        exit();
?>
2 Likes
3

As for mail I get the wrong cert for ports 143,587,993,995 but the correct cert for port 443. It looks like some port mapping on your system may be sending those requests to your wrong server. Have you modified any network routing lately? On a residential system I'd suggest looking at your router NAT or port forwarding.

Have you asked D/O about this? Just from this it looks like some comms config has gone wrong at your end.

2 Likes
4

Hi, Mike. Yes, that's the random file download I get on my computer. I don't know where it's coming from. It should be redirecting to the intro.php script and serving a normal html output. This was not happening until I tried the certificate renewal last night.

There have been no changes of configuration on either server. The DNS entries on DO don't appear to have changed.

I'm not sure about the certificates being invalid for 143 and so on. That never happened before. Is there a way to cover more ports when renewing a certificate?

I'll check configs against a server backup and ask DO if anything changed.

1 Like
5

I found that nginx was configured with default_type set to octet stream. No idea why this changed. I restored it to text/html and reloaded nginx with nginx -s reload. That threw the error:

nginx: [emerg] no port in upstream "php_workers" in /etc/nginx/templates/fastchi_php.tmpl

Which I have resolved by adding the port to the directive:

fastcgi_pass php_workers:443;

Now, one of the sites is being served and has a valid certificate, but the site is without CSS. Fusion still doesn't load, but instead of serving a small file, I get a blank page, which is a start, but something is still wrong.

Will update as I progress but still very grateful for tips and suggestions.

1 Like
6

A cert can be used on any port. You just configure the service handling that port to use the correct cert. Your system is not.

Using port 443 (normal HTTPS) with your mail subdomain gets a valid cert. That requests for mail and any mail port gets a wrong cert is just a config error on your system. One example is routing incoming requests on those ports to the wrong server. Or, whatever is handling those ports for mail for some reason uses the wrong cert. Probably not this latter one because the wrong cert is the one on your server1 so feels more like network / port routing issue.

Hopefully D/O can help you resolve that.

3 Likes
7

Ok, it seems to be working now, at least foe the websites. I can only assume that certbot messed up my configs during renewal. I found the problem and it had nothing to do with what I was working on. For some reason, the fastcgi_pass directive had completely wrong path to php-frm socket file. After fixing that, all php sites came back and appear to be secure.

I will have to investigate further to understand what caused this, but first things first - snapshot.

Thank you for your time and consideration, Mike. I appreciate you working on it with me. :slight_smile:

3 Likes
8

Certbot does not make changes to any server config when doing renew command.

It would only make changes when using the original request. And, only when using the --nginx (or --apache) plugins without the certonly command. If you used --webroot or --standalone or custom --manual options then no updates are made to the server config.

Hopefully that helps you progress the debug.

4 Likes
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.