Mail server cert didn't renew

Help
1

Our mail server's cert didn't renew. It did in the past though, so I'm trying to track down why. What's the best way of zeroing in on a cause? I think (but can't remember) that it renews using a different method to the web servers which are also running on the machine? The mail server itself is on a different IP though.

My domain is: smtp.hatters.org.uk

I ran this command:

certbot renew --dry-run --cert-name smtp.hatters.org.uk

It produced this output:

Domain: smtp.hatters.org.uk
  Type:   connection
  Detail: 185.73.44.59: Fetching http://smtp.hatters.org.uk/.well-known/acme-challenge/B6xauPcA2TlMmOcbXIQQFyFBMn2W7aDb-DV66mGWN38: Connection refused

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

My web server is (include version): Apache 2.4.52

The operating system my web server runs on is (include version): Linux Ubuntu 22.04

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot --version doesn't show anything?

2

If the mail server is running on a different IP, you have two options:

1- Get a certificate on the WEB machine, using either a HTTP-01/TLS-ALPN-01/DNS-01 challenge, then copy it over.

2- Get a certificate on the MAIL machine, using the DNS-01 challenge.

3 Likes
3

Thanks!

I'm not sure how it got renewed last time, but it feels like the first of those methods may be easier.

There already is a cert in /etc/letsencrypt/live/smtp.hatters.org.uk which our mail server is reading. So do I just set up a web server called smtp.hatters.org.uk and do a renew wirth that (using the Apache plugin as we do today with the web servers)?

1 Like
4

From that message it says you are using an HTTP Challenge to get the cert. And, that the HTTP request from the LE server to your server is being refused.

That HTTP request goes to the IP in the public DNS. I see your long history of getting this cert.

Are you running Certbot on the same server as that IP?

If so, check that port 80 is open. If not, what has changed since your last good cert in June?

3 Likes
5

The server has two IPs, one for mail (185.73.44.59) and one for everything else (185.73.44.83) including web. Port 80 is open on the mail server's IP.

I'm racking my brains to think what's changed that could have affected it. I've just tried taking down the firewall (in and outbound) and the (outbound) proxy but it still doesn't want to renew on the dry-run.

Would the above web server plan work? If I delete /etc/letsencrypt/live/smtp.hatters.org.uk, set up a web server with:

<VirtualHost [::]:80 *:80>

  ServerName smtp.hatters.org.uk

</VirtualHost>

Then run certbot --apache -d smtp.hatters.org.uk - would that re-create /etc/letsencrypt/live/smtp.hatters.org.uk for the mail server to read OK?

Just wish I could work out what's changed since last time that could have affected it.

6

On the machine you run Certbot, can you show contents of this

/etc/letsencrypt/renewal/smtp.hatters.org.uk.conf

And output of this

sudo certbot certificates --cert-name smtp.hatters.org.uk
3 Likes
7 
# renew_before_expiry = 30 days
version = 2.11.0
archive_dir = /etc/letsencrypt/archive/smtp.hatters.org.uk
cert = /etc/letsencrypt/live/smtp.hatters.org.uk/cert.pem
privkey = /etc/letsencrypt/live/smtp.hatters.org.uk/privkey.pem
chain = /etc/letsencrypt/live/smtp.hatters.org.uk/chain.pem
fullchain = /etc/letsencrypt/live/smtp.hatters.org.uk/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
account = 03e6800966382802ae97a6a490ab3a9a
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa
webroot_path = /var/www/html,
[[webroot_map]]
mail.hatters.org.uk = /var/www/html

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following matching certs:
  Certificate Name: smtp.hatters.org.uk
    Serial Number: 34cecb2af7bc80b6d8b43037af4f8870f32
    Key Type: ECDSA
    Domains: mail.hatters.org.uk imap.hatters.org.uk mx0.hatters.org.uk pop.hatters.org.uk pop3.hatters.org.uk postfix.hatters.org.uk smtp.hatters.org.uk
    Expiry Date: 2024-09-16 08:02:27+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/smtp.hatters.org.uk/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/smtp.hatters.org.uk/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
8

Hmm. To have gotten a cert with --webroot means those domains must reply to an HTTP challenge on port 80. And, at one time they must have.

But, right now none of the domains in that cert (which are all on IP .59) reply to any HTTP request on port 80. They all get the "Connection Refused".

If I probe ports using nmap both 80 and 443 are "closed". I can reach your other normal mail ports just not these two. Only port 80 is needed for the HTTP Challenge.

What does this show

sudo ss -pant | grep -i listen | grep -Ei ':80|:443|:465'
3 Likes
9 
LISTEN     0      100                  0.0.0.0:465                                     0.0.0.0:*     users:(("master",pid=2206267,fd=31),("smtpd",pid=1027949,fd=6),("smtpd",pid=1023882,fd=6))                                                           

LISTEN     0      511             185.73.44.83:443                                     0.0.0.0:*     users:(("apache2",pid=3114359,fd=5),("apache2",pid=1031255,fd=5),("apache2",pid=1031247,fd=5),("apache2",pid=1031214,fd=5),("apache2",pid=1031207,fd=5),("apache2",pid=1031206,fd=5),("apache2",pid=1031205,fd=5),("apache2",pid=1031204,fd=5),("apache2",pid=1031203,fd=5))

LISTEN     0      511             185.73.44.83:80                                      0.0.0.0:*     users:(("apache2",pid=3114359,fd=3),("apache2",pid=1031255,fd=3),("apache2",pid=1031247,fd=3),("apache2",pid=1031214,fd=3),("apache2",pid=1031207,fd=3),("apache2",pid=1031206,fd=3),("apache2",pid=1031205,fd=3),("apache2",pid=1031204,fd=3),("apache2",pid=1031203,fd=3))

LISTEN     0      5                  127.0.0.1:8053                                    0.0.0.0:*     users:(("named",pid=3114134,fd=34))                                                                                                                  

LISTEN     0      5                  127.0.0.1:8053                                    0.0.0.0:*     users:(("named",pid=3114134,fd=33))                                                                                                                  

LISTEN     0      5                  127.0.0.1:8053                                    0.0.0.0:*     users:(("named",pid=3114134,fd=35))                                                                                                                  

LISTEN     0      511    [2001:ba8:0:4017::53]:80                                         [::]:*     users:(("apache2",pid=3114359,fd=4),("apache2",pid=1031255,fd=4),("apache2",pid=1031247,fd=4),("apache2",pid=1031214,fd=4),("apache2",pid=1031207,fd=4),("apache2",pid=1031206,fd=4),("apache2",pid=1031205,fd=4),("apache2",pid=1031204,fd=4),("apache2",pid=1031203,fd=4))

LISTEN     0      511    [2001:ba8:0:4017::53]:443                                        [::]:*     users:(("apache2",pid=3114359,fd=6),("apache2",pid=1031255,fd=6),("apache2",pid=1031247,fd=6),("apache2",pid=1031214,fd=6),("apache2",pid=1031207,fd=6),("apache2",pid=1031206,fd=6),("apache2",pid=1031205,fd=6),("apache2",pid=1031204,fd=6),("apache2",pid=1031203,fd=6))

LISTEN     0      100                     [::]:465                                        [::]:*     users:(("master",pid=2206267,fd=32),("smtpd",pid=1027949,fd=7),("smtpd",pid=1023882,fd=7))

Maybe I've misunderstood - I thought certbot set up a temporary web server on port 80 to request the renewal, which would explain why there is nothing listening on port 80 on .59 since that's not use for web, only mail. We keep port 80 open for certbot though, no?

1 Like
10

Certbot can setup a standalone temp server but the renewal profile is using --webroot which needs a webserver of some kind to reply on 80 at the IP in the DNS.

For webroot Certbot just writes a challenge token file into that folder which that web server must send back to LE.

Something structurally has changed. So, you either need to restore what was or figure out a better way forward.

3 Likes
11

Going into this a bit more:

1- webroot places a file on your directory structure, which LetsEncrypt then reads.
2- At a minimum, you had the mail domain configured to redirect traffic to the web domain
3- more likely, you ran a temporary server on that domain

You have 2 general options to automate the renewal with http-80 in your situation.

  1. webroot. Configure a web service to run on the domain. it can serve an empty file. It just needs to be configured to serve the webrootfiles.

  2. standalone. Certbot will become the webserver and bind to port 80 of that IP during the challenge phase. this would fail if another webservice is running, but if you're not running anything on that ip, this should be fine.

3 Likes
12

Ah - so it's not setting up its own server? That was a great clue! We stopped Apache from binding to the IP the mail server was on as part of trying to prevent unfiltered mail going out from web sites.

So - re-binding Apache to the mail server's IP has fixed the certbot renewal. But now I think in theory our web server can bypass our mail scanning, so I'll need to work that out...

Thanks so much @MikeMcQ @jvanasco for your speedy help though - much appreciated! :grinning:

4 Likes
13

Yes, as to way forward you could review your Apache. Couldn't it just listen on that mail IP too and handle the incoming HTTP challenge requests for your mail domains? That is my best guess as to what had been working.

Alternatively, you could switch to using the --standalone option and set this option to your mail IP

--http-01-address HTTP01_ADDRESS
The address the server listens to during http-01 challenge. (default: )

You'd have to list all the domain names in your existing cert and probably use --cert-name to ensure you update the cert for your mail domains. I didn't test this but something like this might work:

sudo certbot certonly --standalone --dry-run --http-01-address (mail-IP) --cert-name smtp.hatters.org.uk -d (name1) -d (name2) ...

EDIT:
We cross-posted but maybe above detail on standalone is worth playing with?

7 Likes
14

Nope! "webroot" leverages an existing server for the challenge, "standalone" sets up it's own server. "nginx" and "apache" reconfigure the server.

I am a big fan of "standalone" for situations like this.

4 Likes
15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.