Certbot out of sync with nginx

Help
1

I have a few domains. Recently, one domain stopped being presented as secure by my web server.

The domain hosting service (cloudflare) doesn't show any issues.

On my host however, I find these issues:

On the web server (nginx), I see, from within the config file for the afflicted domain:

server_name [REDACTED].com;
ssl_certificate /etc/letsencrypt/live/[REDACTED].com-0001/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/[REDACTED].com-0001/privkey.pem; # managed by Certbot
access_log /var/log/nginx/[REDACTED].ssl_access_log main;
error_log /var/log/nginx/[REDACTED].ssl_error_log info;

...while, in thedirectory /etc/letseycrypt/live/, I find three directories:

[RECACTED].com
[RECACTED].com-0001
[RECACTED].com-0002

I've seen the "000N" pattern before, no argument there, it's used as a tactic to cope when something changes, and we're avoiding interaction with a human. Keen. But in this case, as you can see, the "0002" instance didn't make it all the way to the nginx section.

Another detail: of the domains I host, this one happens to be the one whose key was updated most recently (2025-05-10); that's when the "0002" file was created; that's why the web server doesn't see the new key; and so I want to not just fix this instance, but also to avoid future cases of it; why did this happen?

One guess: around the same time that this latest key update took place, I rearranged my web server's config file: in the past, the whole thing was on one page. Now, each domain has its own file, under "/etc/nginx/nginx.d/". If that's what made my domains unreachable by letsencrypt (again: if), how might I fix that?

Thanks!

2

I can't give specific advice about this problem without knowing specific details.

Trying to reproduce a problem that occured almost 2 months ago and do so with just the most generic info is impractical.

I will say this ...

The iterations -000X derive from issuing a new cert with differences from the prior in that lineage as you note. So, my first guess is the command you used that caused -0002 did not allow Certbot to update nginx (like using certonly option). Or, the reconfiguration you did to nginx aligned differently in some key way to prevent that.

If it is appropriate just manually correct nginx to refer to -0002 file names.

The scheduled certbot renew will renew the -0002 cert and hopefully reload nginx. This depends on the exact command you used (and which is now in the -0002 certbot renewal config)

You should be deleting certbot config's you no longer need. It is wasteful and can lead to various problems not the least of which are rate limits.

You can test the certbot renew with

sudo certbot renew --dry-run

You can see all your Certbot cert configs with

sudo certbot certificates

You delete unused ones only after you do not reference their files in any active server

sudo certbot delete --cert-name X

Where X is the name from the certificates list

3 Likes
3

Briefly:

I wish that I were not referring to something with elements that long ago. The symptom I'm describing arose only a few days ago, and it took me this long to guess (yes, still guessing), because a failed update took this long to cause a cert to expire.

I'll re-submit tomorrow, next time without wiping the form.

5

You don't need to resubmit. We'll need to see outputs of these to get started.

sudo certbot --version
sudo certbot certificates
ls -l /etc/nginx/nginx.d

And answers to these

My web server is (include version):

The operating system my web server runs on is (include version):
2 Likes
6

Picking back up with this, sorry for the delay. A quick reminder: "michaeljinks.com" has a file name, "michaeljinks.com-0002.conf", which has been missed in an automatic update. Under "/etc/letsencrypt" we have (among others):

$ sudo find /etc/letsencrypt/ -name *michaeljinks*
/etc/letsencrypt/live/michaeljinks.com-0002
/etc/letsencrypt/live/michaeljinks.com
/etc/letsencrypt/live/michaeljinks.com-0001
/etc/letsencrypt/archive/michaeljinks.com-0002
/etc/letsencrypt/archive/michaeljinks.com
/etc/letsencrypt/archive/michaeljinks.com-0001
/etc/letsencrypt/renewal/michaeljinks.com.conf
/etc/letsencrypt/renewal/michaeljinks.com-0002.conf
/etc/letsencrypt/renewal/michaeljinks.com-0001.conf

I should point out: as you'll notice below, a lot of subs have been associated with the wrong domain: under "michaeljins.com", most of the subs under "sswormwood.com" have, somehow, been attached to that key. That's a known issue, I'm still trying to fix it, but (I think) independent of the "ignoring updates" problem.

| MikeMcQ Leader
July 5 |

  • | - |

You don't need to resubmit. We'll need to see outputs of these to get started.

sudo certbot --version

$ sudo certbot --version
certbot 4.0.0

sudo certbot certificates

$ > sudo certbot certificates

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/michaeljinks.com.conf produced an unexpected error: renewal config file {} is missing a required file reference. Skipping.

Found the following certs:

Certificate Name: ellenmorrison.us
Serial Number: 693f66c846cc88954232c7647aa8ebfef75
Key Type: ECDSA
Domains: ellenmorrison.us
Expiry Date: 2025-10-08 04:31:48+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/ellenmorrison.us/fullchain.pem
Private Key Path: /etc/letsencrypt/live/ellenmorrison.us/privkey.pem

Certificate Name: michaeljinks.com-0001
Serial Number: 68301a31c55a1f0eedd1bb5f1a3ac18b946
Key Type: ECDSA
Domains: michaeljinks.com adventures.sswormwood.com brains.sswormwood.com dm.sswormwood.com eva-1-dm.sswormwood.com eva-1.sswormwood.com fix.michaeljinks.com flp.sswormwood.com house.sswormwood.com iai.sswormwood.com multinational.net rye.sswormwood.com sands.sswormwood.com sswormwood.com stevestevesteve.sswormwood.com sysops.sswormwood.com twork.sswormwood.com wandw.sswormwood.com
Expiry Date: 2025-06-26 04:35:28+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/michaeljinks.com-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/michaeljinks.com-0001/privkey.pem

Certificate Name: michaeljinks.com-0002
Serial Number: 2c771fc340b0c78772588649adefa016ad4c
Key Type: ECDSA
Domains: michaeljinks.com
Expiry Date: 2025-10-08 04:32:14+00:00 (INVALID: TEST_CERT)
Certificate Path: /etc/letsencrypt/live/michaeljinks.com-0002/fullchain.pem
Private Key Path: /etc/letsencrypt/live/michaeljinks.com-0002/privkey.pem

Certificate Name: multinational.net-0001
Serial Number: 5c5f8d477f79dfbe7920a9546bc60fa2b3e
Key Type: ECDSA
Domains: multinational.net
Expiry Date: 2025-10-08 04:32:23+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/multinational.net-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/multinational.net-0001/privkey.pem

Certificate Name: sswormwood.com-0001
Serial Number: 6f1de74d32f62630483905ff88976848d93
Key Type: ECDSA
Domains: sswormwood.com
Expiry Date: 2025-10-08 04:32:31+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/sswormwood.com-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/sswormwood.com-0001/privkey.pem

The following renewal configurations were invalid:
/etc/letsencrypt/renewal/michaeljinks.com.conf


ls -l /etc/nginx/nginx.d

$ l /etc/nginx/nginx.d
total 72
-rw-r--r-- 1 root root 528 May 10 22:52 michaeljinks.com.conf
-rw-r--r-- 1 root root 676 May 10 23:22 eva-1-dm.sswormwood.com.conf
-rw-r--r-- 1 root root 610 May 10 23:23 eva-1.sswormwood.com.conf
-rw-r--r-- 1 root root 1014 May 10 23:36 ellenmorrison.us.conf
-rw-r--r-- 1 root root 863 May 11 00:45 wandw.sswormwood.com.conf
-rw-r--r-- 1 root root 869 May 11 00:45 flp.sswormwood.com.conf
-rw-r--r-- 1 root root 871 May 11 00:53 sands.sswormwood.com.conf
-rw-r--r-- 1 root root 958 May 11 00:55 adventures.sswormwood.com.conf
-rw-r--r-- 1 root root 884 May 11 01:04 dm.sswormwood.com.conf
-rw-r--r-- 1 root root 682 May 11 01:09 sswormwood.com.conf
-rw-r--r-- 1 root root 1230 May 11 01:26 multinational.net.conf
-rw-r--r-- 1 root root 690 May 11 01:27 fix.michaeljinks.com.conf
-rw-r--r-- 1 root root 974 May 11 01:27 house.sswormwood.com.conf
-rw-r--r-- 1 root root 659 May 11 01:27 iai.sswormwood.com.conf
-rw-r--r-- 1 root root 927 May 11 01:28 rye.sswormwood.com.conf
-rw-r--r-- 1 root root 993 May 11 01:28 sysops.sswormwood.com.conf
-rw-r--r-- 1 root root 973 May 11 01:28 twork.sswormwood.com.conf
-rw-r--r-- 1 root root 952 May 11 01:29 wg.sswormwood.com.conf

And answers to these

My web server is (include version):

$ equery l nginx

  • Searching for nginx ...
    [IP-] www-servers/nginx-1.26.3-r2:0

The operating system my web server runs on is (include version):

Gentoo linux, up to date as of 29 June

Much thanks.

7

Well, no, it hasn't been missed (note the expiry date of Oct) but it is a cert for the Let's Encrypt staging system which is not valid for production use. But, it doesn't matter as your nginx isn't using that cert anyway.

Requests to michaeljinks.com uses the -0001 cert with the large number of domain names. See: SSL Checker

To renew that cert requires all of those domain names to satisfy a new challenge. One or more of them is likely failing which prevents renewing that cert.

I spot-checked a couple domains and your server replies very differently for them. michaeljinks.com might be okay but somewhat odd results for adventures.sswormwood.com and house.sswormwood.com in that the first gives a 502 Gateway error (to HTTPS requests) and the other a 401 Auth error (for HTTPS requests).

Given what I see I am not surprised Certbot and nginx are out of sync :slight_smile:

Going forward I think it is best to have a single cert for each nginx config file. Or, at least one cert for the registered name and its related subdomains. Mixing lots of domain names in the same cert can be difficult to manage.

Let's start by you showing contents of this file. We'll get this one working and go from there

/etc/nginx/nginx.d/michaeljinks.com.conf
2 Likes
8

A little background: I made these domains months or years apart; I kept poor notes (I fire me), each one felt like a "first"... and I think that's why "michaeljinks.com" happens to have engulfed some others. Obviously I want each domain name to act as its own base.

Thanks for the clue re. SSL Checker. This is how we learn.

re. "testing" certs, with a bad date: I've just been assuming that a file date of 10 June explained the problem. How that came out as a "testing" cert, I have no idea.

Anyhow. "/etc/nginx/nginx.d/michaeljinks.com.conf":

    server {
            listen 443 ssl;
            listen [::]:443 ssl;
            server_name michaeljinks.com;
            ssl_certificate /etc/letsencrypt/live/michaeljinks.com-0001/fullchain.pem; # managed by Certbot
            ssl_certificate_key /etc/letsencrypt/live/michaeljinks.com-0001/privkey.pem; # managed by Certbot
            access_log /var/log/nginx/michaeljinks.ssl_access_log main;
            error_log /var/log/nginx/michaeljinks.ssl_error_log info;
            root /var/www/localhost/htdocs/;
            location ~ ^/~(.+?)(/.*)?$ {
                    alias /home/$1/public_html$2;
                    autoindex on;
            }
    }        server {
            listen 443 ssl;
            listen [::]:443 ssl;
            server_name michaeljinks.com;
            ssl_certificate /etc/letsencrypt/live/michaeljinks.com-0001/fullchain.pem; # managed by Certbot
            ssl_certificate_key /etc/letsencrypt/live/michaeljinks.com-0001/privkey.pem; # managed by Certbot
            access_log /var/log/nginx/michaeljinks.ssl_access_log main;
            error_log /var/log/nginx/michaeljinks.ssl_error_log info;
            root /var/www/localhost/htdocs/;
            location ~ ^/~(.+?)(/.*)?$ {
                    alias /home/$1/public_html$2;
                    autoindex on;
            }
    }

Much thanks.

9

First, we need to fix the nginx conf for michaeljinks. It should have a server block for port 80. But, worse is that it has a duplicated server block. I am surprised nginx did not issue a warning message when it starts. It normally complains when you have two server blocks for same server_name and same listening ports.

The port 80 server block can look like below. I included a location to allow using --webroot which we won't use initially. But, it is handy to have ready in case we change.

In the end, that conf should look like:

server {
    listen 80;
    listen [::]:80;
    server_name michaeljinks.com;
    access_log /var/log/nginx/michaeljinks.ssl_access_log main;
    error_log /var/log/nginx/michaeljinks.ssl_error_log info;
    root /var/www/localhost/htdocs/;
    location /.well-known/acme-challenge/ {
        root /var/www/localhost/htdocs;       # Use different folder if you prefer
    }
    location / {
       return 301 https://$host$request_uri;
    }
}
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name michaeljinks.com;
    ssl_certificate /etc/letsencrypt/live/michaeljinks.com-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/michaeljinks.com-0001/privkey.pem; # managed by Certbot
    access_log /var/log/nginx/michaeljinks.ssl_access_log main;
    error_log /var/log/nginx/michaeljinks.ssl_error_log info;
    root /var/www/localhost/htdocs/;
    location ~ ^/~(.+?)(/.*)?$ {
        alias /home/$1/public_html$2;
        autoindex on;
    }
}
2 Likes
10

After updating your nginx conf we need to clean up your Certbot directories. Then we'll get a new cert just for this domain name

Delete the following. They are not used by your nginx so won't prevent it from running. This Certbot renewal conf is broken so we need to delete it before we re-use this same conf name.

sudo rm /etc/letsencrypt/renewal/michaeljinks.com.conf
sudo rm -R /etc/letsencrypt/live/michaeljinks.com
sudo rm -R /etc/letsencrypt/archive/michaeljinks.com

Now, do below Certbot command. It is only a test. We will do something different for a production cert once this is proven

sudo certbot certonly --nginx --dry-run --cert-name michaeljinks.com -d michaeljinks.com

Let me know result of that before we proceed. I don't want Certbot to be modifying your nginx config until we know everything is in order.

I also want to send some test requests to your nginx to ensure it replies as expected after these changes.

1 Like
11

clerical error here: i do not know how i posted two copies of the same 'listen 443 ssl;' section of michaeljinks.com. there is only one such section on the system.

sigh.

anyhow, MikeMcQ posted a section:

server {
    listen 80;
    listen [::]:80;
    server_name michaeljinks.com;
    access_log /var/log/nginx/michaeljinks.ssl_access_log main;
    error_log /var/log/nginx/michaeljinks.ssl_error_log info;
    root /var/www/localhost/htdocs/;
    location /.well-known/acme-challenge/ {
        root /var/www/localhost/htdocs;       # Use different folder if you prefer
    }
    location / {
       return 301 https://$host$request_uri;
    }
}

(...as well as the ssl part)

on my host i have, in the main nginx.conf file:

      server {
              listen 80 default_server
              server_name _;
              return 301 https://$host$request_uri;
              access_log /var/log/nginx/redirect_ssl.access_log main;
              error_log /var/log/nginx/redirect_ssl.error_log info;
        }

...so is it correct to say that i don't need a "listen 80" for each domain? or, would that be incorrect, and i really do need to give each domain its own "listen 80"? (and might my "fix all 80" approach part of the proboem...?)

12

i did remove...

$ sudo rm -R renewal/michaeljinks.com* live/micaeljinks.com archive/michaeljinks.com
rm: cannot remove 'live/micaeljinks.com': No such file or directory

(and of course the error is just because there was a symlink to a file we removed in the same command)

then:

$ sudo certbot certonly --nginx --dry-run --cert-name michaeljinks.com -d michaeljinks.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for michaeljinks.com
The dry run was successful.
13

First, your other server blocks have a listen for IPv6 so your default should too. And, that listen should have a trailing semi-colon

As for the common redirect block, that can work. It is not optimal for --webroot but if you use the --nginx plugin for Certbot it should be fine. Certbot will actually make a temp server block since you don't have a specific one for that domain but the end result is the same.

You should be able to follow the steps I outlined ... oh, I see you did (sort of)

1 Like
14

There was also a typo in your command

should be ...michael... - right?

1 Like
15

I wasn't clear but please complete the delete steps properly before we proceed. The --dry-run can ignore things that an actual run will not

1 Like
16

First, I am on a typo bonanza here... both:

listen 80 default_server;

...and:

.../live/michaeljinks.com

...both were correct on the host. I don't know why copy-and-paste doesn't like me lately. Carrying on:

To make sure we're clear: the goal in the main nginx.conf (repeat):

        server {
                listen 80 default_server;
                server_name _;
                return 301 https://$host$request_uri;
[...]

...really is to catch anything pointed at port 80, and change it to port 443, that's the point of that section: avoid sections receiving any non-secure traffic. Am I truly leaving out something that has to be there? This setup hasn't been causing errors (as far as I've seen) and it's been like that for some time. For instance, if I deliberately point at:

  http://twork.sswormwood.com

...that gets redirected to the https version, no issues, and no section-specific entries in the main.

If that section of the nginx.conf base file really does need to change, then I'll need further hinting.

Moving on: with the "michaljinks.com" file set removed:

$ sudo certbot certonly --nginx --dry-run --cert-name michaeljinks.com -d michaeljinks.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for michaeljinks.com
The dry run was successful.

Keen? Next?

Much thanks.

17

Yes, I understand. But, the HTTP Challenge uses port 80 so is optimal when the challenge itself is not redirected (but all other traffic should be). Let's Encrypt will follow redirects it just adds unnecessary traffic.

The reason it works today is because of this extra traffic (if using webroot) or using --nginx option which makes a new, temporary, server block for port 80 and the HTTP Challenge.

I plan to have you use --nginx but wanted the option to use an optimal webroot conf which is why I suggested adding a new server block for port 80 for this domain name. I never suggested modifying the default server block. Except for making sure it also listened to IPv6 like all your others.

I don't know about that. The error message from your o/s echo'd back the misspelled name. Let's confirm the directories are as they should be

Please show

ls -l /etc/letsencrypt/renewal/michaeljinks.com.conf
ls -l /etc/letsencrypt/live/michaeljinks.com
ls -l /etc/letsencrypt/archive/michaeljinks.com

Note: We shouldn't see anything from those ls commands if they were deleted properly

1 Like
18

Going out of order here; short part first:

$ ls -l /etc/letsencrypt/renewal/michaeljinks.com.conf
ls: cannot access '/etc/letsencrypt/renewal/michaeljinks.com.conf': No such file or directory

$ sudo ls -l /etc/letsencrypt/live/michaeljinks.com
total 4
-rw-r--r-- 1 root root 692 Jan 16  2024 README
lrwxrwxrwx 1 root root  40 Jan 16  2024 cert.pem -> ../../archive/michaeljinks.com/cert1.pem
lrwxrwxrwx 1 root root  41 Jan 16  2024 chain.pem -> ../../archive/michaeljinks.com/chain1.pem
lrwxrwxrwx 1 root root  45 Jan 16  2024 fullchain.pem -> ../../archive/michaeljinks.com/fullchain1.pem
lrwxrwxrwx 1 root root  43 Jan 16  2024 privkey.pem -> ../../archive/michaeljinks.com/privkey1.pem

$ sudo ls -l /etc/letsencrypt/archive/michaeljinks.com
ls: cannot access '/etc/letsencrypt/archive/michaeljinks.com': No such file or directory

...but that "No such" is expected, yeah, because of the prep step earlier?

That's where I'm fouling up. If I just do a flat copy of that section of the main config file, but with IPv6 in place of IPv4 -- this is wrong, right? I've tried adding to the base, here's one example of my wrongness:


        server {
                listen 80 default_server;
                server_name _;
                return 301 https://$host$request_uri;

                access_log /var/log/nginx/redirect_ssl.access_log main;
                error_log /var/log/nginx/redirect_ssl.error_log info;
        }

        server {
                listen 443 default_server;
                listen [::]:443 ssl;
                server_name _;
                return 301 https://$host$request_uri;

                access_log /var/log/nginx/redirect_ssl.access_log main;
                error_log /var/log/nginx/redirect_ssl.error_log info;
        }

Because I'm clueless I've tried that with and without the "listen [::]:443 ssl;" line, no difference, the nginx process calls it an error.

Related, also clueless: are you saying that the corresponding addition should happen in each server line, add "listen 80" to all the domains -- just so I'm sure I understand, this is to save some work on each of those domains? So, for example, we'd have, in this domain's config?

        server {
                listen 80;
                server_name michaeljinks.com;
                return 301 https://$host$request_uri;

                access_log /var/log/nginx/michaeljinks.access_log main;
                error_log /var/log/nginx/michaeljinks.error_log info;
        }

        server {
                listen 443 ssl;
                listen [::]:443 ssl;
                server_name michaeljinks.com;
                ssl_certificate /etc/letsencrypt/live/michaeljinks.com-0001/fullchain.pem; # managed by Certbot
                ssl_certificate_key /etc/letsencrypt/live/michaeljinks.com-0001/privkey.pem; # managed by Certbot
                access_log /var/log/nginx/michaeljinks.ssl_access_log main;
                error_log /var/log/nginx/michaeljinks.ssl_error_log info;
                root /var/www/localhost/htdocs/;
                location ~ ^/~(.+?)(/.*)?$ {
                        alias /home/$1/public_html$2;
                        autoindex on;
                }
        }

Aside: notice the reference to "michaeljinks.com-0001" here; I may be getting ahead of things but the "0001/0002" miscup is at least part of the backing problem, and that's (no really) a straight copy of what's there now.

Whee.

19

Correct. Please go back and issue the rm commands as I posted them

Correct in that you are totally wrong.

You should not have added default port 443 server block. Remove that. What I meant was your port 80 needs both IPv4 and IPv6 listens to be consistent with all your others.

So, add this one listen line to add IPv6 for default server on port 80

server {
                listen 80 default_server;
                listen [::]:80 default_server;
                server_name _;
                return 301 https://$host$request_uri;

                access_log /var/log/nginx/redirect_ssl.access_log main;
                error_log /var/log/nginx/redirect_ssl.error_log info;
        }

Yeah, we are getting pretty far away from helping with certificates and on to just educating about nginx admin. I'm not sure how much further I want to assist in that education effort.

If you carefully follow my instructions I will get you to a working example for michaeljinks. Perhaps you will be able to use that as a model to get your other ones working.

You, admittedly, don't have a good understanding of how nginx works. You should learn more about that or even consider a different solution. Perhaps a hosting service that provides a way for you to focus on the content of the site and not have to bother with these technical details.

I will carry on helping you get michaeljinks working if you want to follow my instructions. Otherwise I have other things I can be doing :slight_smile: Mind you, I'm an unpaid volunteer offering my time and experience to you for free.

1 Like
20

Oh, duh. I follow now. Fixed.

I know that you (and others) are just offering me favors. FWIW, I don't lack experience in this, rather I'm out of practice. Faint difference I realize, but that's how it got to where it is; this used to be my job, now it's a remnant. I'm deeply grateful. FWIW.

So! Next dumb question: you specified removal of "michaeljinks.com" in those locations. (Yup: did that.) Are we deliberately leaving .0001 and .0002 files as they are (like, because the process expects to find that pattern, recognize that the "clean" file is now gone, so write there on the next go-round), or, should I entirely clean out "michaeljinks.com*"?

I'm assuming the latter, so removed those as well, but set them aside just in case.

21

The "because" is just that we can only do one thing at a time. We get one thing working, then another, then we go back and clean up. We can't clean up too early or we break your nginx altogether.

Specifically, nginx uses the -0001 so if you deleted that you now broke nginx

Frankly, I am exhausted trying to follow your twists and turns. I am trying to walk you through step by step but you keep jumping off the path.

We really haven't made any progress and we are 20 posts in. I don't think I am the best person to help you. My style isn't for everyone. Maybe a different volunteer will work out better.

1 Like