Certbot suddenly fails to renew any certificates

Help
1

My domain is:
naprvyraz.sk

I ran this command:
certbot renew -v

It produced this output:
Processing /etc/letsencrypt/renewal/naprvyraz.sk.conf

Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for naprvyraz.sk and 3 more domains
Performing the following challenges:
http-01 challenge for naprvyraz.sk
http-01 challenge for www.naprvyraz.sk
Waiting for verification...
Challenge failed for domain naprvyraz.sk
Challenge failed for domain www.naprvyraz.sk
http-01 challenge for naprvyraz.sk
http-01 challenge for www.naprvyraz.sk

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: naprvyraz.sk
Type: unauthorized
Detail: 176.58.103.186: Invalid response from http://naprvyraz.sk/.well-known/acme-challenge/6Y04sfIpxno-S1EJIaHQrZFAkWgt3SH7ZsTnR7K8dwc: 404

Domain: www.naprvyraz.sk
Type: unauthorized
Detail: 176.58.103.186: Invalid response from http://www.naprvyraz.sk/.well-known/acme-challenge/ZPZgkWk98Lu-ELXGsIyVxDy6BfFGDTgnKVxprSGnF9c: 404

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate naprvyraz.sk with error: Some challenges have failed.

My web server is (include version):
apache2 2.4.62-1~deb11u2

The operating system my web server runs on is (include version):
Linux 5.10.0-16, Debian 5.10.127-1

My hosting provider, if applicable, is:
n/a

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
4.0.0

I'm not exactly a server guru, I'm just running a simple apache2 server, configured ages ago. I get the email about certificate expiration, I log in, run certbot renew and be done with that. However suddenly I'm getting those errors listed above, I haven't changed anything in years. Maybe cerbot has updated and requires something done differently?

2

This is an anti-pattern. certbot renew should be run daily or so on an automatic basis; you shouldn't need to run anything manually.

At a minimum you've added things to your site. But are you sure you haven't touched the server configuration at all?

3 Likes
3

I'm aware that my update method wasn't great. Ironically, one of the reasons why I logged into my server recently was to check out how hard would it be to switch to an automatic way and I find ... this.

What I meant that my website + SSL + LE has been running for years without an issue. If it helps to post content of any configuration file, feel free to request it.

4

Well, something is different recently. Your most recent certificate has 3 domain names in it where prior you had two certificates for those 3 names. See your history below from crt.sh.

Also, the failure message only shows two domain names, not three. As if it is renewing an older cert from Mar8.

Let's try to sort this out by showing us output of these two commands

sudo certbot certificates

sudo apache2ctl -t -D DUMP_VHOSTS

image
image1833Ă—522 107 KB

3 Likes
5

Interesting, I don't recall doing anything like that in such a near past, hmm. Now I have realised that I have pasted only a partial output, for one domain, maybe that wasn't such a great idea. So, one more time, the complete output of certbot renew -v:

root@localhost:~# certbot renew -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mikro.naprvyraz.sk.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for mikro.naprvyraz.sk
Performing the following challenges:
http-01 challenge for mikro.naprvyraz.sk
Waiting for verification...
Challenge failed for domain mikro.naprvyraz.sk
http-01 challenge for mikro.naprvyraz.sk

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: mikro.naprvyraz.sk
  Type:   unauthorized
  Detail: 176.58.103.186: Invalid response from http://mikro.naprvyraz.sk/.well-known/acme-challenge/0KphQOEv98NVf77RPtNlN_FNDbGg2ou55HKX5DEfKH8: 404

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate mikro.naprvyraz.sk with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/naprvyraz.sk.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for naprvyraz.sk and 3 more domains
Performing the following challenges:
http-01 challenge for naprvyraz.sk
http-01 challenge for www.naprvyraz.sk
Waiting for verification...
Challenge failed for domain naprvyraz.sk
Challenge failed for domain www.naprvyraz.sk
http-01 challenge for naprvyraz.sk
http-01 challenge for www.naprvyraz.sk

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: naprvyraz.sk
  Type:   unauthorized
  Detail: 176.58.103.186: Invalid response from http://naprvyraz.sk/.well-known/acme-challenge/FF7GcQBxt7RawIQadJE2NJgRz5rSUTuz28_3FN5K6Ig: 404

  Domain: www.naprvyraz.sk
  Type:   unauthorized
  Detail: 176.58.103.186: Invalid response from http://www.naprvyraz.sk/.well-known/acme-challenge/rQYDV9NMNbojq57kAuWA8JmXU8ZV1UJbI3Mdo6pvAoY: 404

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate naprvyraz.sk with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/mikro.naprvyraz.sk/fullchain.pem (failure)
  /etc/letsencrypt/live/naprvyraz.sk/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Output of certbot certificates:

root@localhost:~# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: mikro.naprvyraz.sk
    Serial Number: 4a529801612df92865e791e10781f538291
    Key Type: RSA
    Domains: mikro.naprvyraz.sk
    Expiry Date: 2025-06-08 19:43:19+00:00 (VALID: 6 days)
    Certificate Path: /etc/letsencrypt/live/mikro.naprvyraz.sk/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mikro.naprvyraz.sk/privkey.pem
  Certificate Name: naprvyraz.sk
    Serial Number: 42b9fb213a25ca618f1d2c0868fb525c625
    Key Type: RSA
    Domains: naprvyraz.sk wifon.sk www.naprvyraz.sk www.wifon.sk
    Expiry Date: 2025-06-06 19:45:09+00:00 (VALID: 4 days)
    Certificate Path: /etc/letsencrypt/live/naprvyraz.sk/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/naprvyraz.sk/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Output of apache2ctl -t -D DUMP_VHOSTS:

root@localhost:~# apache2ctl -t -D DUMP_VHOSTS
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using ::1. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server naprvyraz.sk (/etc/apache2/sites-enabled/naprvyraz.sk-le-ssl.conf:4)
         port 443 namevhost naprvyraz.sk (/etc/apache2/sites-enabled/naprvyraz.sk-le-ssl.conf:4)
                 alias www.naprvyraz.sk
                 alias mikro.naprvyraz.sk
         port 443 namevhost wifon.sk (/etc/apache2/sites-enabled/wifon.sk-le-ssl.conf:2)
                 alias www.wifon.sk
*:80                   is a NameVirtualHost
         default server naprvyraz.sk (/etc/apache2/sites-enabled/naprvyraz.sk.conf:1)
         port 80 namevhost naprvyraz.sk (/etc/apache2/sites-enabled/naprvyraz.sk.conf:1)
                 alias www.naprvyraz.sk
                 alias mikro.naprvyraz.sk
         port 80 namevhost wifon.sk (/etc/apache2/sites-enabled/wifon.sk.conf:1)
                 alias www.wifon.sk
1 Like
6

Would you show the contents of that file?

Something is odd. All 5 of your domains have the same IP which is fine. And, HTTPS requests to your wifon.sk domain use the certificate shown by Certbot command. Also fine.

But, HTTPS requests to your naprvyraz.sk (and its www and mikrok subdomains) use a cert with all 3 names in it which did NOT show in your certbot list. Somewhere that cert got requested and is being actively used. See the SANs list and expiration date of the cert you are using for those: SSL Checker And, compare that to the Certbot list you showed.

Do you have multiple servers? Are you somehow routing HTTP(s) requests for naprvyraz.sk to a different server?

3 Likes
7

That was a great observation, thanks. Indeed, those certificates totally don't match. Looking at /etc/apache2/sites-enabled/naprvyraz.sk-le-ssl.conf:

MDomain naprvyraz.sk

<IfModule mod_ssl.c>
<VirtualHost *:443>
     ServerAdmin miro.kropacek@gmail.com
     ServerName naprvyraz.sk
     ServerAlias www.naprvyraz.sk
     ServerAlias mikro.naprvyraz.sk

     DocumentRoot /var/www/naprvyraz.sk

     <Directory /var/www/naprvyraz.sk>
         Options Indexes FollowSymLinks
         AllowOverride All
         Require all granted
     </Directory>

     ErrorLog ${APACHE_LOG_DIR}/error.log 
     CustomLog ${APACHE_LOG_DIR}/access.log combined 
 
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/mikro.naprvyraz.sk/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/mikro.naprvyraz.sk/privkey.pem
</VirtualHost>
</IfModule>

and /etc/apache2/sites-enabled/wifon.sk-le-ssl.conf:

<IfModule mod_ssl.c>
<VirtualHost *:443>
     ServerAdmin miro.kropacek@gmail.com
     ServerName wifon.sk
     ServerAlias www.wifon.sk

     DocumentRoot /var/www/wifon.sk

     <Directory /var/www/wifon.sk>
         Options Indexes FollowSymLinks
         AllowOverride All
         Require all granted
     </Directory>

     ErrorLog ${APACHE_LOG_DIR}/error.log 
     CustomLog ${APACHE_LOG_DIR}/access.log combined 
 
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/naprvyraz.sk/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/naprvyraz.sk/privkey.pem
</VirtualHost>
</IfModule>

hasn't revealed anything suspicious to me (except the pretty messed up usage of the certificate paths: wifon.sk uses naprvyraz.sk's and naprvyraz.sk uses mikro.naprvyraz.sk's, no clue what was this about) but your hint gave me an idea to look for other certificates and indeed:

[...]
/etc/apache2/md/accounts/ACME-.letsencrypt.org-0000/account.pem
/etc/apache2/md/domains/naprvyraz.sk/pubcert.pem
/etc/apache2/md/domains/naprvyraz.sk/privkey.pem

were quite suspicious:

root@localhost:~# openssl x509 -in /etc/apache2/md/domains/naprvyraz.sk/pubcert.pem -noout -text | grep DNS
                DNS:mikro.naprvyraz.sk, DNS:naprvyraz.sk, DNS:www.naprvyraz.sk

so that's our guy. However I have not only clue how this got there but what's worse, I don't know how to change it. Naively greping for "pubcert.pem" in /etc didn't reveal any matches.

8

Hmm, checking that path more closely...

root@localhost:/etc# ls -l /etc/apache2/md/domains/naprvyraz.sk
total 20
-rw------- 1 root root 6445 Apr 29 14:49 job.json
-rw------- 1 root root  757 Apr 29 14:49 md.json
-rw------- 1 root root 1704 Apr 29 14:49 privkey.pem
-rw------- 1 root root 3684 Apr 29 14:49 pubcert.pem

shows me not only the same date but also the content of those json files clearly implies that it was LE's job and not mine (phew, so I'm not totally senile).

Are you familiar with those files?

9

When did you start using Apache mod_md instead of Certbot :slight_smile:

Note once you start using mod_md you should remove the SSLCertificateFile and KeyFile that reference Certbot's files. As well as the Include from Certbot

3 Likes
10

Ouch, I know where this comes from. In April I read an article about LE's goal to shorten the renew period and that article recommended to set up Apache's mod_md which supposedly sets everything up and there's no need to care about LE anymore: HTTPS certifikát Let's Encrypt pomocĂ­ Apache a vestavÄ›nĂ©ho modulu mod_md - Root.cz.

So now I'm confused from my own actions. :slight_smile: Since I received the email about my domain's expiration, clearly it didn't work as planned. And on the other hand, certbot doesn't work anymore either because (I guess) mod_md took over admistration.

If it's not too much to ask, can you help me out of this (self-inflicted) trouble?

11

Sure, probably.

First, using mod_md is not "instead of" Let's Encrypt. Let's Encrypt is an ACME Server (and public Certificate Authority) that issues certificates.

You have mod_md still getting certs from LE. It is an ACME Client like Certbot so replaces Certbot.

I would start by removing those 3 lines (or at least commenting them out). Restart (not just graceful reload) Apache and make sure HTTPS connections still work. I am highly confident they will since the cert mod_md got from LE is being used already.

Once you confirm that we'll take the next steps. These involve setting up mod_md for your other domain and then properly deleting the Certbot certificates so it does not keep asking LE for them (and failing).

4 Likes
12

Excuse my wording, yes, what I meant to say that certbot has been replaced by mod_md at that point.

In the meantime I have figured that much, too -- i.e. that most likely my mod_md setup wasn't a total failure and it actually works as it should. The emails from LE were just telling me that the good old certificates "naprvyraz.sk" (and friends) and "mikro.naprvyraz.sk" are about to expire for the last time.

Ok, so after a few hiccups (I didn't realise that I have to keep wifon.sk still intact as it doesn't have mod_mod enabled otherwise apache2 wouldn't start) I was able to confirm that SSLCertificateFile and SSLCertificateKeyFile are indeed not needed. However what is needed is the include:

SSLEngine on

# Intermediate configuration, tweak to your needs
SSLProtocol             all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder     off
SSLSessionTickets       off

SSLOptions +StrictRequire

# Add vhost name to log entries:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common

One or all of those parameters makes apache ssl happy, otherwise I'd be getting SSL errors and the SSL Checker would report "443 not open".

Since the article linked previously is quite dumb-proof (see, I have managed to install mod_md thanks to it), I think I should be fine with doing the same for wifon.sk.

However what I am interested in is the last thing you have mentioned -- deleting the Certbot certificates so it does not keep asking LE for them (and failing).

2 Likes
13

Ah, yes, likely just the SSLEngine On line

The others may be used. I don't recall off-hand if mod_md sets those to some defaults or not. In any case they are reasonable. If you plan to keep the include you should probably copy it to your Apache directory and update the include reference in case you fully delete Certbot.

Once you have mod_md for both do:

sudo certbot delete --cert-name X

Where X is the name from certbot certificates

In your case just doing sudo certbot delete may prompt you which one or all of them to delete too.

Once you are fully on mod_md and these are deleted you could uninstall Certbot

4 Likes
14

Thank you for your assistance. It's ironic that in the end I didn't have to do anything and it would be fine but it was a great insight of how LE/mod_md really work together.

2 Likes