Certbot renew suddenly failing with no relevant config changes

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: caerllewys.net

I ran this command: certbot renew --quiet

It produced this output:

Challenge failed for domain chat.caerllewys.net
Challenge failed for domain jabber.caerllewys.net
Failed to renew certificate www.caerllewys.net with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/www.caerllewys.net/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

My web server is (include version): Apache 2.4.46

The operating system my web server runs on is (include version): Gentoo Linux
(Gentoo Base System release 2.7, profile default/linux/amd64/17.1)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

Yes, of course

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.15.0

Renewal config:

renew_before_expiry = 30 days

version = 1.13.0
archive_dir = /etc/letsencrypt/archive/www.caerllewys.net
cert = /etc/letsencrypt/live/www.caerllewys.net/cert.pem
privkey = /etc/letsencrypt/live/www.caerllewys.net/privkey.pem
chain = /etc/letsencrypt/live/www.caerllewys.net/chain.pem
fullchain = /etc/letsencrypt/live/www.caerllewys.net/fullchain.pem

Options used in the renewal process

[renewalparams]
authenticator = webroot
account = cac105b2e2ba985619b301a2b38ca0e2
rsa_key_size = 4096
server = https://acme-v02.api.letsencrypt.org/directory
webroot_path = /var/httpd/htdocs,
[[webroot_map]]
chat.caerllewys.net = /var/httpd/htdocs
jabber.caerllewys.net = /var/httpd/htdocs
smtp.caerllewys.net = /var/httpd/htdocs
www.caerllewys.net = /var/httpd/htdocs

certbot renew without --quiet reports the following:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/www.caerllewys.net.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate for www.caerllewys.net and 3 more domains
Performing the following challenges:
http-01 challenge for chat.caerllewys.net
http-01 challenge for jabber.caerllewys.net
Using the webroot path /var/httpd/htdocs for all unmatched domains.
Waiting for verification...
Challenge failed for domain chat.caerllewys.net
Challenge failed for domain jabber.caerllewys.net
http-01 challenge for chat.caerllewys.net
http-01 challenge for jabber.caerllewys.net
Cleaning up challenges
Failed to renew certificate www.caerllewys.net with error: Some challenges have failed.

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/www.caerllewys.net/fullchain.pem (failure)

Running post-hook command: /etc/letsencrypt/renewal-hooks/post/reload_daemons.sh
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

This was all working perfectly up until a few days ago, with certbot 1.14.0, then suddenly stopped working with no recent config changes on my part. I updated to 1.15.0 while trying to troubleshoot the problem, but it has made no difference.

The only thing I can think of that could be at issue is that some months ago (early March) I FIXED broken VirtualHost directives from:

ServerAdmin root@caerllewys.net ServerName jabber.caerllewys.net Redirect permanent / https://jabber.caerllewys.net/ ServerAdmin root@caerllewys.net ServerName chat.caerllewys.net Redirect permanent / https://jabber.caerllewys.net/

which was causing http-to-https redirect failures on the two virtualhosts that are now failing, to:

<VirtualHost 10.24.32.15:80 10.24.32.18:80>
ServerAdmin root@caerllewys.net
ServerName jabber.caerllewys.net
Redirect permanent / https://jabber.caerllewys.net/

<VirtualHost 10.24.32.15:80 10.24.32.18:80>
ServerAdmin root@caerllewys.net
ServerName chat.caerllewys.net
Redirect permanent / https://jabber.caerllewys.net/

which works properly. Could this be a factor in the problem? It only in the last few minutes occurred to me that the two subdomains that are failing acme authentication NOW are the two subdomains on which http-https redirection USED to be broken. Do I need to reinitialize certbot? Do I need to change the renewal configuration? Based on other tickets, do I need to add a pre-renewal hook to stop Apache and then restart it afterwards in my post-hook?

I can provide the letsencrypt.log if needed. I currently have the following hooks:

minbar:root:/etc/letsencrypt:12 # cat renewal-hooks/post/reload_daemons.sh
#!/bin/bash

ejabberdctl restart
postfix reload
minbar:root:/etc/letsencrypt:13 # cat renewal-hooks/deploy/deploy_jabber.sh
#!/bin/bash

cat /etc/letsencrypt/live/www.caerllewys.net/privkey.pem /etc/letsencrypt/live/www.caerllewys.net/fullchain.pem > /etc/jabber/server.pem

2

What's the HTTPS <VirtualHost> for jabber.caerllewys.net?

In principle, those redirects aren't an issue as long as the path is conserved, which is the case. However, I'm not getting a regular 404 File not found page from Apache, but a different kind of error page.

3

<VirtualHost 10.24.32.15:443 10.24.32.18:443>
ServerAdmin root@caerllewys.net
ServerName jabber.caerllewys.net
DocumentRoot "/var/httpd/htdocs/jabber"
CustomLog /var/log/apache2/jabber.log combined
ErrorLog /var/log/apache2/jabber.err
LogLevel notice

SSLCertificateFile /etc/letsencrypt/live/www.caerllewys.net/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.caerllewys.net/privkey.pem
Include /etc/apache2/options-ssl-apache.conf
Include /etc/apache2/ssl-strong.conf

ScriptAlias /perl /var/httpd/cgi-perl/

ErrorDocument 403 /cgi-perl/403.cgi
ErrorDocument 404 /cgi-perl/404.cgi

AddType text/html .html .shtml
AddOutputFilter INCLUDES .html .shtml

<Files ~ "\.(html|cgi)$">
    HostnameLookups on
</Files>

<Directory "/var/httpd/htdocs/jabber">
    Options Indexes FollowSymLinks IncludesNOEXEC MultiViews
    AllowOverride None
    Require all granted
</Directory>

<Directory "/var/httpd/cgi-perl">
    ExpiresDefault "access plus 1 minute"
    Options +Includes +ExecCGI
    AllowOverride None
    Require all granted
</Directory>

<Directory "/var/httpd/htdocs/sys">
    Options +Includes +ExecCGI
    AllowOverride None
    Require all granted
</Directory>

<Directory "/var/httpd/icons">
    Options +Indexes +MultiViews
</Directory>

Should this have a ServerAlias chat.caerllewys.net ? Would it make a difference?

4

This DocumentRoot is not the same as the webroot-path set in your renewal configuration file:

Have you made any changes to the DocumentRoot?

1 Like
5

AAAAAHHHH! I totally missed that. Let me edit the renewal config and try it again.

I think I had set it up that way at some point to try to get it to use the same token/nonce to renew all, since separate ones didn't seem to be working.

6

That was the problem! Thank you!

Now I'll spend the next week scratching my head over why it was ever WORKING.

2 Likes
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.