My domain is: gnucash.org
I ran this command: certbot renew --dry-run
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gnucash.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.gnucash.org
http-01 challenge for gnucash.org
Enabled Apache rewrite module
Waiting for verification...
Challenge failed for domain www.gnucash.org
http-01 challenge for www.gnucash.org
Cleaning up challenges
Attempting to renew cert (gnucash.org) from /etc/letsencrypt/renewal/gnucash.org.conf produced an unexpected error: Some challenges have failed.. Skipping.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.gnucash.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.gnucash.org
Enabled Apache rewrite module
Waiting for verification...
Challenge failed for domain www.gnucash.org
http-01 challenge for www.gnucash.org
Cleaning up challenges
Attempting to renew cert (www.gnucash.org) from /etc/letsencrypt/renewal/www.gnucash.org.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/gnucash.org/fullchain.pem (failure)
/etc/letsencrypt/live/www.gnucash.org/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/gnucash.org/fullchain.pem (failure)
/etc/letsencrypt/live/www.gnucash.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 renew failure(s), 0 parse failure(s)
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: www.gnucash.org
Type: unauthorized
Detail: 67.198.37.17: Invalid response from
http://www.gnucash.org/.well-known/acme-challenge/LFR4SSD7fUAvgZsgRS3I2opfaQQwuThJqpv2fHf1_CU:
404
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
- The following errors were reported by the server:
Domain: www.gnucash.org
Type: unauthorized
Detail: 67.198.37.17: Invalid response from
http://www.gnucash.org/.well-known/acme-challenge/VV_0FqkV1JB6fbI4dQp--v1bldHYFJHBR86EIc9c13Y:
404
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
My web server is (include version): Apache2 2.4.41-4ubuntu3.14 amd64
The operating system my web server runs on is (include version): Ubuntu 20.04 LTS - focal
My hosting provider, if applicable, is: none/myself
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version
or certbot-auto --version
if you're using Certbot): certbot 0.40.0
Some more details:
- I've been running certbot since 2016, and its worked great (a few hiccups in the early days)
- It's been auto-renewing ever two months, just fine.
- I attempted to upgrade to Ubuntu 22.04 LTS (from 20.04 LTS) and everything broke, wildly, in every which way. Completely unrelated to certbot, just other random stuff, a long laundry list of crazy stuff that broke.
- I downgraded back to 20.04 LTS and everything went back to normal ... except for certbot.
- Renewals stopped working; I've scoured the net for advice & recommendations, but nothing seems to work. (there are over a dozen pages dealing with my specific error. None cure the problem.)
- To debug the problem, I created the following Apache2 config file:
NameVirtualHost *:80
<VirtualHost *:80>
ServerName www.gnucash.org
ServerAlias gnucash.org
ServerAdmin webmaster@gnucash.org
# Redirect / https://www.gnucash.org/
# Hacks to enable SSL cert renewal
DocumentRoot /var/lib/letsencrypt/http_challenges
# Do NOT put trailing slash at the end of these!!
Alias /.well-known/acme-challenge /var/lib/letsencrypt/http_challenges
# `Allow from All` is the old apache 2.2 config
# `Require all granted` is the new apache 2.4 config
# Do NOT put trailing slash on either Directory or Location!
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted
Satisfy Any
AllowOverride None
ForceType text/plain
Options +Indexes
</Directory>
<Location /.well-known/acme-challenge>
Require all granted
</Location>
</VirtualHost>
-
I created a file:
/var/lib/letsencrypt/http_challenges/foo
with contentsasdf
-- you can view it here:
Index of /.well-known/acme-challenge -
and also here:
http://gnucash.org/ -
These are live, right now; I'm leaving the server open till things get debugged. The actual live website (with expired cert) is here: https://gnucash.org/ -- It still works, as long as you accept the expired-cert menu.
-
The
/var/log/apache2
directory contains five logfiles:
access.log gnucash-access.log other_vhosts_access.log
error.log gnucash-error.log
- I watch these during the renewal. For the two gnucash logfiles, nothing happens (as expected; they are logging the https traffic).
- The
error.log
file shows that certbot restarts or reloads apache; it looks fine. - The
other_vhosts_access.log
and theaccess.log
give conflicting messages. The first shows a 200 and the second shows 404 !! For example:
other_vhosts_access.log:
www.gnucash.org:80 18.224.63.216 - - [25/Mar/2023:05:47:17 +0000] "GET /.well-known/acme-challenge/luAcDxDw1uq4GDJDyT5HUt2rfAHCNUPi_Ev5ggYGtMs HTTP/1.1" 200 407 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
but
access.log:
18.224.63.216 - - [25/Mar/2023:05:47:23 +0000] "GET /.well-known/acme-challenge/4-wwAfdgkn1lM6PH8c3YXFKQH7tY8aP3orR4RAUMcCc HTTP/1.1" 404 457 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
-
Note the timestamps are six seconds apart. Note that two different challenges are being used.
-
The first is always 200, the second is always 404.
-
The second is always six seconds after the first.
-
The challenge keys always differ between the first and second.
-
The contents of
/var/log/letsencrypt/letsencrypt.log
are eye-wateringly long and uninformative. I've read them repeatedly and just can't find anything that seems unexpected.
I'm stumped. I don't know what to do. The strange symptoms suggest that maybe certbot is running twice, even though I invoke it only once? And maybe the second time, the directory is being unexpectedly cleaned up? I'm fresh out of ideas.