Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
It produced this output:
Upgrading certbot-auto 1.1.0 to 1.2.0…
Replacing certbot-auto…
Creating virtual environment…
Installing Python packages…
Installation succeeded.
Processing /etc/letsencrypt/renewal/hork.com.conf
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Waiting for verification…
Challenge failed for domain hork.com
Challenge failed for domain www.hork.com
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Cleaning up challenges
My web server is (include version):
apache 2.4.27-3.fc25
The operating system my web server runs on is (include version):
Fedora-25 linux-4.13.16-100.fc25.x86_64
My hosting provider, if applicable, is:
self
I can login to a root shell on my machine (yes or no, or I don’t know):
yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 1.2.0
This was my first attempt at certificate update, after initial install, 3 month ago with certbot-auto version 1.1, went without a hitch. (and no, nothing was moved since then). I found no further clues in the log file to point me at a specific problem.
I don’t think the directories DocumentRoot/.well-known/acme-challenge/ are being created. If that is true then it is no wonder the challenge file is not found. I think that because:
The modification time of the DocumentRoot directory did not change.
On a 2nd attempt, I ran an infinite loop to monitor the contents of the DocumentRoot every 1s, and that never showed the .well-known
sub-directory. Since the letsencrypt cert has now expired, I reverted back to using a self-signed cert, moved the /etc/letsencrypt tree out of the way and tried from scratch with: certbot-auto certonly --apache
which gave the same result.
Scanning trough the python source (not my cup of tea), I only find a single script concerned with .well-known, which is http_01.py It does not create the directories. Of course if the directory name is captured in a variable, I wouldn’t find it. I know that .well-known wasn’t (accidentally) created in any of the other DocumentRoot directories on this server, which runs 3 virtual hosts.
It would be much appreciated if someone could unravel this mystery to me. Thanks.
You can check all the available certbot options with --help all or read them online.
There you can find the option --debug-challenges. If you use that, you can check the deployment of the challenge file. As @JuergenAuer already said, the apache authenticator uses an other directory than your DocumentRoot. It generates a temporary Apache configuration file and I think the challenge file goes somewhere in /var/letsencrypt (not really sure, but it’s some default directory in /var).
Use the --debug-challenges, when certbot is waiting for user input, see if you can retrieve the challenge file, do a tail -f /var/log/path/to/apache/error/log and let certbot continue: your apache error log might record a 404 file not found error or something.
Also, the letsencrypt.log file should give you a reason why the challenge failed. I would like to see that too. I think it should be at the bottom of the command line output of certbot too.
Thanks Juergen, that produced a lot more output, but no certificate. After creating the .well-know/acme-challenge directories and making sure they were writable for apache, I ran this new command:
I am looking trough all that output, to see what might be significant as to why things don't work.
A new key is generated; Storing nonce; A POST is send to acme-v02.api.letsencrypt.org;
Performing the following challenges:
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Using the webroot path /var/www/hork.com/html for all unmatched domains.
Creating root challenges validation dir at /var/www/hork.com/html/.well-known/acme-challenge
Creating root challenges validation dir at /var/www/hork.com/html/.well-known/acme-challenge
Attempting to save validation to /var/www/hork.com/html/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8
Attempting to save validation to /var/www/hork.com/html/.well-known/acme-challenge/KDB0w6MzNxmA7a0CpU5wbQNI9IrayX_VgurMUBOHcDg
Waiting for verification...
{
"identifier": {
"type": "dns",
"value": "hork.com"
},
"status": "invalid",
"expires": "2020-03-07T20:58:49Z",
"challenges": [
{
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:ietf:params:acme:error:connection",
"detail": "During secondary validation: Fetching http://hork.com/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8: Error getting validation data",
"status": 400
},
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/3080939811/lTbxuA",
"token": "g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8",
"validationRecord": [
{
"url": "http://hork.com/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8",
"hostname": "hork.com",
"port": "80",
"addressesResolved": [
"68.34.23.141"
],
"addressUsed": "68.34.23.141"
}
]
}
]
}
I think I see where things may be going wrong. IF the server is addressed by the IP address rather than by name, it defaults to my other virtual host: www.virtualcrash.com which of course has its own DocumentRoot, where the challenge file will not be found. Is that what is happening here and is there a way around? Thanks.
No, "addressUsed" is just there for you to see which IP address was used to connect to (TCP connections work by connecting to an IP address, not hostname directly) in the case of multiple IP addresses per hostname. Of course the Let's Encrypt ACME server uses the HTTP "host" header, just like any other (non-ancient) web client. In your case you only have one IP address, so you can ignore addressUsed. Of course, a double check to see if it is the same IP address as your server isn't a bad idea
Have you checked your error log for the error generated by the attempt to retrieve the challenge file?
Thanks for that Osiris,
The error_log file indeed shows:
[Sat Feb 29 16:28:31.719867 2020] [core:info] [pid 27428] [client 80.100.40.176:36558] AH00128: File does not exist: /var/www/hork.com/html/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8
[Sat Feb 29 16:28:39.179935 2020] [core:info] [pid 27423] [client 80.100.40.176:36562] AH00128: File does not exist: /var/www/hork.com/html/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8
So, the issue is /not/ that it is looked for in the wrong place, it is still that it does not actually exist. The search continues.
What happens if you just run “certbot-auto renew”, without adding any other options? Certbot is designed to remember what you did last time, you don’t have to specify it over again. Do you need to change the renewal configuration?
What does /etc/letsencrypt/renewal/hork.com.conf contain?
Edit: Also also, 80.100.40.176 isn’t one of Let’s Encrypt’s validation IP addresses.
Every attempt to access the challenge file when certbot isn't providing one will result in such an error indeed. @crashulater should look at the timestamps and only look at the log entries when certbot ran.
Output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Processing /etc/letsencrypt/renewal/hork.com.conf
Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator webroot, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Using the webroot path /var/www/hork.com/html for all unmatched domains.
Waiting for verification…
Challenges loaded. Press continue to submit to CA. Pass “-v” for more info about
challenges.
Challenge failed for domain hork.com
Challenge failed for domain www.hork.com
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Cleaning up challenges
Attempting to renew cert (hork.com) from /etc/letsencrypt/renewal/hork.com.conf produced an unexpected error: Some challenges have failed… Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/hork.com/fullchain.pem (failure)
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/hork.com/fullchain.pem (failure)
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you’re using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.
The good news is that the modification time of .well-known/acme-challenge has changed, so a file was created there, though I wasn’t quick enough to catch that before it obviously was deleted again. Also, no error messages in the httpd log between 55:06 and 55:16, which was the time of the challenge. No new cert though.
Well, that's where --debug-challenges is for! Just don't press continue, but check your documentroot before you press continue and start a tail -f /var/log/path/to/relevant/apache/error/log on the Apache error log, so you can see what happens when you do press "continue" in certbot.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Processing /etc/letsencrypt/renewal/hork.com.conf
Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Waiting for verification…
Challenge failed for domain hork.com
Challenge failed for domain www.hork.com
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Cleaning up challenges
Attempting to renew cert (hork.com) from /etc/letsencrypt/renewal/hork.com.conf produced an unexpected error: Some challenges have failed… Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/hork.com/fullchain.pem (failure)
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/hork.com/fullchain.pem (failure)
Firewall is always a possibility as mine gets dynamically updated by my watch dog. However, I find this in the access_log, which seems to indicate the communication succeeded, though a 404 “file not found” error was generated in each case:
17.58.81.139 - - [29/Feb/2020:17:06:40 -0500] “GET /.well-known/acme-challenge/KZkHos9s__pZ4bbjGrSvKQnpu_FBM9_39th1uwWHT-0: HTTP/1.1” 404 269 “-” “AppleNewsBot”
17.58.81.139 - - [29/Feb/2020:17:06:40 -0500] “GET /.well-known/acme-challenge/2eQDYGamtkEcdh1xL2iWeigUn2z9ede1h7RdJdHOfLg: HTTP/1.1” 404 269 “-” “AppleNewsBot”
17.58.81.139 - - [29/Feb/2020:17:06:41 -0500] “GET /.well-known/acme-challenge/BOLKgnEA4UnGBBnh5rlVpyYQnLbFsjqNqRWug3CH_bs: HTTP/1.1” 404 269 “-” “AppleNewsBot”
17.58.81.139 - - [29/Feb/2020:17:06:41 -0500] “GET /.well-known/acme-challenge/uK1vvrSu_f_OavUVWf-QpXO-AIdHyBY5yQfuYkk3foo: HTTP/1.1” 404 269 “-” “AppleNewsBot”
199.250.251.36 - - [29/Feb/2020:17:08:54 -0500] “GET /.well-known/acme-challenge/BOLKgnEA4UnGBBnh5rlVpyYQnLbFsjqNqRWug3CH_bs: HTTP/1.1” 404 269 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36”
63.117.14.68 - - [29/Feb/2020:17:08:54 -0500] “GET /.well-known/acme-challenge/BOLKgnEA4UnGBBnh5rlVpyYQnLbFsjqNqRWug3CH_bs HTTP/1.1” 404 268 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36”
So, it is still possible that the directory was touch’d but the file not created.
I could move the /etc/letsencrypcertbot-auto t directory out of the way and try a new initialization with:
is your server reachable on port 80? is it serving files/directories starting with dots without blocking them? is the same config applied to port 80 and 443?
Probably not. The useragent header of the Let's Encrypt validation servers don'tuse "AppleNewsBot" or "Safari", but mention quite literally something like "Let's Encrypt validation server". And I don't see such an entry. Perhaps check your access log for succesful entries. Perhaps it's a bug on Let's Encrypts side, you never know!
It shouldn't be necessary to move /etc/letsencrypt out of the way. It should just override your existing certificate. Also, don't forget the l (lowercase L) after htm for your webroot
Thanks for the heads-up 9peppe. Still have visitors with old IE browsers.
Unusual, I may be. But I can live with that
I am also very old, having made my first web site 30 years ago on a Silicon Graphic Iris
This from my httpd.conf
<VirtualHost *:80>
ServerName www.hork.com
ServerAlias ww0.hork.com
ServerAlias hork.com
ServerAlias ocsp.hork.com
ServerAdmin webmaster@hork.com
DocumentRoot /var/www/hork.com/html
There are more virtual hosts in that file and www.hork.com is /not/ the default. This configuration has not changed since I successfully obtained the letsencrypt cert last November, 30th.
(replaced my comment hashes with ; so it won’t scream at you)
So, yes it is reachable on port 80. I just took a peek at http://www.hork.com/.well-known and got me a directory listing. So, no blocking .directories at this time.
