Challenge file not created during update

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: hork.com

I ran this command: certbot-auto renew --apache

It produced this output:
Upgrading certbot-auto 1.1.0 to 1.2.0…
Replacing certbot-auto…
Creating virtual environment…
Installing Python packages…
Installation succeeded.
Processing /etc/letsencrypt/renewal/hork.com.conf
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Waiting for verification…
Challenge failed for domain hork.com
Challenge failed for domain www.hork.com
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Cleaning up challenges

My web server is (include version):
apache 2.4.27-3.fc25

The operating system my web server runs on is (include version):
Fedora-25 linux-4.13.16-100.fc25.x86_64
My hosting provider, if applicable, is:
self
I can login to a root shell on my machine (yes or no, or I don’t know):
yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 1.2.0

This was my first attempt at certificate update, after initial install, 3 month ago with certbot-auto version 1.1, went without a hitch. (and no, nothing was moved since then). I found no further clues in the log file to point me at a specific problem.
I don’t think the directories DocumentRoot/.well-known/acme-challenge/ are being created. If that is true then it is no wonder the challenge file is not found. I think that because:

  1. The modification time of the DocumentRoot directory did not change.
  2. On a 2nd attempt, I ran an infinite loop to monitor the contents of the DocumentRoot every 1s, and that never showed the .well-known
    sub-directory. Since the letsencrypt cert has now expired, I reverted back to using a self-signed cert, moved the /etc/letsencrypt tree out of the way and tried from scratch with: certbot-auto certonly --apache
    which gave the same result.
    Scanning trough the python source (not my cup of tea), I only find a single script concerned with .well-known, which is http_01.py It does not create the directories. Of course if the directory name is captured in a variable, I wouldn’t find it. I know that .well-known wasn’t (accidentally) created in any of the other DocumentRoot directories on this server, which runs 3 virtual hosts.
    It would be much appreciated if someone could unravel this mystery to me. Thanks.
1 Like

Hi @crashulater

there

is your answer.

If you use the --apache plugin, a location definition is created, there the content is added.

So

  • it’s not required to create the two subdirectories /.well-known/acme-challenge
  • it’s not required to create a file

Use the -vvv flag

then you see the details. If you want to view the file, use --webroot.

1 Like

You can check all the available certbot options with --help all or read them online.

There you can find the option --debug-challenges. If you use that, you can check the deployment of the challenge file. As @JuergenAuer already said, the apache authenticator uses an other directory than your DocumentRoot. It generates a temporary Apache configuration file and I think the challenge file goes somewhere in /var/letsencrypt (not really sure, but it’s some default directory in /var).

Use the --debug-challenges, when certbot is waiting for user input, see if you can retrieve the challenge file, do a tail -f /var/log/path/to/apache/error/log and let certbot continue: your apache error log might record a 404 file not found error or something.

Also, the letsencrypt.log file should give you a reason why the challenge failed. I would like to see that too. I think it should be at the bottom of the command line output of certbot too.

1 Like

Thanks Juergen, that produced a lot more output, but no certificate. After creating the .well-know/acme-challenge directories and making sure they were writable for apache, I ran this new command:

certbot-auto renew -vvv --webroot -w /var/www/hork.com/html

I am looking trough all that output, to see what might be significant as to why things don’t work.
A new key is generated; Storing nonce; A POST is send to acme-v02.api.letsencrypt.org;
Performing the following challenges:
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Using the webroot path /var/www/hork.com/html for all unmatched domains.
Creating root challenges validation dir at /var/www/hork.com/html/.well-known/acme-challenge
Creating root challenges validation dir at /var/www/hork.com/html/.well-known/acme-challenge
Attempting to save validation to /var/www/hork.com/html/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8
Attempting to save validation to /var/www/hork.com/html/.well-known/acme-challenge/KDB0w6MzNxmA7a0CpU5wbQNI9IrayX_VgurMUBOHcDg
Waiting for verification…
{
“identifier”: {
“type”: “dns”,
“value”: “hork.com
},
“status”: “invalid”,
“expires”: “2020-03-07T20:58:49Z”,
“challenges”: [
{
“type”: “http-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:connection”,
“detail”: “During secondary validation: Fetching http://hork.com/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8: Error getting validation data”,
“status”: 400
},
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/3080939811/lTbxuA”,
“token”: “g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8”,
“validationRecord”: [
{
“url”: “http://hork.com/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8”,
“hostname”: “hork.com”,
“port”: “80”,
“addressesResolved”: [
“68.34.23.141”
],
“addressUsed”: “68.34.23.141”
}
]
}
]
}
I think I see where things may be going wrong. IF the server is addressed by the IP address rather than by name, it defaults to my other virtual host: www.virtualcrash.com which of course has its own DocumentRoot, where the challenge file will not be found. Is that what is happening here and is there a way around? Thanks.

No, “addressUsed” is just there for you to see which IP address was used to connect to (TCP connections work by connecting to an IP address, not hostname directly) in the case of multiple IP addresses per hostname. Of course the Let’s Encrypt ACME server uses the HTTP “host” header, just like any other (non-ancient) web client. In your case you only have one IP address, so you can ignore addressUsed. Of course, a double check to see if it is the same IP address as your server isn’t a bad idea :wink:

Have you checked your error log for the error generated by the attempt to retrieve the challenge file?

Thanks for that Osiris,
The error_log file indeed shows:
[Sat Feb 29 16:28:31.719867 2020] [core:info] [pid 27428] [client 80.100.40.176:36558] AH00128: File does not exist: /var/www/hork.com/html/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8
[Sat Feb 29 16:28:39.179935 2020] [core:info] [pid 27423] [client 80.100.40.176:36562] AH00128: File does not exist: /var/www/hork.com/html/.well-known/acme-challenge/g_hqYHn6Plg8pT0DvM7bqjcppOqNvA8rnNvUE9jpKz8
So, the issue is /not/ that it is looked for in the wrong place, it is still that it does not actually exist. The search continues.

Please see my earlier post about the “–debug-challenges” option. Especially in combination with the webroot authenticator, this may be of much help.

Also, in the error message you’ve provided it says “During secondary validation”. This has been added just recently (25 days ago) in the ACME server of Let’s Encrypt. It means there is an issue with “multi-perspective validation”: Let’s Encrypt uses multiple challenge retrievals to combat possible security issues with BGP. Read more about it in the blog post about " Multi-Perspective Validation Improves Domain Validation Security".

Also…

What happens if you just run “certbot-auto renew”, without adding any other options? Certbot is designed to remember what you did last time, you don’t have to specify it over again. Do you need to change the renewal configuration?

What does /etc/letsencrypt/renewal/hork.com.conf contain?

Edit: Also also, 80.100.40.176 isn’t one of Let’s Encrypt’s validation IP addresses.

Every attempt to access the challenge file when certbot isn’t providing one will result in such an error indeed. @crashulater should look at the timestamps and only look at the log entries when certbot ran.

Your guys are great. More things to try than one man can handle :wink:
New command (per Osiris):

certbot-auto renew --webroot --debug-challenges -w /var/www/hork.com/html

Output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/hork.com.conf


Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator webroot, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Using the webroot path /var/www/hork.com/html for all unmatched domains.
Waiting for verification…


Challenges loaded. Press continue to submit to CA. Pass “-v” for more info about
challenges.


Challenge failed for domain hork.com
Challenge failed for domain www.hork.com
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Cleaning up challenges
Attempting to renew cert (hork.com) from /etc/letsencrypt/renewal/hork.com.conf produced an unexpected error: Some challenges have failed… Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/hork.com/fullchain.pem (failure)


All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/hork.com/fullchain.pem (failure)


1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

The good news is that the modification time of .well-known/acme-challenge has changed, so a file was created there, though I wasn’t quick enough to catch that before it obviously was deleted again. Also, no error messages in the httpd log between 55:06 and 55:16, which was the time of the challenge. No new cert though.

Well, that’s where --debug-challenges is for! Just don’t press continue, but check your documentroot before you press continue and start a tail -f /var/log/path/to/relevant/apache/error/log on the Apache error log, so you can see what happens when you do press “continue” in certbot.

1 Like

certbot-auto renew

Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/hork.com.conf


Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Waiting for verification…
Challenge failed for domain hork.com
Challenge failed for domain www.hork.com
http-01 challenge for hork.com
http-01 challenge for www.hork.com
Cleaning up challenges
Attempting to renew cert (hork.com) from /etc/letsencrypt/renewal/hork.com.conf produced an unexpected error: Some challenges have failed… Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/hork.com/fullchain.pem (failure)


All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/hork.com/fullchain.pem (failure)


1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

Mention of secondary validation failing. Does that put the ball in the court of the developers? Thnx.

No it doesn’t. It just means too many of the remote validation servers got an error when trying to access the challenge.

It never waited for input from me, Osiris :frowning:

1 Like

Hm, that’s strange. It seems --debug-challenges doesn’t work when using renew… Not sure if that’s intented or a bug.

Even so, you should be able to check your Apache logs for the exact moment when certbot ran. If you don’t see anything, it might mean your webserver isn’t accessible from the Let’s Encrypt validation servers for some reason. Perhaps a firewall which is too strictly set up. Unfortunately, the error message “Error getting validation data” is some kind of “fallback” error message when Boulder (the Let’s Encrypt ACME server) doesn’t know what to make of the error…. So it could be anything :frowning:

1 Like

Firewall is always a possibility as mine gets dynamically updated by my watch dog. However, I find this in the access_log, which seems to indicate the communication succeeded, though a 404 “file not found” error was generated in each case:
17.58.81.139 - - [29/Feb/2020:17:06:40 -0500] “GET /.well-known/acme-challenge/KZkHos9s__pZ4bbjGrSvKQnpu_FBM9_39th1uwWHT-0: HTTP/1.1” 404 269 “-” “AppleNewsBot”
17.58.81.139 - - [29/Feb/2020:17:06:40 -0500] “GET /.well-known/acme-challenge/2eQDYGamtkEcdh1xL2iWeigUn2z9ede1h7RdJdHOfLg: HTTP/1.1” 404 269 “-” “AppleNewsBot”
17.58.81.139 - - [29/Feb/2020:17:06:41 -0500] “GET /.well-known/acme-challenge/BOLKgnEA4UnGBBnh5rlVpyYQnLbFsjqNqRWug3CH_bs: HTTP/1.1” 404 269 “-” “AppleNewsBot”
17.58.81.139 - - [29/Feb/2020:17:06:41 -0500] “GET /.well-known/acme-challenge/uK1vvrSu_f_OavUVWf-QpXO-AIdHyBY5yQfuYkk3foo: HTTP/1.1” 404 269 “-” “AppleNewsBot”
199.250.251.36 - - [29/Feb/2020:17:08:54 -0500] “GET /.well-known/acme-challenge/BOLKgnEA4UnGBBnh5rlVpyYQnLbFsjqNqRWug3CH_bs: HTTP/1.1” 404 269 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36”
63.117.14.68 - - [29/Feb/2020:17:08:54 -0500] “GET /.well-known/acme-challenge/BOLKgnEA4UnGBBnh5rlVpyYQnLbFsjqNqRWug3CH_bs HTTP/1.1” 404 268 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36”
So, it is still possible that the directory was touch’d but the file not created.
I could move the /etc/letsencrypcertbot-auto t directory out of the way and try a new initialization with:

certbot-auto certonly --webroot --debug-challenges -w /var/www/hork.com/htm

If you think that will provide new insight.

this looks unusual.

are you sure that’s the right directory?

is your server reachable on port 80? is it serving files/directories starting with dots without blocking them? is the same config applied to port 80 and 443?

(also, keep an eye on your tls config, you’ve got insecure ciphers in there: https://www.ssllabs.com/ssltest/analyze.html?d=hork.com&hideResults=on)

1 Like

Probably not. The useragent header of the Let’s Encrypt validation servers don’tuse “AppleNewsBot” or “Safari”, but mention quite literally something like “Let’s Encrypt validation server”. And I don’t see such an entry. Perhaps check your access log for succesful entries. Perhaps it’s a bug on Let’s Encrypts side, you never know!

It shouldn’t be necessary to move /etc/letsencrypt out of the way. It should just override your existing certificate. Also, don’t forget the l (lowercase L) after htm for your webroot :wink:

Your secondary validation fails. So Letsencrypt may find your validation file, the other servers not.

Looks like you block some ip addresses.

Read

The multi-perspective validation is new.

PS: Check if your webroot is correct.

Create

/var/www/hork.com/html/.well-known/acme-challenge

there a file (file name 1234), then try to load that file via

http://hork.com/.well-known/acme-challenge/1234

Thanks for the heads-up 9peppe. Still have visitors with old IE browsers.
Unusual, I may be. But I can live with that :slight_smile:
I am also very old, having made my first web site 30 years ago on a Silicon Graphic Iris :smiley:
This from my httpd.conf
<VirtualHost *:80>
ServerName www.hork.com
ServerAlias ww0.hork.com
ServerAlias hork.com
ServerAlias ocsp.hork.com
ServerAdmin webmaster@hork.com
DocumentRoot /var/www/hork.com/html

<VirtualHost *:443>
ServerName www.hork.com
ServerAlias ww0.hork.com
ServerAlias hork.com
ServerAlias ocsp.hork.com
ServerAdmin webmaster@hork.com
DocumentRoot /var/www/hork.com/html
SSLEngine on
; SSLCertificateFile “/etc/pki/tls/certs/www.hork.crt”
; SSLCertificateKeyFile “/etc/pki/tls/private/www.hork.pem”
SSLCertificateFile “/etc/pki/tls/certs/www_hork_com.pem”
SSLCertificateKeyFile “/etc/pki/tls/private/www_hork_com.pem”
SSLCertificateChainFile “/etc/pki/tls/CA/LetsEncrypt.pem”
SSLCARevocationFile “/etc/pki/tls/crl/hork.crl”
SSLCARevocationCheck leaf

There are more virtual hosts in that file and www.hork.com is /not/ the default. This configuration has not changed since I successfully obtained the letsencrypt cert last November, 30th.
(replaced my comment hashes with ; so it won’t scream at you)
So, yes it is reachable on port 80. I just took a peek at http://www.hork.com/.well-known and got me a directory listing. So, no blocking .directories at this time.

1 Like