Failing to renew via either apache or webroot

tansaku · July 26, 2017, 4:35am

My domain is:

https://wiki.healthylondon.org

I ran this command:

sudo certbot renew --webroot -w /opt/bitnami/apache2/htdocs

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/wiki.healthylondon.org.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for wiki.healthylondon.org
Using the webroot path /opt/bitnami/apache2/htdocs for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /opt/bitnami/apache2/htdocs/.well-known/acme-challenge
Attempting to renew cert from /etc/letsencrypt/renewal/wiki.healthylondon.org.conf produced an unexpected error: Failed authorization procedure. wiki.healthylondon.org (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching https://wiki.healthylondon.org.well-known/acme-challenge/<TOKEN>: Error getting validation data. Skipping.

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/wiki.healthylondon.org/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: wiki.healthylondon.org
   Type:   connection
   Detail: Fetching
   https://wiki.healthylondon.org.well-known/acme-challenge/<TOKEN>:
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version):

Apache/2.4.25 (Unix) (via Bitnami)

The operating system my web server runs on is (include version):

Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-116-generic x86_64)

My hosting provider, if applicable, is:

Bitnami MediaWiki stack deployed to Azure

I can login to a root shell on my machine (yes or no, or I don’t know):

yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

no

So I’m able to renew certificates on a staging version of the site, which was originally setup with webroot. However, on this production server where I originally set things up using the apache plugin I am unable to renew the certs. Trying with the apache plugin that I originally used to set this up I get the following:

sudo certbot renew --apache --apache-server-root /opt/bitnami/apache2 --apache-challenge-location /opt/bitnami/apache2 --apache-handle-sites FALSE --apache-vhost-root /opt/bitnami/apache2/conf/bitnami
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/wiki.healthylondon.org.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for wiki.healthylondon.org
Error while running apache2ctl graceful.
httpd not running, trying to start
Action 'graceful' failed.
The Apache error log may have more information.

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs

Cleaning up challenges
Error while running apache2ctl graceful.
httpd not running, trying to start
Action 'graceful' failed.
The Apache error log may have more information.

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs

Encountered exception during recovery
Error while running apache2ctl graceful.
httpd not running, trying to start
Action 'graceful' failed.
The Apache error log may have more information.

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/certbot/error_handler.py", line 99, in _call_registered
    self.funcs[-1]()
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 284, in _cleanup_challenges
    self.auth.cleanup(achalls)
  File "/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py", line 1908, in cleanup
    self.restart()
  File "/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py", line 1797, in restart
    self._reload()
  File "/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py", line 1808, in _reload
    raise errors.MisconfigurationError(str(err))
MisconfigurationError: Error while running apache2ctl graceful.
httpd not running, trying to start
Action 'graceful' failed.
The Apache error log may have more information.

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs

Attempting to renew cert from /etc/letsencrypt/renewal/wiki.healthylondon.org.conf produced an unexpected error: Error while running apache2ctl graceful.
httpd not running, trying to start
Action 'graceful' failed.
The Apache error log may have more information.

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs
. Skipping.

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/wiki.healthylondon.org/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

If I shut down the apache server first I get this error:

sudo certbot renew --apache --apache-server-root /opt/bitnami/apache2 --apache-challenge-location /opt/bitnami/apache2 --apache-handle-sites FALSE --apache-vhost-root /opt/bitnami/apache2/conf/bitnami
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/wiki.healthylondon.org.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for wiki.healthylondon.org
Waiting for verification...
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/wiki.healthylondon.org.conf produced an unexpected error: Failed authorization procedure. wiki.healthylondon.org (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Connection refused. Skipping.

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/wiki.healthylondon.org/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: wiki.healthylondon.org
   Type:   connection
   Detail: Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

After trying various changes to the httpd.conf to try and open up port 80 (to no avail) I gave up and tried the webroot approach - as documented at the start of this post.

I’ve adjusted the httpd.conf to allow the .well-known/acme-challenge folder to be visible - I put in an index.html which I was able to see here (before I deleted it - see below):

https://wiki.healthylondon.org/.well-known/acme-challenge/

but the webroot approach also fails. What seems particularly suspect is that it says it is trying to fetch Fetching https://wiki.healthylondon.org.well-known/acme-challenge/TOKEN so I wonder if the absence of that slash is preventing access to the webserver, although I can see from the logs that the correct hit is coming in:

133.202.198.78 - - [26/Jul/2017:04:00:17 +0000] “GET /.well-known/acme-challenge/ HTTP/1.1” 404 268

I just tried removing the .well-known/acme-challenge/ on the off chance that me having created it was preventing the certbot from creating it - and that didn’t help …

I also just tried doing things manually:

sudo certbot certonly --debug --force-renew -a manual -d wiki.healthylondon.org

and created the file with the contents myself and that also failed:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for wiki.healthylondon.org

-------------------------------------------------------------------------------
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
-------------------------------------------------------------------------------
(Y)es/(N)o: Y

-------------------------------------------------------------------------------
Make sure your web server displays the following content at
http://wiki.healthylondon.org/.well-known/acme-challenge/<TOKEN>-GWi9FcoIrbM before continuing:

<TOKEN>-<KEY>

If you don't have HTTP server configured, you can run the following
command on the target server (as root):

mkdir -p /tmp/certbot/public_html/.well-known/acme-challenge
cd /tmp/certbot/public_html
printf "%s" <TOKEN>-<KEY> > .well-known/acme-challenge<TOKEN>
# run only once per server:
$(command -v python2 || command -v python2.7 || command -v python2.6) -c \
"import BaseHTTPServer, SimpleHTTPServer; \
s = BaseHTTPServer.HTTPServer(('', 80), SimpleHTTPServer.SimpleHTTPRequestHandler); \
s.serve_forever()" 
-------------------------------------------------------------------------------
Press Enter to Continue
Waiting for verification...
Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.14.2', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 742, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 682, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 77, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/dist-packages/certbot/renewal.py", line 296, in renew_cert
    new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
  File "/usr/lib/python2.7/dist-packages/certbot/client.py", line 313, in obtain_certificate
    self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 81, in get_authorizations
    self._respond(resp, best_effort)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 138, in _respond
    self._poll_challenges(chall_update, best_effort)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 202, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. wiki.healthylondon.org (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching https://wiki.healthylondon.org.well-known/acme-challenge/<TOKEN>: Error getting validation data
Please see the logfiles in /var/log/letsencrypt for more details.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: wiki.healthylondon.org
   Type:   connection
   Detail: Fetching
   https://wiki.healthylondon.org.well-known/acme-challenge/<TOKEN>:
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

looking at the access_log I think that I might have cached the file as empty when I checked it could be loaded:

bitnami@bitnami-mediawiki-8a65:/opt/bitnami/apache2/logs$ tail access_log
64.71.168.196 - - [26/Jul/2017:04:14:24 +0000] "HEAD /.well-known/acme-challenge/ HTTP/1.1" 200 -
64.71.168.196 - - [26/Jul/2017:04:14:24 +0000] "GET /.well-known/acme-challenge/ HTTP/1.1" 200 6
133.202.198.78 - - [26/Jul/2017:04:18:42 +0000] "GET /.well-known/acme-challenge/ HTTP/1.1" 404 225
133.202.198.78 - - [26/Jul/2017:04:18:43 +0000] "GET /favicon.ico HTTP/1.1" 200 99678
66.133.109.36 - - [26/Jul/2017:04:19:05 +0000] "GET /.well-known/acme-challenge/<TOKEN> HTTP/1.1" 301 308
66.133.109.36 - - [26/Jul/2017:04:23:37 +0000] "GET /.well-known/acme-challenge/<TOKEN> HTTP/1.1" 301 308
133.202.198.78 - - [26/Jul/2017:04:30:17 +0000] "GET /.well-known/acme-challenge/<TOKEN> HTTP/1.1" 200 -
133.202.198.78 - - [26/Jul/2017:04:30:35 +0000] "GET /.well-known/acme-challenge/<TOKEN> HTTP/1.1" 200 88
133.202.198.78 - - [26/Jul/2017:04:30:36 +0000] "GET /favicon.ico HTTP/1.1" 200 99678
66.133.109.36 - - [26/Jul/2017:04:30:52 +0000] "GET /.well-known/acme-challenge/<TOKEN> HTTP/1.1" 301 308

I was just about to try again but now I’ve tripped the too many request limits (maybe should have been using dry-run?) urgh …

Any insight into why some of the simpler methods don’t work would be very much appreciated. I’ll have another go at this manual method in a few hours …

pfg · July 26, 2017, 4:53am

The first log shows https://wiki.healthylondon.org.well-known/acme-challenge/... as the failing URL. Note that the protocol is HTTPS. Hitting the HTTP version of that URL gives me the following response header:

Location: https://wiki.healthylondon.org.well-known/acme-challenge/...

In other words, the redirect rule you’re using for HTTP to HTTPS redirects is causing the slash to get swallowed, breaking the URL. Something like RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [NC,R=301,L] in the HTTP vhost should work.

(As an aside, this has come up a couple of times before. Did you find this particular rewrite rule somewhere in a guide? I’d like to see if it’s possible to get it fixed there to avoid others running into this.)

mnordhoff · July 26, 2017, 5:47am

In future, if you run into a similar problem, you should start using --dry-run after a couple failures, yeah.

The failed validation rate limit only lasts for 1 hour, so it should have cleared up by now.

tansaku · July 26, 2017, 11:45am

Thanks, that’s fixed it for the manual update. I don’t think I found that syntax in a guide somewhere, or at least when I look back I see the guides I might have referred to have trailing slashes - I think I was re-typing out by hand and just left the slash off and the basic redirect worked so I stopped there - a good learning experience

Interestingly the webroot approach also didn’t work with this change, although it did get a different error:

bitnami@bitnami-mediawiki-8a65:/opt/bitnami/apache2/conf/bitnami$ sudo certbot renew --dry-run  --webroot -w /opt/bitnami/apps/mediawiki/htdocs
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/wiki.healthylondon.org.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for wiki.healthylondon.org
Using the webroot path /opt/bitnami/apps/mediawiki/htdocs for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/wiki.healthylondon.org.conf produced an unexpected error: Failed authorization procedure. wiki.healthylondon.org (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://wiki.healthylondon.org/.well-known/acme-challenge/<TOKEN>: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p". Skipping.
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/wiki.healthylondon.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: wiki.healthylondon.org
   Type:   unauthorized
   Detail: Invalid response from
   http://wiki.healthylondon.org/.well-known/acme-challenge/<TOKEN>:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.

Maybe I’d need to delete the .well-known/acme-challenge dirs that I created myself for that to work …? Just tried that and got the same error.

Would be nice to figure out how to get the webroot approach to work rather than manual … according to the log it does attempt to create the challenge file in the correct location:

2017-07-26 11:41:17,364:INFO:certbot.plugins.webroot:Using the webroot path /opt/bitnami/apps/mediawiki
/htdocs for all unmatched domains.
2017-07-26 11:41:17,364:DEBUG:certbot.plugins.webroot:Creating root challenges validation dir at /opt/b
itnami/apps/mediawiki/htdocs/.well-known/acme-challenge
2017-07-26 11:41:17,383:DEBUG:certbot.plugins.webroot:Attempting to save validation to /opt/bitnami/app
s/mediawiki/htdocs/.well-known/acme-challenge/<TOKEN>
2017-07-26 11:41:17,384:INFO:certbot.auth_handler:Waiting for verification...

but so I guess the apache server isn’t picking up the file system change in time …?

Would also be great to get the apache plugin version working too, that or the webroot so I can put this in a cron job and not have to worry about it again.

Many thanks again for your help

schoen · July 26, 2017, 6:42pm

In this case the next step is normally to try making a test file yourself in the same directory and seeing if you can see it in a web browser. If you can't, there is something about your web server configuration that needs to be adjusted to ensure that these files are served correctly out of the filesystem.

I don't think that's really a thing on Unix (with respect to a local filesystem). In this case, the certificate authority doesn't try to connect to validate until the Let's Encrypt client says that the challenge is ready, and it doesn't say so until it's created the file, and your local OS has told the client that the file write operation has completed.

schoen · July 26, 2017, 6:43pm

For example, can you make a file appear at http://wiki.healthylondon.org/test.txt? If so, can you make a file appear at http://wiki.healthylondon.org/.well-known/acme-challenge/test.txt? If not, can you figure out what part of your web server configuration is preventing these files from being served properly?

tansaku · August 2, 2017, 11:26am

yes, and I know that I can do that, because that’s how I got the manual renewal to take place. I can totally see files in those locations if I place them there

tansaku · August 2, 2017, 11:27am

I also notice that despite what appeared to be a successful manual update that the site is now showing itself as insecure …

tansaku · August 2, 2017, 12:17pm

I've just run the manual update again and it appears to be back and secure now - totally thought I had renewed with a manual approach two days ago.

This time I had to shut down the server completely and run a separate server ...

jared.m · August 2, 2017, 4:57pm

Looks like you issued one today and one before that on July 26. https://crt.sh/?q=wiki.healthylondon.org

It’s probable that when you issued the one month recently, your web server was never reloaded so the new certificate wasn’t applied. That’s an easy thing to miss.

system · September 1, 2017, 4:58pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Can't update certificate, get weird error Help	19	1088	October 26, 2021
[Solved] Webroot Renewal Fails Help	28	5560	December 1, 2017
Certbot Renew Certificate Issue Help	28	1930	March 30, 2019
Problem with certificate renewal Help	23	1668	January 11, 2019
Bitnami Apache Ubuntu Server Certbot Renew not working Help	13	1076	March 7, 2019

Failing to renew via either apache or webroot

Related topics