Can no longer update certs after changing .htaccess file


#1

Please fill out the fields below so we can help you better.

I ran this command:/usr/bin/letsencrypt certonly --keep-until-expiring --non-interactive --staple-ocsp --must-staple --hsts --redirect --rsa-key-size 2048 --uir --webroot -w /home/sporkschivago/public_html -d www.example.com -d example.com -w /home/sporkschivago/public_html/cpanel -d cpanel.example.com -w /home/sporkschivago/public_html/cpcalendars -d cpcalendars.example.com -w /home/sporkschivago/public_html/cpcontacts -d cpcontacts.example.com -w /home/sporkschivago/public_html/webdisk -d webdisk.example.com -w /home/sporkschivago/public_html/webmail -d webmail.example.com -w /home/sporkschivago/public_html/whm -d whm.example.com -w /usr/local/apache/htdocs -d franklin.example.com --email myrealemail@someplace.com --agree-tos

It produced this output:Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Cert is due for renewal, auto-renewing…
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.example.com
http-01 challenge for example.com
http-01 challenge for cpanel.example.com
http-01 challenge for cpcalendars.example.com
http-01 challenge for cpcontacts.example.com
http-01 challenge for webdisk.example.com
http-01 challenge for webmail.example.com
http-01 challenge for whm.example.com
http-01 challenge for franklin.example.com
Using the webroot path /usr/local/apache/htdocs for all unmatched domains.
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. www.example.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.example.com/.well-known/acme-challenge/NgfmRoCfZ0ofQZSJa_P7CH1KpVr4-1vPUO94g8dh8b8: "

<meta http-equi"

IMPORTANT NOTES:

My operating system is (include version):CentOS 7.3.1611 (Core)

My web server is (include version):Apache/2.4.25 (cPanel)

My hosting provider, if applicable, is:Linode

I can login to a root shell on my machine (yes or no, or I don’t know):Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):I use cPanel and WHM version 11.60.0.35, but I do not use it to install the Let’s Encrypt certificates. I have a crontab entry setup and would like to continue to use the crontab entry.

I have setup my server so no one can go to the insecure version of it. For example, any traffic to port 80 gets redirected to port 443, unless of course, it’s a special subdomain. I have created subdomains for the various cpanel programs (webmail, whm, etc). Here’s a snippet of my .htaccess file. I think there’s something wrong with it:

# Tell the browser to check for index.html and index.php, in that order.
# if either exist, load that file by default.
DirectoryIndex index.php index.html

<IfModule mod_headers.c>
# Turn off caching for Google Chrome.
  Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate, post-check=0, pre-check=0"
  Header set Pragma "no-cache"
  Header set Expires "Wed, 11 Jan 1984 05:00:00 GMT"

# Add P3P Privacy Headers to the site (this causes infinite redirects for some reason).
#  Header set P3P "policyref="/w3c/p3p.xml""
</IfModule>

<IfModule mod_rewrite.c>
#Turn RewriteMod on.
  RewriteEngine On

# Allow .well-known through for Let's Encrypt.
  RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge/

# Redirect https://cpanel.example.com to https://www.example.com:2083
  RewriteCond %{HTTP_HOST} ^cpanel.example.com$
  RewriteRule ^(.*)$ "https\:\/\/www\.example\.com\:2083\/$1" [R=301,L]
...
# Redirect all other users to the https version of our website,
# because we have SSL certs now.
  RewriteCond %{HTTPS} !=on
  RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge/
  RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R=301,L]
</IfModule>

I can post the entire file if needed. I recently added that RewriteCond %{REQUEST_URI}… under the RewriteCond %{HTTPS}!=on, thinking that’s what was causing the problems. If I create the directory .well-known/acme-challenge and create a file called test.html, when I go to Chrome and type in http://www.example.com/.well-known/acme-challenge/test.html, I see my file, but it’s being redirected to the https version, because I’m on the Google Chrome’s pre-load list, but if I use wget to retreive the file, it appears to be retrieved over port 80. I believe Let’s Encrypt’s binary needs to conduct it’s business over port 80, so everything should be good there, as far as I can tell. Any idea what I’m doing wrong?


#2

Let’s Encrypt does check over port 80, but will follow redirects ( including to https )

Can you test with a file with no extension ( i.e. just “test” not “test.html” ) and check that it returns plain text as a response.

If that doesn’t show an obvious answer, please leave the “test” file there and provide us with your real domain name to test.


#3

Thank you for the response serverco. So it’s okay to always redirect the traffic to port 443 for the .well-known stuff? To test, I will remove the .html extension, as suggested, and then go to http://www.example.com/.well-known/acme-challenge/test in Google Chrome, and then look at the source, to make sure just text is being returned. Or is there a better way to test (instead of using Chrome?)

Thanks!


#4

Yes

I use curl - "curl -i http://www.example.com/.well-known/acme-challenge/test " will show the headers as well, and if it’s sent as raw text.


#5

I tried using lynx and Chrome, both seem to load the file in plain-text, as far as I can tell. I view page-source in Chrome and only the word Hello is there. That’s all I have in the test file.

Can I send the domain name in a PM or does it need to be provided in the public forum? Thanks!


#6

I used curl, as suggested. This is the response I got back:
root@franklin:[/home/sporkschivago]# curl -i http://www.example.com/.well-known/acme-challenge/test
HTTP/1.1 200 OK
Date: Tue, 24 Jan 2017 18:29:37 GMT
Server: Apache
Last-Modified: Tue, 24 Jan 2017 15:49:42 GMT
Accept-Ranges: bytes
Content-Length: 6
Cache-Control: max-age=0, no-cache, no-store, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT

Hello

#7

That looks fine.

In the full log at this point …

I can’t see what all the text is ( probably because the html is formatted in this forum ) … does the response it’s seeing there give any clues as to what response it’s getting ?


#8

It’s actually cut off in the log, when I SSH into the machine and look at /var/log/letsencrypt/letsencrypt.log

Here’s what it shows at the very end of the file, where that response is, but without the HTML tags:

!DOCTYPE html
html
    head
    meta http-equiv="Content-type" content="text/html; charset=utf-8"
    meta http-equi"

There’s two new lines before the !DOCTYPE tag, they seem to be part of the response. To me, it almost seems like Let’s Encrypt is looking at the header, instead of the response. Could that be?


#9

It shouldn’t be getting a html response - it should be getting plain text. Unfortunately it doesn’t give enough info to say if that’s your “404 - page not found” page or not. Does the access log show anything related to the access attempt ?


#10

I see also in the log this:

Domain: www.example.com
Type:   unauthorized
Detail: Invalid response from http://www.jetbbs.com/.well-known/acme-challenge/NgfmRoCfZ0ofQZSJa_P7CH1KpVr4-1vPUO94g8dh8b8: "


<!DOCTYPE html>
<html>
    <head>
    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    <meta http-equi"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A record(s) for that domain contain(s) the right IP address.

Again, to me, it looks like my server is sending more than just text. What should the typical response look like, if it was being retrieved by curl? Are there supposed to be headers? If not, for some reason, the response file seems like it’s be treated as HTML and maybe something is adding the HTML tags (Apache for instance?)


#11

Let me check Apache’s access logs, hold on please. I’ll edit this post once I check them. I think we were cross-posting. You answered my questions. I think we’re thinking the same thing.

**EDIT: I see this in the access_logs.

66.133.109.36 - - [24/Nov/2016:03:10:38 -0500] "GET /.well-known/acme-challenge/w6wdbZza913OIFpnEJM1Qrsbi099g19Av55OC2V0bsM HTTP/1.1" 301 180 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [24/Nov/2016:03:10:38 -0500] "GET /.well-known/acme-challenge/pIIEpTQZ6QEX3k8aZRve2EGc4pZ_iMAoxyebxT7eaks HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [24/Nov/2016:03:10:39 -0500] "GET /.well-known/acme-challenge/kZXRRoedL1Wcj55c4h44vUX9FLyV6WKcboFJPm6-YmA HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [24/Nov/2016:03:10:38 -0500] "GET /.well-known/acme-challenge/rvOKgO81WZfrZ6M7mR_XbJYk2_3oWtfIs4t_BgoEpfw HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [24/Nov/2016:03:10:39 -0500] "GET /.well-known/acme-challenge/a3I9gazat9sMfKcvtUEfszgIIExtnYbYrIDOboSr_bE HTTP/1.1" 301 181 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [24/Nov/2016:03:10:39 -0500] "GET /.well-known/acme-challenge/w6wdbZza913OIFpnEJM1Qrsbi099g19Av55OC2V0bsM HTTP/1.1" 401 12586 "http://cpanel.example.com/.well-known/acme-challenge/w6wdbZza913OIFpnEJM1Qrsbi099g19Av55OC2V0bsM" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [24/Nov/2016:03:10:39 -0500] "GET /.well-known/acme-challenge/mYTdPtKz7x1POOYV-K0_bhyL4CorLQnMQDwlevd24eU HTTP/1.1" 301 177 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [24/Nov/2016:03:10:39 -0500] "GET /.well-known/acme-challenge/a3I9gazat9sMfKcvtUEfszgIIExtnYbYrIDOboSr_bE HTTP/1.1" 401 12601 "http://webmail.example.com/.well-known/acme-challenge/a3I9gazat9sMfKcvtUEfszgIIExtnYbYrIDOboSr_bE" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [24/Nov/2016:03:10:39 -0500] "GET /.well-known/acme-challenge/mYTdPtKz7x1POOYV-K0_bhyL4CorLQnMQDwlevd24eU HTTP/1.1" 401 12532 "http://whm.example.com/.well-known/acme-challenge/mYTdPtKz7x1POOYV-K0_bhyL4CorLQnMQDwlevd24eU" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

I wonder why cpanel, webmail, and whm are returning 401’s. I’ve been looking at this as right now, there’s a problem with just example.com and www.example.com, but not with the subdomain ones. I thought Let’s Encrypt was erroring out before it even tried those subdomains. Could there be a problem with my cpanel, whm, and webmail .htaccess files (for example, maybe I forgot to include the .well-known rewrite rule)? Could that cause the error message I’m seeing from Let’s Encrypt?

**EDIT: Now, when I run the command, I see in the access_log:
66.133.109.36 - - [24/Jan/2017:14:08:09 -0500] "GET /.well-known/acme-challenge/zTRFeEBbBgzF7WLHHurtFCMwgBbZLG0vx8ChtSsOK5Y HTTP/1.1" 404 10197 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

So it seems it is now returning a 404 for some reason. Not sure what I did, minus create the directory and leave the test file there.


#12

by default these are incompletely different web-roots. This is one place where you do need to stop the redirect of whm, cpanel etc to https and the alternate ports / locations.


#13

Yes, but I thought I was doing that. This is the .htaccess file under the /home/sporkschivago/public_html/cpanel directory, for instance:

<ifModule mod_headers.c>
  # Try to turn off caching for Google Chrome.
  Header set Expires "Thu, 19 Nov 1981 08:52:00 GM"
  Header set Cache-Control "no-store, no-cache, must-revalidate, post-check=0, pre-check=0"
  Header set Pragma "no-cache"
</ifModule>

RewriteEngine on

# Allow .well-known through for Let's Encrypt.
RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge/

# Redirect http requests to https because we have free SSL certs.
RewriteCond %{HTTP_HOST} ^cpanel.example.com$
RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
RewriteCond %{REQUEST_URI} !^/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
RewriteRule ^(.*)$ "https\:\/\/www\.example\.com\:2083\/$1" [R=301,L]

# Redirect http(s)://(www.)example.com/cpanel to https://www.example.com:2083/$1
RewriteCond %{REQUEST_URI} ^(.*)$
RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge/
RewriteRule ^(.*)$ "https\:\/\/www\.example\.com\:2083\/$1" [R=301,L]

# php -- BEGIN cPanel-generated handler, do not edit
# NOTE this account's php is controlled via FPM and the vhost, this is a place holder.
# Do not edit. This next line is to support the cPanel php wrapper (php_cli).
# AddType application/x-httpd-ea-php71 .php .phtml
# php -- END cPanel-generated handler, do not edit

I thought the first RewriteCond would stop all the other redirects, if it found the string .well-known. I even added the test at the very end there, where it redirects to port 2083, just in case. I did this recently though, because I thought maybe it was still redirecting to port 2083, even for the .well-known stuff.

Now that I’m getting the 404 on just example.com, I’m not sure how to proceed. I don’t know what happened. I can still access that test file just fine, using curl and Chrome. I don’t see why when Let’s Encrypt is ran, it receives a 404.


#14

Can you access a similar “test” file for the whm, cpanel, mail subdomains though ?


#15

Good call. Sorry for the delay, our baby’s sick and I had some contractors in the house looking at upgrading the electrical for us.

I just tried running the test for whm.example.com, using curl. This is the response I got:

HTTP/1.1 301 Moved Permanently
Date: Tue, 24 Jan 2017 20:45:04 GMT
Server: Apache
Location: https://www.example.com:2087/.well-known/acme-challenge/test
Content-Length: 267
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.example.com:2087/.well-known/acme-challenge/test">here</a>.</p>
</body></html>

So I need to fix that. I will go through and test all the subdomains until curl returns test. You said Let’s Encrypt will follow the redirects and I can leave it redirecting to the secure version of my site, right? I will work on writing better redirects for the subdomains, so the port number doesn’t get added when the works .well-known/acme-challenge is in the URL. If this doesn’t fix the problem, I’ll post back here.


#16

The 301 redirect is OK.

If you then do a curl -i https://www.example.com:2087/.well-known/acme-challenge/test

what do you get ?


#17

When I run the curl command you tell me to run, I get what appears to be the login page for whm.

I have modified all the various .htaccess files and have created test files under the .well-known/acme-challenge subdirectories for each subdomain.

I test every single of them, including the main domain, by running:
curl -i http://domain.com/.well-known/acme-challenge/test
curl -i https://domain.com/.well-known/acme-challenge/test
curl -i http://www.domain.com/.well-known/acme-challenge/test
curl -i https://www.domain.com/.well-known/acme-challenge/test
curl -i http://subdomain.domain.com/.well-known/acme-challenge/test
curl -i https://subdomain.domain.com/.well-known/acme-challenge/test

Each test file has the words subdomain-test, where subdomain is the name of the subdomain (ie, whm-test, cpanel-test, domain-test, etc).

Now, when I run the curl commands, every single one properly displays the test file. However, in Apache’s access_log, I see

66.133.109.36 - - [24/Jan/2017:16:36:25 -0500] "GET /.well-known/acme-challenge/j2pkZo7rZMdRE6-hACSIk3doIlzKPZEnTzdYoa8Rmv0 HTTP/1.1" 404 10197 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Despite the fact that I can use curl to go to http://mydomain.com/.well-known/acme-challenge/test and https://mydomain.com/.well-known/acme-challenge/test

I’ve tried removing the acme-challenge directory, thinking maybe let’s encrypt might want to create the directory itself, but that didn’t remove the 404. Any ideas?

**EDIT - I forgot to try franklin.domain.com/.well-known/acme-challenge/test. So, I tried it in curl. The http version returns the test file, but the https version returns a 404. cPanel stores the default document root in /var/www/html. /usr/local/apache/htdocs is a symbolic link to this directory. There are no .htaccess files in the /var/www/html directory.

I see there’s a default entry for unbound IPs in the Apache config (put there by cPanel). It says something like VirtualHost::80* or however it goes and sets the DocumentRoot to /var/www/html. I tried adding one for port 443 but that didn’t work. I don’t think this is the problem though, because Let’s Encrypt should be able to work over port 80 there, right? Even though port 443 takes you to a different DocumentRoot, Let’s Encrypt will try accessing port 80, and if it can, it’ll successfully complete the challenge, right?


#18

I’m running a bit out of time on this, so I’ve decided to post the command I run with some more info, minus my real email address. From running verbose, I can see jetbbs.com goes through, but www.jetbbs.com doesn’t. The Apache access_log shows a 404 is being returned, however, it shows it can access the challenge file for jetbbs.com, which shares the same DocumentRoot. I don’t understand why jetbbs.com is going through but www.jetbbs.com isn’t. I have modsec installed, I have ConfigServer Firewall installed, I’ve checked the logs to make sure it wasn’t blocking the let’s encrypt IP address, it’s not. I’ve searched the iptables firewall and don’t see it. I’m running out of ideas. At first, it was accepting the challenge, now, all of a sudden, it returns a 404. I can setup a test file in the .well-known/acme-challenge directory and access it from www.jetbbs.com and jetbbs.com.

/usr/bin/letsencrypt certonly --keep-until-expiring --non-interactive --staple-ocsp --must-staple --hsts --redirect --rsa-key-size 2048 --uir --webroot -w /home/sporkschivago/public_html -d www.jetbbs.com -d jetbbs.com -w /home/sporkschivago/public_html/cpanel -d cpanel.jetbbs.com -w /home/sporkschivago/public_html/cpcalendars -d cpcalendars.jetbbs.com -w /home/sporkschivago/public_html/cpcontacts -d cpcontacts.jetbbs.com -w /home/sporkschivago/public_html/webdisk -d webdisk.jetbbs.com -w /home/sporkschivago/public_html/webmail -d webmail.jetbbs.com -w /home/sporkschivago/public_html/whm -d whm.jetbbs.com -w /usr/local/apache/htdocs -d franklin.jetbbs.com --email realemail@someplace.com --agree-tos --verbose

In the log, I see where it fails for www.jetbbs.com. Sifting through it, I can see stuff like:
"type": "urn:acme:error:unauthorized",\n "detail": "Invalid response from http://www.jetbbs.com/.well-known/acme-challenge/GY_cDmbC6ejX2TejOiZQyJFtS7Xb-srd0f5sQ8FRsvQ...

And I can see where the 404 page is starting to be returned, but I just don’t understand why. I have an A record for www.jetbbs.com, I have an A record for jetbbs.com. I can access the test file from Chrome for both www.jetbbs.com and just jetbbs.com, I don’t get why this is failing all of a sudden.

I see some references to nginx for some reason. Not really sure why though, because I’m running Apache. Could that be causing some issues?

**EDIT: I’ll add something that might be important but might not be. Last time I renewed, I was playing with new settings. I set the RSA key size to 4096. But I didn’t really know what I was doing and have decided to switch it back to the 2048. Could that be messing things up? I noticed a /etc/letsencrypt/renewal/www.jetbbs.com.conf file that was created this morning, at midnight, and it shows: rsa_key_size = 4096 Even though I’m specifically telling letsencrypt I want to use a key size of 2048.


#19

The key size shouldn’t be causing any issue - no ( it would give a different error if it was).

By the look of things the configuration for www.jetbbs.com is different for http and https

for http …

curl http://www.jetbbs.com/.well-known/acme-challenge/test
franklin-test

for https …

curl https://www.jetbbs.com/.well-known/acme-challenge/test
www.jetbbs.com-test

So I get a different response when looking at the “same test file” between http and https … I’m guessing they are pointing to different web roots ?


#20

Hold on, something’s wrong here. How come on the server and on my home PC, when I go to http://www.jetbbs.com/.well-known/acme-challenge/test I get a different response?

Here’s my output:

curl http://www.jetbbs.com/.well-known/acme-challenge/test
  < !DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  < html>< head>
  < title>301 Moved Permanently< /title>
  < /head>< body>
  < h1>Moved Permanently< /h1>
  < p>The document has moved < a href="https://www.jetbbs.com/.well-known/acme-challenge/test">here< /a>.< /p>
  < /body>< /html>

curl https://www.jetbbs.com/.well-known/acme-challenge/test
  www.jetbbs.com-test

In Apache’s httpd.conf, I see:
##################################################
##################################################
#
# Define default vhosts for shared IPs
#
##################################################
##################################################

<VirtualHost 45.33.78.219:80>
    ServerName 45.33.78.219
    DocumentRoot /var/www/html
    ServerAdmin myrealemail@someplace.com

    <IfModule suphp_module>
        suPHP_UserGroup nobody nobody
    </IfModule>

</VirtualHost>

##################################################
##################################################
#
# Define default vhosts for unbound IPs
#
##################################################
##################################################

<VirtualHost *>
    ServerName franklin.jetbbs.com
    DocumentRoot /var/www/html
    ServerAdmin myrealemail@someplace.com

    <IfModule suphp_module>
        suPHP_UserGroup nobody nobody
    </IfModule>

</VirtualHost>
...
<VirtualHost 45.33.78.219:80 [2600:3c03:0000:0000:f03c:91ff:fee0:11b4]:80>
  ServerName jetbbs.com
#  ServerAlias www.jetbbs.com
  DocumentRoot /home/sporkschivago/public_html
...
<VirtualHost 45.33.78.219:443 [2600:3c03:0000:0000:f03c:91ff:fee0:11b4]:443>
  ServerName jetbbs.com
#  ServerAlias www.jetbbs.com
  DocumentRoot /home/sporkschivago/public_html
...

There’s no ServerName www.jetbbs.com. Could this be causing the issues? I don’t see why when I run curl, I get the 301, but when you run curl, you don’t get the 301 but are instead accessing the virtual host for unbound IPs (I think that’s the one you’re accessing, right?) I had to comment out the ServerAlias in one of the cPanel templates. That’s the only way to disable them. I really only wanted it disabled for the subdomains, because we shouldn’t have www.subdomain.domain.com and www.hostname.domain.com. But the way cPanel does it, in the template, if I comment it out for one, I comment it out for all. I had setup an A record and when I go to www.jetbbs.com or jetbbs.com, I see the right page, on my PC. Obviously, something’s wrong here. I’ll try clearing my cache. I even tried using incognito mode, but I get the same results.