Unable to generate cert (failed to authenticate some domains)

Help
1

I am unable to generate certs. I believe it's because my web server is already using https (with a cert from a vendor) and has http automatically being redirected to https. It looks like the domain authentication happens over http?

If I turn off the current http to https redirection, can I turn it back on after? Will I have to turn off the redirection for certificate renewals?

My domain is: acme-journal.org

I ran this command: certbot certonly --apache

It produced this output:

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: acme-journal.org
  Type:   connection
  Detail: Fetching http://acme-journal.org/.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY: Connection reset by peer

  Domain: www.acme-journal.org
  Type:   connection
  Detail: Fetching http://www.acme-journal.org/.well-known/acme-challenge/xRjO0-4QSrHI8B8W8SczsR_UDem1yDwrHjrxEuQfcmY: Connection reset by peer

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Some challenges have failed.

My web server is (include version):
Apache/2.4.6 (CentOS)

The operating system my web server runs on is (include version):
CentOS 7.9.2009 (Core)

My hosting provider, if applicable, is:
N/A

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
1.25.0

2

Your redirects don't work too well:

(note how one slash is missing.)

# curl -IL  http://acme-journal.org/.well-known/acme-challenge
HTTP/1.1 301 Moved Permanently
Date: Fri, 25 Mar 2022 21:37:06 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Location: https://acme-journal.org.well-known/acme-challenge
Content-Type: text/html; charset=iso-8859-1

curl: (6) Could not resolve host: acme-journal.org.well-known

And this doesn't make too much sense:

(firewall?)

# curl -IL  http://acme-journal.org/.well-known/acme-challenge/11
curl: (56) Recv failure: Connection reset by peer
1 Like
3

I'm afraid there is no HTTP to HTTPS redirect in place indeed.

You should first fix your webserver. Let's Encrypt follows HTTP to HTTPS redirects, so that wouldn't be an issue. But with a broken redirect, well, that won't work.

2 Likes
4

I've fixed the redirect (I think/hope).

It's an entry in the <VirtualHost *:80> block:
Redirect permanent / https://acme-journal.org/

I added a slash on the end of https://acme-journal.org/, but the certbot request still fails. Is this the incorrect way to redirect http to https? Is there another problem?

5

Yes, there is. But what? I don't know exactly. See the following examples:

osiris@desktop ~ $ curl -Lv http://acme-journal.org/.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY
*   Trying 142.207.145.31:80...
* Connected to acme-journal.org (142.207.145.31) port 80 (#0)
> GET /.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY HTTP/1.1
> Host: acme-journal.org
> User-Agent: curl/7.72.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
osiris@desktop ~ $ telnet acme-journal.org 80
Trying 142.207.145.31...
Connected to acme-journal.org.
Escape character is '^]'.
GET / HTTP/1.1 
Host: acme-journal.org

HTTP/1.1 301 Moved Permanently
Date: Fri, 25 Mar 2022 21:51:50 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Location: https://acme-journal.org/
Content-Length: 233
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://acme-journal.org/">here</a>.</p>
</body></html>
^C
Connection closed by foreign host.
osiris@desktop ~ $ curl -Lv http://acme-journal.org/.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY
*   Trying 142.207.145.31:80...
* Connected to acme-journal.org (142.207.145.31) port 80 (#0)
> GET /.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY HTTP/1.1
> Host: acme-journal.org
> User-Agent: curl/7.72.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
osiris@desktop ~ $

So everything works when I manually use telnet to do a very low-level HTTP request on port 80, but when curl tries to do a HTTP request, with just a liiiiiiitle bit extra headers, it fails? WEIRD! And it's just not curl, I also get the connection reset by peer error in my Chrome browser.

Please check your webserver logs for errors.

Waaaaaaaait a minute: everything works, including the redirect, UNLESS the requested path is /.well-known/acme-challenge/.. http://acme-journal.org/.well-known/acme-challenge is just fine, but when you add the slash (i.e.: http://acme-journal.org/.well-known/acme-challenge/), the result is a connection reset by peer.

Do you have a specific directive in your Apache in place for that path?

3 Likes
6

Do you have a specific directive in your Apache in place for that path?

Could it be this entry in the <VirtualHost *:443> block?

RewriteRule "^/$" "/index.php/acme" [R]

I don't touch Apache very often, and this server was set up by someone else no longer employed here. Doing my best to learn/relearn this stuff.

Edit: If you meant a directive specific to .well-known/acme-challenge , then no, there's nothing in my Apache for that path.

Edit2: There are a few instances of
"GET /.well-known/acme-challenge HTTP/1.1" 404 224
in the apache access log. Nothing in the error log, as far as I can tell.

Edit3: This is what I see with a curl request:

curl -Lv http://acme-journal.org/.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY
* About to connect() to acme-journal.org port 80 (#0)
*   Trying 142.207.145.31...
* Connected to acme-journal.org (142.207.145.31) port 80 (#0)
> GET /.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY HTTP/1.1
> User-Agent: curl/7.29.0
> Host: acme-journal.org
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Fri, 25 Mar 2022 22:42:49 GMT
< Server: Apache
< X-Frame-Options: SAMEORIGIN
< Location: https://acme-journal.org/.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY
< Content-Length: 303
< Content-Type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
* Connection #0 to host acme-journal.org left intact
* Issue another request to this URL: 'https://acme-journal.org/.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY'
* Found bundle for host acme-journal.org: 0x6a9030
* About to connect() to acme-journal.org port 443 (#1)
*   Trying 142.207.145.31...
* Connected to acme-journal.org (142.207.145.31) port 443 (#1)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=acme-journal.org,OU=Domain Control Validated
*       start date: Mar 24 21:43:32 2020 GMT
*       expire date: Apr 19 16:38:04 2022 GMT
*       common name: acme-journal.org
*       issuer: CN=Go Daddy Secure Certificate Authority - G2,OU=http://certs.godaddy.com/repository/,O="GoDaddy.com, Inc.",L=Scottsdale,ST=Arizona,C=US
> GET /.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY HTTP/1.1
> User-Agent: curl/7.29.0
> Host: acme-journal.org
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Fri, 25 Mar 2022 22:42:49 GMT
< Server: Apache
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
< Content-Length: 268
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /.well-known/acme-challenge/gce9fXVnLwmiAp3U2GTUSf5B8rTAKqDm9DrwJzgR9GY was not found on this server.</p>
</body></html>
* Connection #1 to host acme-journal.org left intact
7

Can you show the complete HTTPS vhost config?
[Or modify the HTTP vhost config to accept the challenge requests there]

1 Like
8

Here's the complete HTTPS vhost config:

<VirtualHost *:443>
        ServerAdmin lib-systems@unbc.ca
        DocumentRoot    /var/www/html/acme
        ServerName acme-journal.org
        ServerAlias www.acme-journal.org
        SSLEngine On
        SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
        SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
        SSLHonorCipherOrder on
        SSLCompression off

        # HSTS (mod_headers is required) (15768000 seconds = 6 months)
        Header always set Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"

        DirectoryIndex index.php

        SSLCertificateFile      [redacted]
        SSLCertificateKeyFile   [redacted]
        SSLCertificateChainFile [redacted]

        RewriteEngine on

        #Attempting to block referers that are causing minor DoS
        #RewriteLog "/var/log/httpd/rewrite_testing.log"
        #RewriteLogLevel 3
        RewriteCond %{HTTP_REFERER} cbdoilww\.com [NC,OR]
        RewriteCond %{HTTP_REFERER} hempcbdoilww\.com [NC]
        RewriteRule "^" "-" [F]

        RewriteRule "^/$" "/index.php/acme" [R]

        RewriteRule "/index.php/cpsr" "https://ojs.unbc.ca/index.php/cpsr" [R,L]
        RewriteRule "/index.php/design" "https://ojs.unbc.ca/index.php/design" [R,L]
        RewriteRule "/index.php/joe" "https://ojs.unbc.ca/index.php/joe" [R,L]

        ErrorLog logs/acme-ojs-error_log
        CustomLog logs/acme-ojs-access_log common

        # MAINTENANCE MODE ENABLE. Uncomment next line
        #RewriteRule ^.*$ /maint/index.html [L]
</VirtualHost>
9

I would add:
RewriteCond %{REQUEST_URI} !^/.well-known/acme-challenge [NC]

above:

1 Like
10

I added that RewriteCond, it's still failing with the same error.

12

Try adding a location statement in the HTTP block to not forward the challenge requests to HTTPS.

something like:

  #skip challenge requests and
  <LocationMatch "^/(?!\.well-known)">
    #send all other requests to HTTPS
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1
  </LocationMatch>

NOTE: You will need to set a DocumentRoot.

1 Like
13

I tried adding the suggested LocationMatch block

certbot still fails with the same error.

I tested with curl, and it does appear the redirection to https is no longer occurring when requesting one of the /.well-known/acme-challenge/ URLs.

14

This redirection seems scrambled/btoken:

curl -Ii acme-journal.org
HTTP/1.1 302 Found
Date: Mon, 28 Mar 2022 21:44:07 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Location: https://acme-journal.org/var/www/html/
Content-Type: text/html; charset=iso-8859-1

This reply seems incomplete/broken:

curl -Ii acme-journal.org/.well-known/acme-challenge/Test_File-1234
curl: (52) Empty reply from server

Recall:

Please show us the HTTP vhost config section.

1 Like
15

I believe I have identified the culprit, and it is our organization's perimeter firewall. It is blocking "acme-protocol" traffic. I will attempt to get that rectified, and then report back whether that fixes the issue or not.

Edit: Perimeter firewall was updated to allow the "acme-protocol" traffic, and certificates were successfully retrieved. Thanks for the help with the vhost config changes!

1 Like
16

Your redirect is still broken.

And you're sending the certificate without the chain. Your server needs the fullchain.pem file instead of cert.pem.

1 Like
17

Ah, it looks like I had to add both SSLCertificateFile and SSLCertificateChainFile in the vhost config, both pointing to the fullchain.pem. Apparently this is fixed in a newer version of Apache, and SSLCertificateFile will pick up the full chain if it is in the cert file?

I switched back to the redirect method I had in place previously, which was using "Redirect permanent", but now I have it inside the LocationMatch block suggested previously. So that CertBot challenges shouldn't get redirected. Is it better now?

18

If you add both directives, SSLCertificateFile goes to cert.pem and SSLCertificateChainFile goes to chain.pem.

Using fullchain.pem on SSLCertificateFile replaces both.

2 Likes
19

CertBot only generated fullchain.pem, no cert.pem or chain.pem. I tried using fullchain.pem on SSLCertificateFile originally, and it was missing the chain (as you pointed out previously). It appears Apache 2.4.8 is the one that lets you do the fullchain.pem with SSLCertificateFile, and I'm on Apache 2.4.6.

20

Check if you find them in

  • /etc/letsencrypt/live/CERTNAME/cert.pem and
  • /etc/letsencrypt/live/CERTNAME/chain.pem

:wink:

(yes, some versions of apache behave like that. but they need different files.)

1 Like
21

Ah, finally figured out how to quote.

Anyway, I checked that folder yesterday and I swear it only had the fullchain.pem and privkey.pem. But you're right, there's also cert.pem and chain.pem in there.