Renewal fails with login

My domain is: theveninskeep.com

I ran this command: certbot renew --dry-run

It produced this output:

Invalid Response from https://theveninskeep.com/.well-known/acme-challenge/QgIpC-uyq5n77hZH11YgwUKYMeSjn59F2SAV2MRntAs: 400

My web server is (include version):

Apache/2.4.41 (Ubuntu)

The operating system my web server runs on is (include version):

Ubuntu 20.04.4 LTS

My hosting provider, if applicable, is:

I own the hardware

I can login to a root shell on my machine (yes or no, or I don't know):

yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

1.29.0

Additional Detail: I'm not much of a web guy. I installed the cert originally and then enabled a login portal. I'm assuming that certbot is failing because without a login, it'll return a 400 error. How do I leave the login structure alone and still update the cert?

You can create separate authentication rules for the /.well-known/acme-challenge directory to allow unrestricted access to that path without impacting your existing login.

I can't share specific examples at the moment, but I'll check back when I can and see how you are faring.

3 Likes

Agree with what linkp said but your http 400 error is because you have your HTTP server (for port 80) setup as an HTTPS server (ssl enabled). See the Let's Debug test site. My own tests don't see an incorrect redirect so likely a wrongly configured VirtualHost

If you need help resolving that let's start by seeing what this shows:

apachectl -t -D DUMP_VHOSTS
6 Likes

Another option is to proxy the traffic to /.well-known/acme-challenge onto a higher port, such as 8001, and run Certbot in standalone mode with the --http-01-port flag set to that port. The advantage of this method is that Certbot does not need to modify any files in the site's directory structure.

7 Likes

VirtualHost configuration:
*:443 TheveninsKeep.com (/etc/apache2/sites-enabled/TheveninsKeep.com-le-ssl.conf:2)
*80 TheveninsKeep.com (/etc/apache2/sites-enabled/TheveninsKeep.com.conf:1)

I think this is configured the way I want. All http requests should resolve as https requests?

from the *80 conf:
<Directory "/var/www/TheveninsKeep.com/html">
AuthType Basic
AuthName "Restricted Content"
AuthUserFile //password file here :slight_smile: //
Require valid-user
</Directory>
RewriteEngine on
RewriteCond %{SERVER_NAME} =TheveninsKeep.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

I assume it'd be something like

    <Directory "/var/www/TheveninsKeep.com/html/.well-known/acme-challenge/(That string)">
            AuthType None
            Require all granted
    <\/Directory>

yeah?

Would you show that entire file please

8 Likes
<VirtualHost *:80>
ServerAdmin [REDACTED]@TheveninsKeep.com
ServerName TheveninsKeep.com
ServerAlias TheveninsKeep.com
DocumentRoot /var/www/TheveninsKeep.com/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
        <Directory "/var/www/TheveninsKeep.com/html">
                AuthType Basic
                AuthName "Restricted Content"
                AuthUserFile <PASSWORD FILE>
                Require valid-user
        </Directory>
RewriteEngine on
RewriteCond %{SERVER_NAME} =TheveninsKeep.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
1 Like

The ServerAlias isn't adding anything.

ServerName  TheveninsKeep.com
ServerAlias TheveninsKeep.com

Requiring authentication will create a problem for HTTP authentication requests:

[quote="jhrogers, post:8, topic:180845"]

DocumentRoot /var/www/TheveninsKeep.com/html
        <Directory "/var/www/TheveninsKeep.com/html">
                AuthType Basic
                AuthName "Restricted Content"
                AuthUserFile <PASSWORD FILE>
                Require valid-user
        </Directory>

As shown by:

wget TheveninsKeep.com
--2022-07-10 18:24:39--  http://theveninskeep.com/
Resolving theveninskeep.com (theveninskeep.com)... 207.244.205.45
Connecting to theveninskeep.com (theveninskeep.com)|207.244.205.45|:80... connected.
HTTP request sent, awaiting response... 400 Bad Request
2022-07-10 18:24:40 ERROR 400: Bad Request.
7 Likes

I am not a web guy. My understanding from a security perspective is that I would rather just flat fail/error HTTP requests with the message of "Hey, use https, duh"

So i go back to how do i get certbot to talk appropriately to get the new cert?

1 Like

Well, you can do that if you want but a simple redirect is friendly and just as safe (and you have one setup already). See the Let's Encrypt topic about port 80.

I think you need to remove that security test to allow Let's Encrypt server challenge to succeed. Or, switch to a DNS challenge which is often more complex to setup.

I was thinking you had the port 80 VirtualHost setup wrongly as an SSL server because of the message below. But, maybe the message is misleading and it is your Apache auth that is blocking the challenge.

curl http://theveninskeep.com/.well-known/acme-challenge/TestForum123

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
 Instead use the HTTPS scheme to access this URL, please.<br />
</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at TheveninsKeep.com Port 80</address>
</body></html>
10 Likes

So I could leave the rewrite rule (or update it to the redirect version) and change Auth from basic to none on port 80 and that'd more or less work?

Yes. The rewrite will cause an http redirect to your https (port 443) server. When certbot runs with the apache plug-in it will temporarily insert code to handle the http auth challenge in the port 80 server and remove that temp code when complete.

You should comment out all 6 lines related to your auth for port 80 and remove them once you satisfy yourself it is working.

EDIT: Above is correct except I just realized I don't know if you are using the apache plug-in. From the error message in your first post I think not. The only difference if you are not is we need to add a rewritecond to the port 80 rewrite to handle the acme http challenge. Can you show the contents of the renewal conf file in /etc/letsencrypt/renewal and we'll sort this out.

7 Likes
# renew_before_expiry = 30 days
version = 1.26.0
archive_dir = /etc/letsencrypt/archive/theveninskeep.com
cert = /etc/letsencrypt/live/theveninskeep.com/cert.pem
privkey = /etc/letsencrypt/live/theveninskeep.com/privkey.pem
chain = /etc/letsencrypt/live/theveninskeep.com/chain.pem
fullchain = /etc/letsencrypt/live/theveninskeep.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = b5281dfabfaab2d8e0b0c2e57c9a496e
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory
key_type = rsa

OK. Nevermind. You are using apache plug-in so ignore my EDIT

Once you remove the auth on port 80 and restart apache try running this again:

certbot renew --dry-run

Warning: there are a number of reports this morning of problems with the staging system. Staging is used for --dry-run tests. So, it might not work for that reason.

In that case, just leave off --dry-run and see. You are due for a renewal so that's fair to try. Let us know what happens.

7 Likes

Updated the .conf to be

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
RewriteEngine on
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{SERVER_NAME}%{REQUEST_URI}

And now the rewrite is working as I originally wanted (That'd been bugging me). But the dry run is still giving an error:

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: theveninskeep.com
  Type:   unauthorized
  Detail: 207.244.205.45: Invalid response from http://theveninskeep.com/.well-known/acme-challenge/2eDrBeprzhVkTQC4XT3Sb0RWagaFonZGVrRL0wlGi4k: 400

Is this because the request is getting rewritten to https? do I need to comment that out, update the cert, then put it back?

Hmm. I don't see any redirect for HTTP requests. I only get http error codes 400 Bad Request. Did you remove that auth code? Because if you did we need to look at other causes. You may have duplicate or overlapping VirtualHosts.

Inside a VirtualHost that only listens on port 80, you do not need to check for https being off. It will always be off.

If you want, make the redirect in port 80 to look like below. We do not want acme challenges redirecting. But, I still think the apache plug-in would avoid this.

RewriteEngine on
RewriteCond %{REQUEST_URI} !^/.well-known/acme-challenge [NC]
RewriteRule (.*) https://%{SERVER_NAME}%{REQUEST_URI}
7 Likes

No, we can see it fail via HTTP:

7 Likes

Change made, still not working. Is it possible that the traffic is getting caught in my firewall filters? What address would certbot be coming back from?

I see Apache 2.4.41 (Ubuntu) responding so probably not caught in firewall. I get the identical response I showed in post #11.

Note certbot begins the request. It sends message to Let's Encrypt server. The LE server sends requests back to your server to ensure you control that domain name. The curl examples we are using mimic the LE server requests.

curl -i   http://theveninskeep.com
(just showing the headers for Server value, see post #11 for data)

HTTP/1.1 400 Bad Request
Date: Tue, 12 Jul 2022 22:59:06 GMT
Server: Apache/2.4.41 (Ubuntu)
Content-Length: 445
Connection: close
Content-Type: text/html; charset=iso-8859-1

Let's start back at the beginning. What does this show:

apachectl -t -D DUMP_VHOSTS
7 Likes