Renew 'unauthorized' -- no server changes, IPv4 only

mberding · April 29, 2024, 2:11pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: theprojectland.com

I ran this command: sudo certbot renew -v

It produced this output:

Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for theprojectland.com
Performing the following challenges:
http-01 challenge for theprojectland.com
Waiting for verification...
Challenge failed for domain theprojectland.com
http-01 challenge for theprojectland.com

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: theprojectland.com
  Type:   unauthorized
  Detail: 44.229.66.22: Invalid response from http://theprojectland.com/.well-known/acme-challenge/PbB9ZMg0lG0tud9gv20gYZOAegbdhvTVgoidJpgsLHk: 404

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate theprojectland.com with error: Some challenges have failed.

My web server is (include version): Apache/2.4.58 (Amazon Linux)

The operating system my web server runs on is (include version): Amazon Linux 2023

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.8.0

This domain has been set up and functional for a year or more now. No recent server changes of any kind. Server has multiple domains, all with certbot, and this domain is the first to give me any trouble. Server is IPv4 only, there's no IPv6 DNS records. I have no clue why this cert isn't renewing and after searching through the forum, still don't have a clue.

rg305 · April 29, 2024, 4:17pm

Hi @mberding, and welcome to the LE community forum

I have one:

As with all things Apache [on this forum], let's begin at the beginning and review the output of:
sudo apachectl -t -D DUMP_VHOSTS

mberding · April 29, 2024, 10:34pm

@rg305

I love how linux keeps evolving and changing things on us.

[michaelberding@ip-10-20-0-141 ~]$ sudo apachectl -t -D DUMP_VHOSTS
[sudo] password for michaelberding:
Passing arguments to httpd using apachectl is no longer supported.
You can only start/stop/restart httpd using this script.
To pass extra arguments to httpd, see the httpd.service(8)
man page.

Figured it out. I think this is what we're looking for:

[michaelberding@ip-10-20-0-141 ~]$ sudo httpd -t -D DUMP_VHOSTS
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server ip-10-20-0-141.us-west-2.compute.internal (/etc/httpd/conf.d/ssl.conf:56)
         port 443 namevhost ip-10-20-0-141.us-west-2.compute.internal (/etc/httpd/conf.d/ssl.conf:56)
         port 443 namevhost systems.berdingconsulting.com (/etc/httpd/conf/httpd-le-ssl.conf:2)
         port 443 namevhost internetwantlist.com (/etc/httpd/conf/httpd-le-ssl.conf:17)
         port 443 namevhost apimonitor.berdingconsulting.com (/etc/httpd/conf/httpd-le-ssl.conf:31)
         port 443 namevhost supercron.me (/etc/httpd/conf/httpd-le-ssl.conf:45)
         port 443 namevhost threadmanager.berdingconsulting.com (/etc/httpd/conf/httpd-le-ssl.conf:58)
         port 443 namevhost mikasa.berdingconsulting.com (/etc/httpd/conf/httpd-le-ssl.conf:72)
         port 443 namevhost megamws.com (/etc/httpd/conf/httpd-le-ssl.conf:86)
         port 443 namevhost ca.megamws.com (/etc/httpd/conf/httpd-le-ssl.conf:100)
         port 443 namevhost theprojectland.com (/etc/httpd/conf/httpd-le-ssl.conf:114)
*:80                   is a NameVirtualHost
         default server systems.berdingconsulting.com (/etc/httpd/conf/httpd.conf:362)
         port 80 namevhost systems.berdingconsulting.com (/etc/httpd/conf/httpd.conf:362)
         port 80 namevhost internetwantlist.com (/etc/httpd/conf/httpd.conf:376)
         port 80 namevhost apimonitor.berdingconsulting.com (/etc/httpd/conf/httpd.conf:387)
         port 80 namevhost supercron.me (/etc/httpd/conf/httpd.conf:398)
         port 80 namevhost threadmanager.berdingconsulting.com (/etc/httpd/conf/httpd.conf:409)
         port 80 namevhost mikasa.berdingconsulting.com (/etc/httpd/conf/httpd.conf:420)
         port 80 namevhost megamws.com (/etc/httpd/conf/httpd.conf:431)
         port 80 namevhost ca.megamws.com (/etc/httpd/conf/httpd.conf:442)
         port 80 namevhost theprojectland.com (/etc/httpd/conf/httpd.conf:454)

rg305 · April 29, 2024, 11:38pm

I don't see anything wrong there...

Let's have a look the renewal config file:
/etc/letsencrypt/renewal/theprojectland.com.conf ?

[if not exactly that name, show: "ls -l /etc/letsencrypt/renewal/"]

mberding · April 30, 2024, 2:49pm

/etc/letsencrypt/renewal/theprojectland.com.conf contains:

# renew_before_expiry = 30 days
version = 2.8.0
archive_dir = /etc/letsencrypt/archive/theprojectland.com
cert = /etc/letsencrypt/live/theprojectland.com/cert.pem
privkey = /etc/letsencrypt/live/theprojectland.com/privkey.pem
chain = /etc/letsencrypt/live/theprojectland.com/chain.pem
fullchain = /etc/letsencrypt/live/theprojectland.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = 934f14a99ef8fc6be536b0f88de3ade1
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa

rg305 · April 30, 2024, 3:06pm

Let's have a look at the Apache config file:

mberding · April 30, 2024, 3:21pm

The entire config file is rather long, but starting a line 454 which handles this site is pasted below. It's exactly like the other virtual host sections in this file.

<VirtualHost _default_:80>
       	ServerName theprojectland.com
        DocumentRoot "/ebs/vhosts/theprojectland.com/html"
        <Directory /ebs/vhosts/theprojectland.com/html>
                AllowOverride All
        </Directory>
RewriteEngine off
RewriteCond %{SERVER_NAME} =theprojectland.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

MikeMcQ · April 30, 2024, 4:08pm

The --apache authenticator requires RewriteEngine On

It inserts temp code into your Apache config which relies on rewrite engine

mberding · April 30, 2024, 5:12pm

The --apache authenticator requires RewriteEngine On

Fascinating. I turned that off (a very long time ago) because I have ancient devices that can't use modern SSL and can still access the site via http.

I changed ReWriteEnginge to 'On', and magically certbot was able to renew the certificate. Who knew!

Now I'll just need to upgrade the ancient device that was using http to something that is getting OS updates.

Thanks for your help!

petercooperjr · April 30, 2024, 5:23pm

You can remove (or comment out) the RewriteCond/RewriteRule lines to stop automatically redirecting to https, while keeping the RewriteEngine on allowing for other rewriting rules (like those certbot wants to use).

mberding · April 30, 2024, 5:49pm

Bonus! Thanks for the tip, my ancient iPhone continues to live on displaying the site.

orangepizza · April 30, 2024, 6:54pm

not really related for this forum but you may want GitHub - atauenis/webone: HTTP 1.x proxy that makes old web browsers usable again in the Web 2.0 world.