Http-01 challenge failed due to an unauthorized response when renewing the cert

Help
1

Hi, I am having the follwoing issue would appreciate some help. the strange thing is it is been working for 2.5 years and suddenly it is not working.

My domain is: drposture.app

I ran this command: sudo certbot --dry-run --apache certonly -d drposture.app

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for drposture.app
Waiting for verification...
Challenge failed for domain drposture.app
http-01 challenge for drposture.app
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: drposture.app
   Type:   unauthorized
   Detail: 2a01:4f8:1c17:df57::1: Invalid response from
   http://drposture.app/.well-known/acme-challenge/lUgPLeigw4B-N7oIeaPKhgq5sPqawnYmy9VguMidvPE:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version): Apache/2.4.41 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 20.04.6 LTS

My hosting provider, if applicable, is: hetzner

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.40.0

apachectl -s Info:

apachectl -S                                       
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:6)
         port 443 namevhost api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:6)
         port 443 namevhost bat.drposture.app (/etc/apache2/sites-enabled/bat.drposture.app-le-ssl.conf:2)
         port 443 namevhost batapi.drposture.app (/etc/apache2/sites-enabled/batapi.drposture.app-le-ssl.conf:2)
         port 443 namevhost cms.drposture.app (/etc/apache2/sites-enabled/cms.drposture.app-le-ssl.conf:2)
         port 443 namevhost qa.cms.drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app-le-ssl.conf:2)
*:*                    is a NameVirtualHost
         default server drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:2)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:2)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/cms.drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app.conf:1)
*:80                   is a NameVirtualHost
         default server api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:5)
         port 80 namevhost api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:5)
         port 80 namevhost drposture.app (/etc/apache2/sites-enabled/bat.drposture.app.conf:1)
         port 80 namevhost batapi.drposture.app (/etc/apache2/sites-enabled/batapi.drposture.app.conf:1)
         port 80 namevhost cms.drposture.app (/etc/apache2/sites-enabled/cms.drposture.app.conf:8)
         port 80 namevhost drposture.app (/etc/apache2/sites-enabled/drposture.app.conf:5)
                 alias www.drposture.app
         port 80 namevhost qa.cms.drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app.conf:5)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
Mutex mpm-accept: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: ENABLE_USR_LIB_CGI_BIN
User: name="www-data" id=33
Group: name="www-data" id=33

My apache conf:

<VirtualHost *:*>
    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
</VirtualHost>

<VirtualHost *:80>

    <Directory "/var/www/drposture.app/.well-known/acme-challenge/">
        Options None
        AllowOverride None
        Require all granted
    </Directory>
    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:5500/
    ProxyPassReverse / http://127.0.0.1:5500/
   ServerName drposture.app
    ServerAlias www.drposture.app
    ErrorLog ${APACHE_LOG_DIR}drposture_app_conf-error.log
    CustomLog ${APACHE_LOG_DIR}drposture_app_conf-access.log common
RewriteEngine on
RewriteCond %{SERVER_NAME} =drposture.app [OR]
RewriteCond %{SERVER_NAME} =www.drposture.app
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

drposture.app-le-ssl.conf

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:5500/
    ProxyPassReverse / http://127.0.0.1:5500/
    ServerName drposture.app
    ServerAlias www.drposture.app
    ErrorLog ${APACHE_LOG_DIR}drposture_app_conf-error.log
    CustomLog ${APACHE_LOG_DIR}drposture_app_conf-access.log common
RewriteEngine on
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{SERVER_NAME} =drposture.app [OR]
# RewriteCond %{SERVER_NAME} =www.drposture.app
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/www.drposture.app-0001/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.drposture.app-0001/privkey.pem
</VirtualHost>
</IfModule>
2

That's weird.

Why are there *:80 and *:443 virtualhosts, which are normal, and a virtualhost for *:*? That's going to lead to problems.

3

As is this:

Hi @d-shiri Welcome and thanks for providing the info up front!
I would expect to see something more like:

<Directory "/var/www/drposture.app>

The validation servers know where to find .well-known/acme-challenge/

Your domain checks out for http challenge:

And DNS challenge:

First glance for me and I have to say I am not "proxy oriented" so I'll sit back and learn from other volunteers here.

4

Thanks for the reply, I changed it. but still the same error.

5

thanks for the reply. even when I delete that part the issue persists.

6

You probably also want to remove the proxy pass stuff from your *:80 vhosts.. That one should only redirect to HTTPS. But currently I don't see a HTTP to HTTPS redirect, not even for your main domain (thus not limited to the challenge).

Even better is to exclude the path /.well-known/acme-challenge/ from the redirect.

7

Absolutely! And I would also recommend excluding it from the proxy:

# challenge requests should not be proxied
    <Location "/.well-known/acme-challenge/">
        ProxyPass !
    </Location>

I dont know how to use a proxy, but I do know how NOT to use one

8

If you don't redirect that path to HTTPS and don't have the proxy stuff in your HTTP vhost, this isn't necessary :slight_smile:

9

Ok thanks, @Rip & @Osiris
I will make the changes and report the result here. I need to do it in an hour because I hit the hourly limit. :smiley:

10

You should be using the staging server which has more generous limits while testing... then get your cert from production. :grinning:

11

using the staging server I can test again. thanks for the tip :slight_smile:
I commented out all the unusual stuff and ended up with this:

#<VirtualHost *:*>
#    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
#</VirtualHost>
#
<VirtualHost *:80>

    <Directory "/var/www/drposture.app">
        Options None
        AllowOverride None
        Require all granted
    </Directory>
    <Location "/.well-known/acme-challenge/">
        ProxyPass !
    </Location>
#    ProxyPreserveHost On
#    ProxyPass / http://127.0.0.1:5500/
#    ProxyPassReverse / http://127.0.0.1:5500/
     ServerName drposture.app
     DocumentRoot /var/www/drposture.app
#    ServerAlias www.drposture.app
#    ErrorLog ${APACHE_LOG_DIR}drposture_app_conf-error.log
#    CustomLog ${APACHE_LOG_DIR}drposture_app_conf-access.log common
#RewriteEngine on
#RewriteCond %{SERVER_NAME} =drposture.app [OR]
#RewriteCond %{SERVER_NAME} =www.drposture.app
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

but still not working :frowning:

12

Since you commented out the proxy configuration as @Osiris suggested, you can completely remove this block.

13

still did not work. maybe something more fundamental is wrong

14

Well, your site is still not redirecting HTTP to HTTPS in general, so I concur that something more fundamental is not entirely correct.

Could you post the output from apachectl -t -D DUMP_VHOSTS again?

15

OK so maybe you should place a "testfile" in the place where the validation servers will be looking and check to see if it is publicly available:

echo "test" > /var/www/drposture.app/.well-known/acme-challenge/testfile
curl http://drposture.app/.well-known/acme-challenge/testfile

This might shed some light on a possible issue - might not. Worth a try to be sure..

16 
[drposture /etc/apache2/sites-available]# apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:6)
         port 443 namevhost api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:6)
         port 443 namevhost bat.drposture.app (/etc/apache2/sites-enabled/bat.drposture.app-le-ssl.conf:2)
         port 443 namevhost batapi.drposture.app (/etc/apache2/sites-enabled/batapi.drposture.app-le-ssl.conf:2)
         port 443 namevhost cms.drposture.app (/etc/apache2/sites-enabled/cms.drposture.app-le-ssl.conf:2)
         port 443 namevhost qa.cms.drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app-le-ssl.conf:2)
*:*                    is a NameVirtualHost
         default server drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:2)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:2)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/cms.drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app.conf:1)
*:80                   is a NameVirtualHost
         default server api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:5)
         port 80 namevhost api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:5)
         port 80 namevhost drposture.app (/etc/apache2/sites-enabled/bat.drposture.app.conf:1)
         port 80 namevhost batapi.drposture.app (/etc/apache2/sites-enabled/batapi.drposture.app.conf:1)
         port 80 namevhost cms.drposture.app (/etc/apache2/sites-enabled/cms.drposture.app.conf:8)
         port 80 namevhost drposture.app (/etc/apache2/sites-enabled/drposture.app.conf:5)
         port 80 namevhost qa.cms.drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app.conf:5)
17 
curl -I http://drposture.app/.well-known/acme-challenge/test-file

HTTP/1.1 404 Not Found
Date: Thu, 23 May 2024 19:43:11 GMT
Server: Apache/2.4.41 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1
18

This is still there.

19

This suggests there is something wrong with the paths.

ls /var/www/drposture.app/.well-known/acme-challenge

Should show the file. If not there will be some digging to do.
And BTW you have a bunch of certificates for a bunch of subdomains, and many recent. So I am wondering what is the difference.. or what has been possibly changed.

And checking the certificate transparency logs:

Shows certificates issued as recently as a couple days ago. (yesterday)

How is your configuration different for drposture.app than www.drposture.app or batapi.drposture.app
They should be combined into a single cert or a wildcard certificate.
What gives here.. ??

Please do this and lets fix the issue.

20

Supplemental information.

This is the presently being served certificate crt.sh | 13052872281 by

This one is "It's all good. We have not detected any issues."

These 2 "Doesn't match Common Name or/and SANs"

Edit: this is what I see for each of those domain names using curl -k -i

$ curl -k -i https://api.drposture.app/
HTTP/1.1 404 Not Found
Date: Fri, 24 May 2024 02:24:45 GMT
Server: Kestrel
Content-Length: 0

$ curl -k -i https://www.drposture.app/
HTTP/1.1 404 Not Found
Date: Fri, 24 May 2024 02:24:47 GMT
Server: Kestrel
Content-Length: 0

$ curl -k -i https://drposture.app/
HTTP/1.1 404 Not Found
Date: Fri, 24 May 2024 02:24:50 GMT
Server: Kestrel
Content-Length: 0

@d-shiri you stated

Yet I see Server: Kestrel for all those domain names.

Edit again:
All 3 domain names map to the same IPv4 Address

$ nslookup drposture.app hydrogen.ns.hetzner.com.
Server:         hydrogen.ns.hetzner.com.
Address:        213.133.100.98#53

Name:   drposture.app
Address: 49.12.242.201

$ nslookup www.drposture.app hydrogen.ns.hetzner.com.
Server:         hydrogen.ns.hetzner.com.
Address:        213.133.100.98#53

Name:   www.drposture.app
Address: 49.12.242.201

$ nslookup api.drposture.app hydrogen.ns.hetzner.com.
Server:         hydrogen.ns.hetzner.com.
Address:        213.133.100.98#53

Name:   api.drposture.app
Address: 49.12.242.201

Yet for HTTP not HTTPS I see Apache, but not all 3 domain names respond the same (which may or may not be an issue; I suspect at least www would be an issue.)

$ curl -k -i http://api.drposture.app/.well-known/acme-challenge/sometestfile
HTTP/1.1 301 Moved Permanently
Date: Fri, 24 May 2024 02:53:50 GMT
Server: Apache/2.4.41 (Ubuntu)
Location: https://api.drposture.app/.well-known/acme-challenge/sometestfile
Content-Length: 356
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://api.drposture.app/.well-known/acme-challenge/sometestfile">here</a>.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at api.drposture.app Port 80</address>
</body></html>

$ curl -k -i http://www.drposture.app/.well-known/acme-challenge/sometestfile
HTTP/1.1 301 Moved Permanently
Date: Fri, 24 May 2024 02:53:55 GMT
Server: Apache/2.4.41 (Ubuntu)
Location: https://www.drposture.app/.well-known/acme-challenge/sometestfile
Content-Length: 356
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.drposture.app/.well-known/acme-challenge/sometestfile">here</a>.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at www.drposture.app Port 80</address>
</body></html>

$ curl -k -i http://drposture.app/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Date: Fri, 24 May 2024 02:53:59 GMT
Server: Apache/2.4.41 (Ubuntu)
Content-Length: 275
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at drposture.app Port 80</address>
</body></html>