Http-01 challenge failed due to an unauthorized response when renewing the cert

Help
1

Hi, I am having the follwoing issue would appreciate some help. the strange thing is it is been working for 2.5 years and suddenly it is not working.

My domain is: drposture.app

I ran this command: sudo certbot --dry-run --apache certonly -d drposture.app

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for drposture.app
Waiting for verification...
Challenge failed for domain drposture.app
http-01 challenge for drposture.app
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: drposture.app
   Type:   unauthorized
   Detail: 2a01:4f8:1c17:df57::1: Invalid response from
   http://drposture.app/.well-known/acme-challenge/lUgPLeigw4B-N7oIeaPKhgq5sPqawnYmy9VguMidvPE:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version): Apache/2.4.41 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 20.04.6 LTS

My hosting provider, if applicable, is: hetzner

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.40.0

apachectl -s Info:

apachectl -S                                       
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:6)
         port 443 namevhost api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:6)
         port 443 namevhost bat.drposture.app (/etc/apache2/sites-enabled/bat.drposture.app-le-ssl.conf:2)
         port 443 namevhost batapi.drposture.app (/etc/apache2/sites-enabled/batapi.drposture.app-le-ssl.conf:2)
         port 443 namevhost cms.drposture.app (/etc/apache2/sites-enabled/cms.drposture.app-le-ssl.conf:2)
         port 443 namevhost qa.cms.drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app-le-ssl.conf:2)
*:*                    is a NameVirtualHost
         default server drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:2)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:2)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/cms.drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app.conf:1)
*:80                   is a NameVirtualHost
         default server api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:5)
         port 80 namevhost api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:5)
         port 80 namevhost drposture.app (/etc/apache2/sites-enabled/bat.drposture.app.conf:1)
         port 80 namevhost batapi.drposture.app (/etc/apache2/sites-enabled/batapi.drposture.app.conf:1)
         port 80 namevhost cms.drposture.app (/etc/apache2/sites-enabled/cms.drposture.app.conf:8)
         port 80 namevhost drposture.app (/etc/apache2/sites-enabled/drposture.app.conf:5)
                 alias www.drposture.app
         port 80 namevhost qa.cms.drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app.conf:5)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
Mutex mpm-accept: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: ENABLE_USR_LIB_CGI_BIN
User: name="www-data" id=33
Group: name="www-data" id=33

My apache conf:

<VirtualHost *:*>
    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
</VirtualHost>

<VirtualHost *:80>

    <Directory "/var/www/drposture.app/.well-known/acme-challenge/">
        Options None
        AllowOverride None
        Require all granted
    </Directory>
    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:5500/
    ProxyPassReverse / http://127.0.0.1:5500/
   ServerName drposture.app
    ServerAlias www.drposture.app
    ErrorLog ${APACHE_LOG_DIR}drposture_app_conf-error.log
    CustomLog ${APACHE_LOG_DIR}drposture_app_conf-access.log common
RewriteEngine on
RewriteCond %{SERVER_NAME} =drposture.app [OR]
RewriteCond %{SERVER_NAME} =www.drposture.app
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

drposture.app-le-ssl.conf

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:5500/
    ProxyPassReverse / http://127.0.0.1:5500/
    ServerName drposture.app
    ServerAlias www.drposture.app
    ErrorLog ${APACHE_LOG_DIR}drposture_app_conf-error.log
    CustomLog ${APACHE_LOG_DIR}drposture_app_conf-access.log common
RewriteEngine on
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{SERVER_NAME} =drposture.app [OR]
# RewriteCond %{SERVER_NAME} =www.drposture.app
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/www.drposture.app-0001/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.drposture.app-0001/privkey.pem
</VirtualHost>
</IfModule>
2

That's weird.

Why are there *:80 and *:443 virtualhosts, which are normal, and a virtualhost for *:*? That's going to lead to problems.

2 Likes
3

As is this:

Hi @d-shiri Welcome and thanks for providing the info up front!
I would expect to see something more like:

<Directory "/var/www/drposture.app>

The validation servers know where to find .well-known/acme-challenge/

Your domain checks out for http challenge:

And DNS challenge:

First glance for me and I have to say I am not "proxy oriented" so I'll sit back and learn from other volunteers here.

3 Likes
4

Thanks for the reply, I changed it. but still the same error.

1 Like
5

thanks for the reply. even when I delete that part the issue persists.

1 Like
6

You probably also want to remove the proxy pass stuff from your *:80 vhosts.. That one should only redirect to HTTPS. But currently I don't see a HTTP to HTTPS redirect, not even for your main domain (thus not limited to the challenge).

Even better is to exclude the path /.well-known/acme-challenge/ from the redirect.

2 Likes
7

Absolutely! And I would also recommend excluding it from the proxy:

# challenge requests should not be proxied
    <Location "/.well-known/acme-challenge/">
        ProxyPass !
    </Location>

I dont know how to use a proxy, but I do know how NOT to use one

2 Likes
8

If you don't redirect that path to HTTPS and don't have the proxy stuff in your HTTP vhost, this isn't necessary :slight_smile:

9

Ok thanks, @Rip & @Osiris
I will make the changes and report the result here. I need to do it in an hour because I hit the hourly limit. :smiley:

1 Like
10

You should be using the staging server which has more generous limits while testing... then get your cert from production. :grinning:

4 Likes
11

using the staging server I can test again. thanks for the tip :slight_smile:
I commented out all the unusual stuff and ended up with this:

#<VirtualHost *:*>
#    RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
#</VirtualHost>
#
<VirtualHost *:80>

    <Directory "/var/www/drposture.app">
        Options None
        AllowOverride None
        Require all granted
    </Directory>
    <Location "/.well-known/acme-challenge/">
        ProxyPass !
    </Location>
#    ProxyPreserveHost On
#    ProxyPass / http://127.0.0.1:5500/
#    ProxyPassReverse / http://127.0.0.1:5500/
     ServerName drposture.app
     DocumentRoot /var/www/drposture.app
#    ServerAlias www.drposture.app
#    ErrorLog ${APACHE_LOG_DIR}drposture_app_conf-error.log
#    CustomLog ${APACHE_LOG_DIR}drposture_app_conf-access.log common
#RewriteEngine on
#RewriteCond %{SERVER_NAME} =drposture.app [OR]
#RewriteCond %{SERVER_NAME} =www.drposture.app
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

but still not working :frowning:

1 Like
12

Since you commented out the proxy configuration as @Osiris suggested, you can completely remove this block.

2 Likes
13

still did not work. maybe something more fundamental is wrong

14

Well, your site is still not redirecting HTTP to HTTPS in general, so I concur that something more fundamental is not entirely correct.

Could you post the output from apachectl -t -D DUMP_VHOSTS again?

15

OK so maybe you should place a "testfile" in the place where the validation servers will be looking and check to see if it is publicly available:

echo "test" > /var/www/drposture.app/.well-known/acme-challenge/testfile
curl http://drposture.app/.well-known/acme-challenge/testfile

This might shed some light on a possible issue - might not. Worth a try to be sure..

2 Likes
16 
[drposture /etc/apache2/sites-available]# apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:6)
         port 443 namevhost api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:6)
         port 443 namevhost bat.drposture.app (/etc/apache2/sites-enabled/bat.drposture.app-le-ssl.conf:2)
         port 443 namevhost batapi.drposture.app (/etc/apache2/sites-enabled/batapi.drposture.app-le-ssl.conf:2)
         port 443 namevhost cms.drposture.app (/etc/apache2/sites-enabled/cms.drposture.app-le-ssl.conf:2)
         port 443 namevhost qa.cms.drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app-le-ssl.conf:2)
*:*                    is a NameVirtualHost
         default server drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:2)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/api.drposture.app-le-ssl.conf:2)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/cms.drposture.app.conf:1)
         port * namevhost drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app.conf:1)
*:80                   is a NameVirtualHost
         default server api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:5)
         port 80 namevhost api.drposture.app (/etc/apache2/sites-enabled/api.drposture.app.conf:5)
         port 80 namevhost drposture.app (/etc/apache2/sites-enabled/bat.drposture.app.conf:1)
         port 80 namevhost batapi.drposture.app (/etc/apache2/sites-enabled/batapi.drposture.app.conf:1)
         port 80 namevhost cms.drposture.app (/etc/apache2/sites-enabled/cms.drposture.app.conf:8)
         port 80 namevhost drposture.app (/etc/apache2/sites-enabled/drposture.app.conf:5)
         port 80 namevhost qa.cms.drposture.app (/etc/apache2/sites-enabled/qa.cms.drposture.app.conf:5)
17 
curl -I http://drposture.app/.well-known/acme-challenge/test-file

HTTP/1.1 404 Not Found
Date: Thu, 23 May 2024 19:43:11 GMT
Server: Apache/2.4.41 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1
18

This is still there.

1 Like
19

This suggests there is something wrong with the paths.

ls /var/www/drposture.app/.well-known/acme-challenge

Should show the file. If not there will be some digging to do.
And BTW you have a bunch of certificates for a bunch of subdomains, and many recent. So I am wondering what is the difference.. or what has been possibly changed.

And checking the certificate transparency logs:

Shows certificates issued as recently as a couple days ago. (yesterday)

How is your configuration different for drposture.app than www.drposture.app or batapi.drposture.app
They should be combined into a single cert or a wildcard certificate.
What gives here.. ??

Please do this and lets fix the issue.

2 Likes
20

Supplemental information.

This is the presently being served certificate crt.sh | 13052872281 by

This one is "It's all good. We have not detected any issues."

These 2 "Doesn't match Common Name or/and SANs"

Edit: this is what I see for each of those domain names using curl -k -i

$ curl -k -i https://api.drposture.app/
HTTP/1.1 404 Not Found
Date: Fri, 24 May 2024 02:24:45 GMT
Server: Kestrel
Content-Length: 0

$ curl -k -i https://www.drposture.app/
HTTP/1.1 404 Not Found
Date: Fri, 24 May 2024 02:24:47 GMT
Server: Kestrel
Content-Length: 0

$ curl -k -i https://drposture.app/
HTTP/1.1 404 Not Found
Date: Fri, 24 May 2024 02:24:50 GMT
Server: Kestrel
Content-Length: 0

@d-shiri you stated

Yet I see Server: Kestrel for all those domain names.

Edit again:
All 3 domain names map to the same IPv4 Address

$ nslookup drposture.app hydrogen.ns.hetzner.com.
Server:         hydrogen.ns.hetzner.com.
Address:        213.133.100.98#53

Name:   drposture.app
Address: 49.12.242.201

$ nslookup www.drposture.app hydrogen.ns.hetzner.com.
Server:         hydrogen.ns.hetzner.com.
Address:        213.133.100.98#53

Name:   www.drposture.app
Address: 49.12.242.201

$ nslookup api.drposture.app hydrogen.ns.hetzner.com.
Server:         hydrogen.ns.hetzner.com.
Address:        213.133.100.98#53

Name:   api.drposture.app
Address: 49.12.242.201

Yet for HTTP not HTTPS I see Apache, but not all 3 domain names respond the same (which may or may not be an issue; I suspect at least www would be an issue.)

$ curl -k -i http://api.drposture.app/.well-known/acme-challenge/sometestfile
HTTP/1.1 301 Moved Permanently
Date: Fri, 24 May 2024 02:53:50 GMT
Server: Apache/2.4.41 (Ubuntu)
Location: https://api.drposture.app/.well-known/acme-challenge/sometestfile
Content-Length: 356
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://api.drposture.app/.well-known/acme-challenge/sometestfile">here</a>.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at api.drposture.app Port 80</address>
</body></html>

$ curl -k -i http://www.drposture.app/.well-known/acme-challenge/sometestfile
HTTP/1.1 301 Moved Permanently
Date: Fri, 24 May 2024 02:53:55 GMT
Server: Apache/2.4.41 (Ubuntu)
Location: https://www.drposture.app/.well-known/acme-challenge/sometestfile
Content-Length: 356
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.drposture.app/.well-known/acme-challenge/sometestfile">here</a>.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at www.drposture.app Port 80</address>
</body></html>

$ curl -k -i http://drposture.app/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Date: Fri, 24 May 2024 02:53:59 GMT
Server: Apache/2.4.41 (Ubuntu)
Content-Length: 275
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at drposture.app Port 80</address>
</body></html>
4 Likes