Certbot kept failing to obtain new certificate on newly installed Apache/2.4.38 on Debian 10 due to Rewrite Rule misconfiguration

spititan · June 28, 2020, 5:13am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

zhangxiaopan.net
www.zhangxiaopan.net

I ran this command:
sudo certbot --apache -d zhangxiaopan.net -d www.zhangxiaopan.net

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for www.zhangxiaopan.net
http-01 challenge for zhangxiaopan.net
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/wD4OrUZoXpK10UKC_DtLm5l1ayHv7QdVn7pOk3Jhpqk [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”, www.zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/YQiD25FsjUa2Fa84hyOjgRpdBfIADkNn2D5rMbsTvxg [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: zhangxiaopan.net
Type: unauthorized
Detail: Invalid response from
http://zhangxiaopan.net/.well-known/acme-challenge/wD4OrUZoXpK10UKC_DtLm5l1ayHv7QdVn7pOk3Jhpqk
[35.226.195.156]: “\n\n404 Not
Found\n\n

Not Found
\n<p”
Domain: www.zhangxiaopan.net
Type: unauthorized
Detail: Invalid response from
http://www.zhangxiaopan.net/.well-known/acme-challenge/YQiD25FsjUa2Fa84hyOjgRpdBfIADkNn2D5rMbsTvxg
[35.226.195.156]: “\n\n404 Not
Found\n\n

Not Found
\n<p”
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.

My web server is (include version):

root@wordpress-vm-4:/var/www/html# /usr/sbin/apache2 -v
Server version: Apache/2.4.38 (Debian)
Server built: 2019-10-15T19:53:42

The operating system my web server runs on is (include version):

Debian 10
root@wordpress-vm-4:/var/www/html# uname -a
Linux wordpress-vm-4 4.19.0-9-cloud-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux

My hosting provider, if applicable, is:

N/A, I built everything on top of Google Compute Engine (VM)

I can login to a root shell on my machine (yes or no, or I don’t know):

Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

N/A

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

root@wordpress-vm-4:/var/www/html# certbot --version
certbot 0.31.0

===== Additional Information

Here is the log in /var/log/letsencry/letsencrypt.log

      "addressesResolved": [
        "35.226.195.156"
      ],
      "addressUsed": "35.226.195.156"
    }
  ]
}

]
}
2020-06-28 05:01:03,350:DEBUG:acme.client:Storing nonce: 0101WNGuRrzUZsN_E9W5CImVe0lrHBs5Wnj_5bptHTJLf0o
2020-06-28 05:01:03,350:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: zhangxiaopan.net
Type: unauthorized
Detail: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/wD4OrUZoXpK10UKC_DtLm5l1ayHv7QdVn7pOk3Jhpqk [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”

Domain: www.zhangxiaopan.net
Type: unauthorized
Detail: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/YQiD25FsjUa2Fa84hyOjgRpdBfIADkNn2D5rMbsTvxg [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2020-06-28 05:01:03,361:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 168, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 239, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/wD4OrUZoXpK10UKC_DtLm5l1ayHv7QdVn7pOk3Jhpqk [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”, www.zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/YQiD25FsjUa2Fa84hyOjgRpdBfIADkNn2D5rMbsTvxg [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”

2020-06-28 05:01:03,361:DEBUG:certbot.error_handler:Calling registered functions
2020-06-28 05:01:03,361:INFO:certbot.auth_handler:Cleaning up challenges
2020-06-28 05:01:03,530:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.31.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1365, in main
return config.func(config, plugins)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1119, in run
certname, lineage)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 121, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 410, in obtain_and_enroll_certificate
cert, chain, key, _ = self.obtain_certificate(domains)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 353, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 389, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 168, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 239, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/wD4OrUZoXpK10UKC_DtLm5l1ayHv7QdVn7pOk3Jhpqk [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”, www.zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/YQiD25FsjUa2Fa84hyOjgRpdBfIADkNn2D5rMbsTvxg [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”
root@wordpress-vm-4:/var/log/letsencrypt#

Here is the Apache config

hangxiaopan@wordpress-vm-4:/var/www/html$ sudo apache2ctl -S
VirtualHost configuration:
*:443 zhangxiaopan.net (/etc/apache2/sites-enabled/zhangxiaopan.net.conf:1)
*:80 zhangxiaopan.net (/etc/apache2/sites-enabled/zhangxiaopan.net.conf:1)
ServerRoot: “/etc/apache2”
Main DocumentRoot: “/var/www/html”
Main ErrorLog: “/var/log/apache2/error.log”
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
PidFile: “/var/run/apache2/apache2.pid”
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name=“www-data” id=33
Group: name=“www-data” id=33

=== Virtual Host Config

<VirtualHost *:80 *:443>
ServerAdmin admin@your_email_domain
ServerName zhangxiaopan.net
ServerAlias www.zhangxiaopan.net
DocumentRoot /var/www/zhangxiaopan.net

    <IfModule mod_rewrite.c>
      RewriteEngine On
      RewriteOptions InheritDownBefore
      RewriteRule "^/.well-known/acme-challenge/" - [END]
    </IfModule>

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

I had suspected it is .htaccess config because I see ‘certbot’ tried to enable some rewrite engine which my apache did not have by default (later I enabled those the rewrite module manually but it still failed), so I searched the internet to tried some random advice (like https://blog.rimuhosting.com/2018/11/29/solve-letsencrypt-including-certbot-problems-caused-by-rogue-htaccess-files/), but got no lucky.

Could somebody please help?

spititan · June 28, 2020, 6:49am

Btw, I tied to use https://letsdebug.net/ to debug my website, and it reports no error. So the problem seems to be somewhere between certbot and my Apache configurations

_az · June 28, 2020, 6:55am

It might be because of this joined listen address - I'm not sure whether Certbot's configuration parser can deal with it. Try removing the :443 and having it only listen on :80.

I don't think the mod_rewrite bit should be necessary either.

spititan · June 28, 2020, 3:07pm

Mmm, I removed the suspicious multiple listening port and change it to
<VirtualHost *>
ServerAdmin admin@your_email_domain
ServerName zhangxiaopan.net
ServerAlias www.zhangxiaopan.net
DocumentRoot /var/www/zhangxiaopan.net
ErrorLog {APACHE_LOG_DIR}/error.log CustomLog {APACHE_LOG_DIR}/access.log combined

,
but the program still failed:

zhangxiaopan@wordpress-vm-4:~$ sudo apache2ctl -S
VirtualHost configuration:
: zhangxiaopan.net (/etc/apache2/sites-enabled/zhangxiaopan.net.conf:1)
ServerRoot: “/etc/apache2”
Main DocumentRoot: “/var/www/html”
Main ErrorLog: “/var/log/apache2/error.log”
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
PidFile: “/var/run/apache2/apache2.pid”
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name=“www-data” id=33
Group: name=“www-data” id=33

zhangxiaopan@wordpress-vm-4:~$ sudo certbot --apache -d zhangxiaopan.net -d www.zhangxiaopan.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for www.zhangxiaopan.net
http-01 challenge for zhangxiaopan.net
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/v7Ca6IPNCCT5irKfPc4MNCS6B1JvbcX2roP4qRGdN1c [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”, www.zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/EWiZiFNqbcqqsnw9dXM3L6Arb6r3tJTZKwQy7KtucJo [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: zhangxiaopan.net
   Type:   unauthorized
   Detail: Invalid response from
   http://zhangxiaopan.net/.well-known/acme-challenge/v7Ca6IPNCCT5irKfPc4MNCS6B1JvbcX2roP4qRGdN1c
   [35.226.195.156]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: www.zhangxiaopan.net
   Type:   unauthorized
   Detail: Invalid response from
   http://www.zhangxiaopan.net/.well-known/acme-challenge/EWiZiFNqbcqqsnw9dXM3L6Arb6r3tJTZKwQy7KtucJo
   [35.226.195.156]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

spititan · June 28, 2020, 3:17pm

I also manually created the file /var/www/zhangxiaopan.net/.well-known/acme-challenge/LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k, and then successfully accessed that file in web browser

http://zhangxiaopan.net/.well-known/acme-challenge/LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k

spititan · June 28, 2020, 3:45pm

I spent some time on reading the /var/log/letsencrypt/letsencrypt.log, and found that the certbot tried to modify the config to set up a temporary file at /var/lib/letsencrypt and access it thought browser, and I also noticed that failure seems to be an authorization, a.k.a. it is a 403 instead of 404. I suspect that the configuration snippets added by certbot somehow did not work in my apache, did anybody see error like this before?

“challenges”: [
{
“type”: “http-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:unauthorized”,
“detail”: “Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k [35.226.195.156]: “\u003c!DOCTYPE HTML PUBLIC \”-//IETF//DTD HTML 2.0//EN\”\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not Found\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot Found\u003c/h1\u003e\n\u003cp"",
“status”: 403
},
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/5531127551/3yDatw”,
“token”: “LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k”,
“validationRecord”: [
{
“url”: “http://zhangxiaopan.net/.well-known/acme-challenge/LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k”,
“hostname”: “zhangxiaopan.net”,
“port”: “80”,
“addressesResolved”: [
“35.226.195.156”
],
“addressUsed”: “35.226.195.156”
}
]

spititan · June 28, 2020, 5:51pm

I managed to suspend the certbot process during its operation and noticed the trick it does is set up a rewrite as following then access the specified URL:

root@wordpress-vm-4:/etc/apache2# cat /etc/apache2/le_http_01_challenge_pre.conf
RewriteEngine on
RewriteRule ^/.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

root@wordpress-vm-4:/etc/apache2# cat /etc/apache2/le_http_01_challenge_post.conf
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted

<Location /.well-known/acme-challenge>
Require all granted

root@wordpress-vm-4:/etc/apache2# cat /etc/apache2/sites-enabled/zhangxiaopan.net.conf
<VirtualHost *>
Include /etc/apache2/le_http_01_challenge_pre.conf
ServerAdmin admin@your_email_domain
ServerName zhangxiaopan.net
ServerAlias www.zhangxiaopan.net
DocumentRoot /var/www/zhangxiaopan.net
ErrorLog {APACHE_LOG_DIR}/error.log CustomLog {APACHE_LOG_DIR}/access.log combined
Include /etc/apache2/le_http_01_challenge_post.conf

and then set up a few files in /var/lib/letsencrypt/http_challenges

root@wordpress-vm-4:/etc/apache2# ls -la /var/lib/letsencrypt/http_challenges
total 24
drwxr-xr-x 2 root root 4096 Jun 28 17:39 .
drwxr-xr-x 5 root root 4096 Jun 28 17:24 …
-rw-r–r-- 1 root root 87 Jun 28 17:24 EHB0fyyr1mx8oBHZl9_RuwbnwYc1joTK34YdkXHTMl8
-rw-r–r-- 1 root root 87 Jun 28 17:24 uqEGWuYeZPWHwcy8kADtyqyG5A4hAk-IMw_Xit35wK8

However, after I manually reload apache config and try to access http://zhangxiaopan.net/.well-known/acme-challenge/EHB0fyyr1mx8oBHZl9_RuwbnwYc1joTK34YdkXHTMl8
The browser gives me 404 Not Found.

I suspect that the rewrite rule added by certbot somehow does not work in my apache configuration.

JuergenAuer · June 28, 2020, 5:57pm

Hi @spititan

is this

really your configuration? If yes, it's wrong. See the documentation of your Apache.

spititan · June 28, 2020, 6:08pm

@JuergenAuer

Well, the apache document says:

https://httpd.apache.org/docs/2.4/mod/core.html#virtualhost

<VirtualHost addr[:port] [addr[:port]] …> …

So the port part should be optional,

BTW, sudo apache2ctl configtest

says the syntax is OK.

JuergenAuer · June 28, 2020, 6:16pm

Yes, but that may be the reason Certbot doesn’t understand your configuration.

Create a named vHost *:80.

spititan · June 28, 2020, 6:28pm

@ JuergenAuer,

Thanks for the suggestion, I will give it a try.

While I was continuing dedugging the apache, I turned on the loglevel to mod_rewrite and notice the following error message:

Sun Jun 28 18:14:08.569326 2020] [rewrite:trace3] [pid 3343:tid 140107252225792] mod_rewrite.c(483): [client 73.70.19.225:53197] 73.70.19.225 - - [zhangxiaopan.net/sid#7f6d4a273a40][rid#7f6d4a1cc0a0/initial] applying pattern '^/.well-known/acme-challenge/' to uri '/.well-known/acme-challenge/EHB0fyyr1mx8oBHZl9_RuwbnwYc1joTK34YdkXHTMl8'
[Sun Jun 28 18:14:08.569334 2020] [rewrite:trace8] [pid 3343:tid 140107252225792] mod_rewrite.c(483): [client 73.70.19.225:53197] 73.70.19.225 - - [zhangxiaopan.net/sid#7f6d4a273a40][rid#7f6d4a1cc0a0/initial] Rule has END flag, no further rewriting for this request

then I noticed that I have a no-acme-challenge-rewrite.conf enabled,

<IfModule mod_rewrite.c>
  RewriteOptions InheritDownBefore
  RewriteRule "^/.well-known/acme-challenge/" - [END]
</IfModule>

I guess it was put by certbot there for safeguard for other sites not doing verification, I disabled it and now the URL http://zhangxiaopan.net/.well-known/acme-challenge/EHB0fyyr1mx8oBHZl9_RuwbnwYc1joTK34YdkXHTMl8 can be accessed through the web browser, although I still do not have idea how to not make that conf take effect in the VirtualHost zhangxiaopan.net.

spititan · June 28, 2020, 6:41pm

Ah, I explicitly enable the rewrite engine and ignore other rules in the VirtualHost configuration as the following, now cerbot is happy:

RewriteEngine on
RewriteOptions IgnoreInherit

Thanks for everybody’s help!

spititan · June 28, 2020, 7:16pm

Now https://zhangxiaopan.net/ works, and I updated the subject of this post to make it more accurately describe the problem and solution.

system · July 28, 2020, 7:16pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems: Help	7	13156	December 16, 2021
Terrible mistakes, tried to reinstall certbot and failed Help	5	2092	September 26, 2018
Certbot failed to authenticate some domains (authenticator: apache) Help	3	206	June 2, 2024
Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems Help	2	2136	June 22, 2023
Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems: Domain: www.iimeedu.com Type: unauthorized Detail: 184.107.37.82: Invalid response from http://www.iimeedu.com/.well-known/acme Help	38	816	January 16, 2023

Certbot kept failing to obtain new certificate on newly installed Apache/2.4.38 on Debian 10 due to Rewrite Rule misconfiguration

Not Found

Not Found

Not Found

Not Found

Not Found

Not Found

Not Found

Not Found

Not Found

Not Found

Not Found

Not Found

Related topics