Certbot kept failing to obtain new certificate on newly installed Apache/2.4.38 on Debian 10 due to Rewrite Rule misconfiguration

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

zhangxiaopan.net
www.zhangxiaopan.net

I ran this command:
sudo certbot --apache -d zhangxiaopan.net -d www.zhangxiaopan.net

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for www.zhangxiaopan.net
http-01 challenge for zhangxiaopan.net
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/wD4OrUZoXpK10UKC_DtLm5l1ayHv7QdVn7pOk3Jhpqk [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”, www.zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/YQiD25FsjUa2Fa84hyOjgRpdBfIADkNn2D5rMbsTvxg [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”

IMPORTANT NOTES:

My web server is (include version):

root@wordpress-vm-4:/var/www/html# /usr/sbin/apache2 -v
Server version: Apache/2.4.38 (Debian)
Server built: 2019-10-15T19:53:42

The operating system my web server runs on is (include version):

Debian 10
root@wordpress-vm-4:/var/www/html# uname -a
Linux wordpress-vm-4 4.19.0-9-cloud-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux

My hosting provider, if applicable, is:

N/A, I built everything on top of Google Compute Engine (VM)

I can login to a root shell on my machine (yes or no, or I don’t know):

Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

N/A

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

root@wordpress-vm-4:/var/www/html# certbot --version
certbot 0.31.0

===== Additional Information

Here is the log in /var/log/letsencry/letsencrypt.log

      "addressesResolved": [
        "35.226.195.156"
      ],
      "addressUsed": "35.226.195.156"
    }
  ]
}

]
}
2020-06-28 05:01:03,350:DEBUG:acme.client:Storing nonce: 0101WNGuRrzUZsN_E9W5CImVe0lrHBs5Wnj_5bptHTJLf0o
2020-06-28 05:01:03,350:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: zhangxiaopan.net
Type: unauthorized
Detail: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/wD4OrUZoXpK10UKC_DtLm5l1ayHv7QdVn7pOk3Jhpqk [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”

Domain: www.zhangxiaopan.net
Type: unauthorized
Detail: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/YQiD25FsjUa2Fa84hyOjgRpdBfIADkNn2D5rMbsTvxg [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2020-06-28 05:01:03,361:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 168, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 239, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/wD4OrUZoXpK10UKC_DtLm5l1ayHv7QdVn7pOk3Jhpqk [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”, www.zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/YQiD25FsjUa2Fa84hyOjgRpdBfIADkNn2D5rMbsTvxg [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”

2020-06-28 05:01:03,361:DEBUG:certbot.error_handler:Calling registered functions
2020-06-28 05:01:03,361:INFO:certbot.auth_handler:Cleaning up challenges
2020-06-28 05:01:03,530:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.31.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1365, in main
return config.func(config, plugins)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1119, in run
certname, lineage)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 121, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 410, in obtain_and_enroll_certificate
cert, chain, key, _ = self.obtain_certificate(domains)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 353, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 389, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 168, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 239, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/wD4OrUZoXpK10UKC_DtLm5l1ayHv7QdVn7pOk3Jhpqk [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”, www.zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/YQiD25FsjUa2Fa84hyOjgRpdBfIADkNn2D5rMbsTvxg [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”
root@wordpress-vm-4:/var/log/letsencrypt#

Here is the Apache config

hangxiaopan@wordpress-vm-4:/var/www/html$ sudo apache2ctl -S
VirtualHost configuration:
*:443 zhangxiaopan.net (/etc/apache2/sites-enabled/zhangxiaopan.net.conf:1)
*:80 zhangxiaopan.net (/etc/apache2/sites-enabled/zhangxiaopan.net.conf:1)
ServerRoot: “/etc/apache2”
Main DocumentRoot: “/var/www/html”
Main ErrorLog: “/var/log/apache2/error.log”
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
PidFile: “/var/run/apache2/apache2.pid”
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name=“www-data” id=33
Group: name=“www-data” id=33

=== Virtual Host Config

<VirtualHost *:80 *:443>
ServerAdmin admin@your_email_domain
ServerName zhangxiaopan.net
ServerAlias www.zhangxiaopan.net
DocumentRoot /var/www/zhangxiaopan.net

    <IfModule mod_rewrite.c>
      RewriteEngine On
      RewriteOptions InheritDownBefore
      RewriteRule "^/.well-known/acme-challenge/" - [END]
    </IfModule>

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

I had suspected it is .htaccess config because I see ‘certbot’ tried to enable some rewrite engine which my apache did not have by default (later I enabled those the rewrite module manually but it still failed), so I searched the internet to tried some random advice (like https://blog.rimuhosting.com/2018/11/29/solve-letsencrypt-including-certbot-problems-caused-by-rogue-htaccess-files/), but got no lucky.

Could somebody please help?

1 Like

Btw, I tied to use https://letsdebug.net/ to debug my website, and it reports no error. So the problem seems to be somewhere between certbot and my Apache configurations

1 Like

It might be because of this joined listen address - I’m not sure whether Certbot’s configuration parser can deal with it. Try removing the :443 and having it only listen on :80.

I don’t think the mod_rewrite bit should be necessary either.

1 Like

Mmm, I removed the suspicious multiple listening port and change it to
<VirtualHost *>
ServerAdmin admin@your_email_domain
ServerName zhangxiaopan.net
ServerAlias www.zhangxiaopan.net
DocumentRoot /var/www/zhangxiaopan.net
ErrorLog {APACHE_LOG_DIR}/error.log CustomLog {APACHE_LOG_DIR}/access.log combined

,
but the program still failed:

zhangxiaopan@wordpress-vm-4:~$ sudo apache2ctl -S
VirtualHost configuration:
: zhangxiaopan.net (/etc/apache2/sites-enabled/zhangxiaopan.net.conf:1)
ServerRoot: “/etc/apache2”
Main DocumentRoot: “/var/www/html”
Main ErrorLog: “/var/log/apache2/error.log”
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
PidFile: “/var/run/apache2/apache2.pid”
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name=“www-data” id=33
Group: name=“www-data” id=33

zhangxiaopan@wordpress-vm-4:~$ sudo certbot --apache -d zhangxiaopan.net -d www.zhangxiaopan.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for www.zhangxiaopan.net
http-01 challenge for zhangxiaopan.net
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/v7Ca6IPNCCT5irKfPc4MNCS6B1JvbcX2roP4qRGdN1c [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”, www.zhangxiaopan.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.zhangxiaopan.net/.well-known/acme-challenge/EWiZiFNqbcqqsnw9dXM3L6Arb6r3tJTZKwQy7KtucJo [35.226.195.156]: “\n\n404 Not Found\n\n

Not Found

\n<p”
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: zhangxiaopan.net
   Type:   unauthorized
   Detail: Invalid response from
   http://zhangxiaopan.net/.well-known/acme-challenge/v7Ca6IPNCCT5irKfPc4MNCS6B1JvbcX2roP4qRGdN1c
   [35.226.195.156]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: www.zhangxiaopan.net
   Type:   unauthorized
   Detail: Invalid response from
   http://www.zhangxiaopan.net/.well-known/acme-challenge/EWiZiFNqbcqqsnw9dXM3L6Arb6r3tJTZKwQy7KtucJo
   [35.226.195.156]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

I also manually created the file /var/www/zhangxiaopan.net/.well-known/acme-challenge/LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k, and then successfully accessed that file in web browser

http://zhangxiaopan.net/.well-known/acme-challenge/LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k

I spent some time on reading the /var/log/letsencrypt/letsencrypt.log, and found that the certbot tried to modify the config to set up a temporary file at /var/lib/letsencrypt and access it thought browser, and I also noticed that failure seems to be an authorization, a.k.a. it is a 403 instead of 404. I suspect that the configuration snippets added by certbot somehow did not work in my apache, did anybody see error like this before?

“challenges”: [
{
“type”: “http-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:unauthorized”,
“detail”: “Invalid response from http://zhangxiaopan.net/.well-known/acme-challenge/LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k [35.226.195.156]: “\u003c!DOCTYPE HTML PUBLIC \”-//IETF//DTD HTML 2.0//EN\”\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not Found\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot Found\u003c/h1\u003e\n\u003cp"",
“status”: 403
},
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/5531127551/3yDatw”,
“token”: “LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k”,
“validationRecord”: [
{
“url”: “http://zhangxiaopan.net/.well-known/acme-challenge/LuG43U8aAtyiDAJhxPj5DEbrF1HutFQskvariFx6v0k”,
“hostname”: “zhangxiaopan.net”,
“port”: “80”,
“addressesResolved”: [
“35.226.195.156”
],
“addressUsed”: “35.226.195.156”
}
]

I managed to suspend the certbot process during its operation and noticed the trick it does is set up a rewrite as following then access the specified URL:

root@wordpress-vm-4:/etc/apache2# cat /etc/apache2/le_http_01_challenge_pre.conf
RewriteEngine on
RewriteRule ^/.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

root@wordpress-vm-4:/etc/apache2# cat /etc/apache2/le_http_01_challenge_post.conf
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted

<Location /.well-known/acme-challenge>
Require all granted

root@wordpress-vm-4:/etc/apache2# cat /etc/apache2/sites-enabled/zhangxiaopan.net.conf
<VirtualHost *>
Include /etc/apache2/le_http_01_challenge_pre.conf
ServerAdmin admin@your_email_domain
ServerName zhangxiaopan.net
ServerAlias www.zhangxiaopan.net
DocumentRoot /var/www/zhangxiaopan.net
ErrorLog {APACHE_LOG_DIR}/error.log CustomLog {APACHE_LOG_DIR}/access.log combined
Include /etc/apache2/le_http_01_challenge_post.conf

and then set up a few files in /var/lib/letsencrypt/http_challenges

root@wordpress-vm-4:/etc/apache2# ls -la /var/lib/letsencrypt/http_challenges
total 24
drwxr-xr-x 2 root root 4096 Jun 28 17:39 .
drwxr-xr-x 5 root root 4096 Jun 28 17:24 …
-rw-r–r-- 1 root root 87 Jun 28 17:24 EHB0fyyr1mx8oBHZl9_RuwbnwYc1joTK34YdkXHTMl8
-rw-r–r-- 1 root root 87 Jun 28 17:24 uqEGWuYeZPWHwcy8kADtyqyG5A4hAk-IMw_Xit35wK8

However, after I manually reload apache config and try to access http://zhangxiaopan.net/.well-known/acme-challenge/EHB0fyyr1mx8oBHZl9_RuwbnwYc1joTK34YdkXHTMl8
The browser gives me 404 Not Found.

I suspect that the rewrite rule added by certbot somehow does not work in my apache configuration.

Hi @spititan

is this

really your configuration? If yes, it’s wrong. See the documentation of your Apache.

2 Likes

@JuergenAuer

Well, the apache document says:

https://httpd.apache.org/docs/2.4/mod/core.html#virtualhost

<VirtualHost addr[:port] [addr[:port]] …> …

So the port part should be optional,

BTW, sudo apache2ctl configtest

says the syntax is OK.

1 Like

Yes, but that may be the reason Certbot doesn’t understand your configuration.

Create a named vHost *:80.

2 Likes

@ JuergenAuer,

Thanks for the suggestion, I will give it a try.

While I was continuing dedugging the apache, I turned on the loglevel to mod_rewrite and notice the following error message:

Sun Jun 28 18:14:08.569326 2020] [rewrite:trace3] [pid 3343:tid 140107252225792] mod_rewrite.c(483): [client 73.70.19.225:53197] 73.70.19.225 - - [zhangxiaopan.net/sid#7f6d4a273a40][rid#7f6d4a1cc0a0/initial] applying pattern '^/.well-known/acme-challenge/' to uri '/.well-known/acme-challenge/EHB0fyyr1mx8oBHZl9_RuwbnwYc1joTK34YdkXHTMl8'
[Sun Jun 28 18:14:08.569334 2020] [rewrite:trace8] [pid 3343:tid 140107252225792] mod_rewrite.c(483): [client 73.70.19.225:53197] 73.70.19.225 - - [zhangxiaopan.net/sid#7f6d4a273a40][rid#7f6d4a1cc0a0/initial] Rule has END flag, no further rewriting for this request

then I noticed that I have a no-acme-challenge-rewrite.conf enabled,

<IfModule mod_rewrite.c>
  RewriteOptions InheritDownBefore
  RewriteRule "^/.well-known/acme-challenge/" - [END]
</IfModule>

I guess it was put by certbot there for safeguard for other sites not doing verification, I disabled it and now the URL http://zhangxiaopan.net/.well-known/acme-challenge/EHB0fyyr1mx8oBHZl9_RuwbnwYc1joTK34YdkXHTMl8 can be accessed through the web browser, although I still do not have idea how to not make that conf take effect in the VirtualHost zhangxiaopan.net.

1 Like

Ah, I explicitly enable the rewrite engine and ignore other rules in the VirtualHost configuration as the following, now cerbot is happy:

RewriteEngine on
RewriteOptions IgnoreInherit

Thanks for everybody’s help!

3 Likes

Now https://zhangxiaopan.net/ works, and I updated the subject of this post to make it more accurately describe the problem and solution.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.