Invalid response from acme-challenge (apache) after (biggish) server hardware/os migration

piersg · March 4, 2022, 3:48am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: cjz.com.au

I ran this command: sudo certbot renew --dry-run

It produced this output:

Before I paste the output, let me say I can understand exactly what the mix of 401, 403 and 404's mean. In most cases apache is just straight proxying the vhosts to other servers where certbot will never have any write ability. It feels like I'm just missing some apache config higher up that intercepts */.well-known/acme-challenge?

OK, the output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/chippo.cjz.com.au.conf

Unable to read ssl_module file; not disabling session tickets.
Simulating renewal of an existing certificate for ftp.cjz.com.au and 4 more domains

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: abacus.cjz.com.au
Type: unauthorized
Detail: Invalid response from http://abacus.cjz.com.au/.well-known/acme-challenge/2TYQKAdnlmTUrZ262AtqwocsPiqPONxs1aKXeB4YpBI [203.87.126.94]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Domain: ftp.cjz.com.au
Type: unauthorized
Detail: Invalid response from https://ftp.cjz.com.au/.well-known/acme-challenge/BQCBKUQLc-td3P2L9v4q7nSY4yNimQUtUmKl8PfJSwQ [203.87.126.94]: "\r\n404 Not Found\r\n\r\n

Not Found

\r\n

The requested URL was not found on this server"

Domain: mnk.cjz.com.au
Type: unauthorized
Detail: Invalid response from https://mnk.cjz.com.au/.well-known/acme-challenge/0gKFzTFIIhEZT-uhvWdDsEoZQ7tsBGIWPAu8_emFJfk [203.87.126.94]: "\n\n401 Unauthorized\n\n

Unauthorized</"

Domain: view.cjz.com.au
Type: unauthorized
Detail: Invalid response from https://view.cjz.com.au/.well-known/acme-challenge/Kt_ls0cYrT_qBcJ-NPRkMj3jeBl58BPwNAhERdmgeVg [203.87.126.94]: "\n\n401 Unauthorized\n\n

Unauthorized</"

Domain: chippo.cjz.com.au
Type: unauthorized
Detail: Invalid response from https://chippo.cjz.com.au/.well-known/acme-challenge/I0BJpuM8D5q5NiK_2jt3qQt-7lEaVs3ERT58okyCnZI [203.87.126.94]: "\n\n401 Unauthorized\n\n

Unauthorized</"

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate chippo.cjz.com.au with error: Some challenges have failed.

All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/chippo.cjz.com.au/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):Apache/2.4.51

The operating system my web server runs on is (include version): macos 12.2.1

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.23.0

Further info: apache vhosts file and /etc/letsencypt were directly copied from older server (macos 10.13, certbot 0.33.0). Obviously, this might be a terrible idea but it seemed to work quite well until I thought to try a renew test. Trying a straight 'certbot apache' produces the same errors.

apachectl -s output:
VirtualHost configuration:
*:443 is a NameVirtualHost
default server ftp.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:70)
port 443 namevhost ftp.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:70)
port 443 namevhost chippo.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:95)
alias chippo.cjz.com.au
port 443 namevhost mnk.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:129)
alias mnk.cjz.com.au
port 443 namevhost view.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:176)
*:80 is a NameVirtualHost
default server ftp.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:61)
port 80 namevhost ftp.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:61)
port 80 namevhost chippo.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:86)
port 80 namevhost mnk.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:120)
port 80 namevhost abacus.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:154)
port 80 namevhost view.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:165)
port 80 namevhost static.cjz.com.au (/private/etc/apache2/other/au.com.cjz.vhosts.conf:248)
ServerRoot: "/usr"
Main DocumentRoot: "/Library/WebServer/Documents"
Main ErrorLog: "/private/var/log/apache2/error_log"
Mutex ssl-stapling: using_defaults
Mutex ldap-cache: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/private/var/run/" mechanism=default
Mutex mpm-accept: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex rewrite-map: using_defaults
PidFile: "/private/var/run/httpd.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="_www" id=70
Group: name="_www" id=70

_az · March 4, 2022, 4:00am

Certbot temporarily injects some Apache configuration which does exactly this.

It would seem though, that in your case, this is no longer working correctly.

You could check in your most recent /var/log/letsencrypt/letsencrypt.log file to check whether Certbot is finding and modifying the correct Apache configuration files, and the contents of those changes. (Or post the full log here).

piersg · March 4, 2022, 5:35am

This seems to be the relevant section - can't see any errors:

2022-03-04 16:24:26,285:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: abacus.cjz.com.au in: /private/etc/apache2/other/au.com.cjz.vhosts.conf
2022-03-04 16:24:26,285:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: chippo.cjz.com.au in: /private/etc/apache2/other/au.com.cjz.vhosts.conf
2022-03-04 16:24:26,285:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: chippo.cjz.com.au in: /private/etc/apache2/other/au.com.cjz.vhosts.conf
2022-03-04 16:24:26,286:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: ftp.cjz.com.au in: /private/etc/apache2/other/au.com.cjz.vhosts.conf
2022-03-04 16:24:26,286:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: ftp.cjz.com.au in: /private/etc/apache2/other/au.com.cjz.vhosts.conf
2022-03-04 16:24:26,286:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: mnk.cjz.com.au in: /private/etc/apache2/other/au.com.cjz.vhosts.conf
2022-03-04 16:24:26,286:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: mnk.cjz.com.au in: /private/etc/apache2/other/au.com.cjz.vhosts.conf
2022-03-04 16:24:26,286:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: view.cjz.com.au in: /private/etc/apache2/other/au.com.cjz.vhosts.conf
2022-03-04 16:24:26,287:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: view.cjz.com.au in: /private/etc/apache2/other/au.com.cjz.vhosts.conf
2022-03-04 16:24:26,287:DEBUG:certbot_apache._internal.http_01:writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]
    
2022-03-04 16:24:26,287:DEBUG:certbot_apache._internal.http_01:writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>

_az · March 4, 2022, 6:11am

If you try:

certbot certonly --apache -d ftp.cjz.com.au --dry-run --debug-challenges

wait a few seconds when prompted, then press Enter to continue, does it succeed or still not?

piersg · March 4, 2022, 6:16am

So I was able to see certbot adding 2 files and inserting Includes to them around each vhost entry

, like so:

<VirtualHost *:80>
    Include /etc/apache2/other/le_http_01_challenge_pre.conf

(etc)

So that all seems to be working.

Going to try:

backup my config
put the certbot-modified config files in
restart apache
See if I can reach the acme-challenge folder

piersg · March 4, 2022, 6:46am

Seeing a lot of AH01630 Client Denied by Server Configuration Error in the log. This is NOT an apache 2.2 -> 2.4 update, the old server was 2.4.something, but maybe some config is whack ..

9peppe · March 4, 2022, 9:33am

Try this command:

https://httpd.apache.org/docs/current/mod/mod_info.html#startup

piersg · March 7, 2022, 10:35pm

after 2.5 working days of "I have to be going insane", figured out that something is periodically stopping apachectl restart from actually restarting apache until you actually KILL it or reboot the machine.

Not a certbot issue I would think.

system · April 6, 2022, 10:35pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Certbot: Invalid response http://www.example.org/.well-known/acme-challenge Help	52	1115	July 30, 2023
Invalid response from http://zak.paulx2.uk/.well-known/acme-challenge Help	4	448	August 21, 2023
Certbot renewal unauthorized error, acme-challenge invalid response Help	10	5108	January 17, 2020
Invalid response from acme-challenge Help	7	2885	December 24, 2019
Invalid response from .well-known/acme-challenge Help	32	9013	January 23, 2022

Invalid response from acme-challenge (apache) after (biggish) server hardware/os migration

Forbidden

Not Found

Unauthorized</"

Unauthorized</"

Unauthorized</"

Related topics