Certbot failed to authenticate some domains (authenticator: apache)

V-Mar · November 17, 2022, 12:21am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: 15kv.com

I ran this command:sudo certbot

It produced this output:
Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: 15kv.com
Type: unauthorized
Detail: 71.143.193.233: Invalid response from http://15kv.com/.well-known/acme-challenge/4187ReSPETkSGAmKx4KOILPrfg36FG8s-LqESGo2cDQ: 403
My web server is (include version):

The operating system my web server runs on is (include version):
macOS Monterey
My hosting provider, if applicable, is:
Myself
I can login to a root shell on my machine (yes or no, or I don't know):
Yep
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): nope

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):1.32.0

V-Mar · November 17, 2022, 12:25am

Here are some dumps from the letsencrypt.log as well as apachectl -S

I should note that I have been running just the run command ie "sudo certbot" then at the prompt choosing the number that corresponds to the domain I want the cert for. On the ones that worked it was magic. The certs were created and the vhost files were updated with all the relevant changes and everything just worked. Those website now redirected all requests to https and all is good. But some of these other ones just fail to get and install the cert no matter what I try.

I have checked Let’s debug and it says everything should work.
I can load http://vmarmachine.shop/.well-known/acme-challenge/
with no issues. (I had to manually create those directories so I could test it)

I have run certbot on several other domains on this same machine. In the vhosts file they all are setup basically the same, yet some work and other give this same “unauthorized” error. I struggling to figure out what might be different between the ones that work and the ones that don’t. I even have a few that the domain level cert worked fine, but I cannot —expand to add the “www” third level to the cert. I get the same unauthorized error. All the domains have their DNS hosted at AWS and are all setup the same. Simple DNS, nothing too fancy.

From /var/log/letsencrypt/letsencrypt.log

2022-11-16 15:59:51,952:DEBUG:certbot._internal.main:certbot version: 1.32.0
2022-11-16 15:59:51,952:DEBUG:certbot._internal.main:Location of certbot entry point: /opt/homebrew/bin/certbot
2022-11-16 15:59:51,952:DEBUG:certbot._internal.main:Arguments:
2022-11-16 15:59:51,952:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,Plug$
2022-11-16 15:59:51,968:DEBUG:certbot._internal.log:Root logging level set at 30
2022-11-16 15:59:51,968:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2022-11-16 15:59:52,111:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.53
2022-11-16 15:59:52,367:DEBUG:certbot_apache._internal.configurator:[Errno 2] No such file or directory: '/etc/apache2/libexec/apache2/mod_ssl.so'
Traceback (most recent call last):
File "/opt/homebrew/Cellar/certbot/1.32.0/libexec/lib/python3.11/site-packages/certbot_apache/_internal/configurator.py", line 297, in _open_module_file
with open(ssl_module_location, mode="rb") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/etc/apache2/libexec/apache2/mod_ssl.so'
2022-11-16 15:59:52,369:WARNING:certbot_apache._internal.configurator:Unable to read ssl_module file; not disabling session tickets.
2022-11-16 15:59:52,369:DEBUG:certbot._internal.plugins.disco:No installation (PluginEntryPoint#nginx): Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executa$
Traceback (most recent call last):
File "/opt/homebrew/Cellar/certbot/1.32.0/libexec/lib/python3.11/site-packages/certbot/_internal/plugins/disco.py", line 160, in prepare
self._initialized.prepare()
File "/opt/homebrew/Cellar/certbot/1.32.0/libexec/lib/python3.11/site-packages/certbot_nginx/_internal/configurator.py", line 194, in prepare
raise errors.NoInstallationError(
certbot.errors.NoInstallationError: Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executable, and your PATH is set correctly.
2022-11-16 15:59:52,370:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_darwin.DarwinConfigurator object at 0x104c9cb50>
Prep: True
2022-11-16 15:59:52,371:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_apache._internal.override_darwin.DarwinConfigurator object at 0x104c9cb50> and installer$
2022-11-16 15:59:52,371:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2022-11-16 15:59:52,376:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_a$
2022-11-16 15:59:52,382:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2022-11-16 15:59:52,599:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2022-11-16 15:59:52,783:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 659
2022-11-16 15:59:52,785:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Wed, 16 Nov 2022 23:59:52 GMT
Content-Type: application/json
Content-Length: 659
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

Results of apachectl -S
*:80 is a NameVirtualHost
default server daserver.org (/private/etc/apache2/extra/httpd-vhosts.conf:28)
port 80 namevhost daserver.org (/private/etc/apache2/extra/httpd-vhosts.conf:28)
alias sf.daserver.org
alias carlisle.daserver.org
port 80 namevhost 15kv.com (/private/etc/apache2/extra/httpd-vhosts.conf:37)
alias www.15kv.com
port 80 namevhost patlacroix.com (/private/etc/apache2/extra/httpd-vhosts.conf:247)
alias www.patlacroix.com
port 80 namevhost pinkyweitzman.com (/private/etc/apache2/extra/httpd-vhosts.conf:262)
alias www.pinkyweitzman.com
port 80 namevhost plantsf.org (/private/etc/apache2/extra/httpd-vhosts.conf:273)
alias www.plantsf.org
alias plantsf.org
port 80 namevhost prichard.net (/private/etc/apache2/extra/httpd-vhosts.conf:288)
alias www.prichard.net
port 80 namevhost vmar.com (/private/etc/apache2/extra/httpd-vhosts.conf:328)
alias www.vmar.com
alias vmar.com
port 80 namevhost vmarmachine.shop (/private/etc/apache2/extra/httpd-vhosts.conf:361)
alias www.vmarmachine.shop
port 80 namevhost v-mar.com (/private/etc/apache2/extra/httpd-vhosts.conf:379)
port 80 namevhost wino.com (/private/etc/apache2/extra/httpd-vhosts.conf:410)
alias www.wino.com
port 80 namevhost wino.fm (/private/etc/apache2/extra/httpd-vhosts.conf:432)
alias www.wino.fm
port 80 namevhost www.v-mar.com (/private/etc/apache2/extra/httpd-vhosts.conf:473)
*:443 is a NameVirtualHost
default server wino.com (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:2)
port 443 namevhost wino.com (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:2)
alias www.wino.com
port 443 namevhost vmar.com (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:26)
alias www.vmar.com
port 443 namevhost plantsf.org (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:62)
alias www.plantsf.org
alias plantsf.org
port 443 namevhost org.org (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:79)
alias www.org.org
port 443 namevhost v-mar.com (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:96)
alias www.v-mar.com
port 443 namevhost patlacroix.com (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:129)
alias www.patlacroix.com
ServerRoot: "/usr"
Main DocumentRoot: "/Volumes/storage/web"
Main ErrorLog: "/private/var/log/apache2/error_log"
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/private/var/run/" mechanism=default
Mutex mpm-accept: using_defaults
Mutex cache-socache: using_defaults
Mutex watchdog-callback: using_defaults
PidFile: "/private/var/run/httpd.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="_www" id=70
Group: name="_www" id=70

_az · November 17, 2022, 12:30am

Can you check in this file for any lines related to the /.well-known/acme-challenge request? A 403 of this kind should usually produce something in this file.

V-Mar · November 17, 2022, 12:43am

[Wed Nov 16 11:46:49.525267 2022] [autoindex:error] [pid 51088] [client 20.198.221.121:58206] AH01276: Cannot serve directory /Volumes/storage/web/Sites/vmar.com/.well-known/: No matching DirectoryIndex (index.html) found, and server-generated directory index forbidden by Opt$

That's the only reference, but strangely, this is one of the domains that has a working cert.

rg305 · November 17, 2022, 4:40am

And then what?
It must of had question... before it could give that output.

This is a file of interest:

We should start there looking for clues.

Like how did a request that should have been handled by that file create the error message:

And ideal test would place a test text file in that location.

V-Mar · November 17, 2022, 3:59pm

To answer your first question, that was in the original post,
After sudo certbot it brings up a list of all the domains it can find in the vhosts file. I choose the number(s) corresponding to the domains I wanted to add the cert for ie. 45 46
Then certbot attempts to verify the domains. Some of them worked fine, the cert was created and the vhosts file was modified and the second httpd-vhosts-le-ssl.conf file was also modified with the newly added :443 entries to match the new cert. All of this worked perfectly for some of the domains on my server (we have about 25 domains on this machine along with several subdomains each, so around 100 if you count them all). Other ones fail with the error listed in the original post.

Second question, yeah, I've looked thru the vhosts file for days now and there really isn't anything different between the ones that worked and the ones that failed so it makes me think it's something else.

Last question, yeah I did put an index.html in there. It returns "nothing to see here", you could click the link and verify that it works for you as well. I should note that the few domains that I used the simple run command (sudo certbot) I did not need to manually add a /.well-know/acme-challenge directory, and there is not one there now, so I think the Apache plugin does something different, or maybe removes the directory after it is done with the verification? I have no idea.

One thing I should add. I also checked the permissions for the various website directories and they are all set the same.
Example:
drwxrwxr-x 65 vince staff 2080 Sep 6 2021 15kv.com

V-Mar · November 17, 2022, 4:12pm

I just tried again using certbot certonly --dry-run -d vmarmachine.shop -d www.vmarmachine.shop

Could it be the missing mod-ssl.so ?? It still doesn't explain why some of the certs worked and other didn't ? (Edit: Did some more reading, this may be a red herring.)

Here is the letsencrypt.log :

2022-11-17 08:03:43,633:DEBUG:certbot._internal.main:certbot version: 1.32.0
2022-11-17 08:03:43,633:DEBUG:certbot._internal.main:Location of certbot entry point: /opt/homebrew/bin/certbot
2022-11-17 08:03:43,633:DEBUG:certbot._internal.main:Arguments: ['--dry-run', '-d', 'vmarmachine.shop', '-d', 'www.vmarmachine.shop']
2022-11-17 08:03:43,633:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-11-17 08:03:43,659:DEBUG:certbot._internal.log:Root logging level set at 30
2022-11-17 08:03:43,660:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2022-11-17 08:03:44,738:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.53
2022-11-17 08:03:45,001:DEBUG:certbot_apache._internal.configurator:[Errno 2] No such file or directory: '/etc/apache2/libexec/apache2/mod_ssl.so'
Traceback (most recent call last):
File "/opt/homebrew/Cellar/certbot/1.32.0/libexec/lib/python3.11/site-packages/certbot_apache/_internal/configurator.py", line 297, in _open_module_file
with open(ssl_module_location, mode="rb") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/etc/apache2/libexec/apache2/mod_ssl.so'
2022-11-17 08:03:45,003:WARNING:certbot_apache._internal.configurator:Unable to read ssl_module file; not disabling session tickets.
2022-11-17 08:03:45,005:DEBUG:certbot._internal.plugins.disco:No installation (PluginEntryPoint#nginx): Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executable, and your PATH is set correctly.
Traceback (most recent call last):
File "/opt/homebrew/Cellar/certbot/1.32.0/libexec/lib/python3.11/site-packages/certbot/_internal/plugins/disco.py", line 160, in prepare
self._initialized.prepare()
File "/opt/homebrew/Cellar/certbot/1.32.0/libexec/lib/python3.11/site-packages/certbot_nginx/_internal/configurator.py", line 194, in prepare
raise errors.NoInstallationError(
certbot.errors.NoInstallationError: Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executable, and your PATH is set correctly.
2022-11-17 08:03:45,006:DEBUG:certbot._internal.plugins.selection:Multiple candidate plugins: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_darwin.DarwinConfigurator object at 0x1059bd710>
Prep: True

rg305 · November 17, 2022, 4:56pm

Sorry, but I cringe whever I encounter Apache in this forum.
Have you considered using the webroot authentication method?

V-Mar · November 17, 2022, 5:46pm

Yeah, that worked. (using webroot)

It still kinda irks me that half a dozen or so domains worked with apache authenticator. I'm sure you can understand how frustrating intermittent or inconsistent errors can be. I'm more obsessed with understanding what worked or didn't work with seemingly identical domains.

But, more importantly, I have a way to finish getting all the domains switched over to https

Thanks for the help.
I don't really want to check the "solution" box, since we never did get to the root of the problem, but it's a very usable workaround

Finally, will it make any difference when it comes time to renew if some domains were initially setup with apache and others with webroot?

rg305 · November 17, 2022, 6:48pm

No; They will each renew as obtained.

rg305 · November 17, 2022, 6:49pm

Maybe a fresh pair of might find the reason.

V-Mar · November 17, 2022, 6:52pm

Agreed, my eyes were going cross-eyed after a while.
But for now I have almost everything up and running using --webroot
So thanks for that.

V-Mar · November 21, 2022, 8:28pm

Well, I'll consider this issue closed.
Just for anyone who finds this thread in the future who is having similar problems with the Apache plugin, the solution for me, was to stop trying to get the plugin to work and just use the webroot authentication.

Everything worked retrieving the certs and installing them. I had to manually edit the https_vhosts.conf and httpd_vhosts_le_ssl.conf files, but that was pretty easy following the instructions.

The only hiccup with the webroot install for me, was that the renewal files were not populated correctly for some of the domains. When I tried to test them using renew --dry-run I have 2 or 3 certs that give this error:
Simulating renewal of an existing certificate for wino.com and www.wino.com
Failed to renew certificate wino.com with error: Missing command line flag or config entry for this setting:
Input the webroot for wino.com:

It turns out the webroot line was missing from the /etc/letsencrypt/renewal/your-cert-name.conf
I had to add the line webroot_path = /path-to-your-website-folder, (note the comma at the end of the line)

After adding the line with the relevant path to each of the renewal files that were throwing the error everything worked.

Hope this helps anyone who encounters similar issues.
Thanks to all who helped me track down the problems.

system · December 21, 2022, 8:28pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.