Renewal fails with "Unable to find a virtual host listening on port 80"

Help
1

My domain is: *.eumnestes.org

I ran this command: sudo certbot renew

It produced this output:
$ sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/eumnestes.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
ssl_module is statically linked but --apache-bin is missing; not disabling session tickets.
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for eumnestes.org and 4 more domains
Performing the following challenges:
http-01 challenge for wordhoard.eumnestes.org
http-01 challenge for www.eumnestes.org
http-01 challenge for annocast.eumnestes.org
http-01 challenge for annolex.eumnestes.org
http-01 challenge for eumnestes.org
Cleaning up challenges
Failed to renew certificate eumnestes.org with error: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/eumnestes.org/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

My web server is (include version):
$ httpd -v
Server version: Apache/2.4.46 (Unix)
Server built: Feb 28 2021 04:17:49

Note that this is the Apple-supplied Apache, not Homebrew, etc.

The operating system my web server runs on is (include version):

$ sw_vers
ProductName:	macOS
ProductVersion:	11.3.1
BuildVersion:	20E241
$ uname -m
arm64

My hosting provider, if applicable, is: none

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
$ certbot --version
certbot 1.14.0

Contrary to what certbot thinks, there are virtual hosts listening on port 80:

$ sudo apachectl -S
VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server annocast.eumnestes.org (/private/etc/apache2/extra/httpd-vhosts.conf:25)
         port 80 namevhost annocast.eumnestes.org (/private/etc/apache2/extra/httpd-vhosts.conf:25)
         port 80 namevhost annolex.eumnestes.org (/private/etc/apache2/extra/httpd-vhosts.conf:44)
         port 80 namevhost wordhoard.eumnestes.org (/private/etc/apache2/extra/httpd-vhosts.conf:75)
         port 80 namevhost www.eumnestes.org (/private/etc/apache2/extra/httpd-vhosts.conf:89)
                 alias eumnestes.org
                 wild alias *.eumnestes.org
*:443                  is a NameVirtualHost
         default server wordhoard.eumnestes.org (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:3)
         port 443 namevhost wordhoard.eumnestes.org (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:3)
         port 443 namevhost www.eumnestes.org (/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf:21)
                 alias eumnestes.org
                 alias www.eumnestes.org
ServerRoot: "/usr"
Main DocumentRoot: "/Library/WebServer/Documents"
Main ErrorLog: "/private/var/log/apache2/error_log"
Mutex authdigest-opaque: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex authdigest-client: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/private/var/run/" mechanism=default
Mutex mpm-accept: using_defaults
PidFile: "/private/var/run/httpd.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="_www" id=70
Group: name="_www" id=70

The Apache access log shows that nothing gets as far as requesting a file from .well-known/acme-challenge (but it's got plenty of other successful requests in it). The web sites are working fine, and everything checks out at letsdebug.net, but I just can't renew the cert.

The problem started when I migrated this set-up from an Intel Mac running Mojave to an M1 Mac running Big Sur. Everything under /etc/letsencrypt and the Apache configuration came over without any problems and, as I mentioned, are working fine except for the certificate renewal. I keep thinking I must've mangled something in the migration but I don't know what.

I put a debug statement in _relevant_vhosts in certbot/certbot-apache/certbot_apache/_internal/http_01.py and determined that the following line:

for vhost in self.configurator.vhosts:

does not find anything in self.configurator.vhosts so the loop never iterates, but I have not yet figured out where or how that list of vhosts should be getting populated.

Sorry to go on, but I thought I should get everything on the table. The warning "ssl_module is statically linked" some lines up from the error in the renewal does not seem right to me. As far as I can see, it is actually a dynamic library:

$ ls -l /usr/libexec/apache2/mod_ssl.so
-rwxr-xr-x  1 root  wheel  728080 Jan  1  2020 /usr/libexec/apache2/mod_ssl.so
$ grep mod_ssl.so /etc/apache2/httpd.conf
LoadModule ssl_module libexec/apache2/mod_ssl.so

So there may be something about rooting around in the binary for the OpenSSL version that does not work with ARM binaries. Or not -- just a guess, and this warning may or may not have anything to do with the failure to locate any virtual hosts.

Thanks for reading this far, if you have :-).

1 Like
2

Hi @craigberry

what's the question?

You want to create one certificate with a lot of domain names.

But there is no matching vHost with the same list of domain names.

So the not working result is expected.

Change your certificate configuration or change your vHost configuration.

1 Like
3

But there are matching vhosts. Look at the apachectl -S output I posted. Apache can see the virtual hosts but certbot cannot. Why?

4

There is no matching vHost. Read your output.

1 Like
5

It seems that certbot is unable to match the apache output.
hmm...
Perhaps you have multiple versions of apache installed.
Have you tried using the --webroot option?

2 Likes
6

Welcome to the Let's Encrypt Community, Craig :slightly_smiling_face:

Are you trying to acquire a wildcard certificate for *.eumnestes.org?

If so, you need to use a dns-01 authentication method rather than an http-01 authentication method (like apache).

https://certbot.eff.org/docs/using.html#dns-plugins

Here are the domain names that certbot is trying to verify control of:

Here are the Apache port 80 vHost definitions you have:

I think what @JuergenAuer is meaning here is that certbot is looking for a single matching vHost with all of those domain names, not split amongst several vHosts. If you request certificates for the domain names individually, only having multiple domain names in one certificate if they are within a single vHost, you will probably be fine.

Try these:

sudo certbot --apache -d "eumnestes.org,www.eumnestes.org"

sudo certbot --apache -d "wordhoard.eumnestes.org"

sudo certbot --apache -d "annocast.eumnestes.org"

sudo certbot --apache -d "annolex.eumnestes.org"

Wildcard server aliases are questionable.

Screenshot_20210507-162957_Samsung Internet
Screenshot_20210507-162957_Samsung Internet1440×1687 155 KB

Wildcard DNS A records are also questionable.

https://toolbox.googleapps.com/apps/dig/#A/

Why have ServerAlias www.eumnestes.org when you have ServerName www.eumnestes.org?

3 Likes
7

Thanks for the reply, @rg305. There is only one Apache installation. I may eventually try --webroot or a DNS challenge, but I would like to get to the bottom of why certbot can't read the Apache configuration since my upgrade to Big Sur.

3 Likes
8

No. I used the asterisk as a shorthand for the fact that I have several subdomains; my bad for using a syntax in my problem report that actually means something else. The subdomains share a single certificate, which is supported by certbot. Certbot had no trouble creating the certificate for multiple subdomains, which I specified on the command line with -d for each one when I created the cert last December. When I renewed in March, certbot renew worked flawlessly with this configuration. However, trying to renew again now (after moving to Big Sur) indicates that certbot can no longer read the Apache configuration and decides that I have no virtual hosts at all even though I clearly do.

These all give the exact same error. Multiple virtual hosts seems to be a red herring. Note that someone else has now reported the exact same problem on Big Sur https://community.letsencrypt.org/t/where-should-vhosts-file-be/151306 and in that case there is a single domain.

I will make a mental note to look into these sometime, but they have never caused trouble for me in the past and don't seem relevant to the problem at hand.

The port 443 virtual hosts were created by certbot when I initially created the certificate. I don't know why it did that but it has never seemed to be a problem.

.

2 Likes
9

I'll need to look into this one further. :thinking:

You don't have a certificate that covers giantsecurityhole.eumnestes.org, which has both a valid IP address and a responding VirtualHost due to the wildcard A record and wildcard server alias, respectively. Without a wildcard certificate, you should never have a wildcard A record. Some do try to use Apache hostname matching to filter out these vulnerabilities. Why board up the windows when you can stop attackers at the gate?

It likely won't cause an issue. It's like telling people that your name is Craig and that you are also known as Craig. It creates a small bit of unnecessary redundancy that's wasteful to Apache when matching hostnames.

3 Likes
10

I don't really understand the attack scenario you have in mind here. Care to explain?

3 Likes
11

No problem. :slightly_smiling_face:

Here are three screenshots:

Screenshot_20210508-130017_Samsung Internet
Screenshot_20210508-130017_Samsung Internet1440×2722 245 KB

Screenshot_20210508-130554_Samsung Internet
Screenshot_20210508-130554_Samsung Internet1440×2449 242 KB

Screenshot_20210508-130040_Samsung Internet
Screenshot_20210508-130040_Samsung Internet1440×2722 477 KB

3 Likes
12

Yes, the cert isn't valid for that domain, but I don't really see why that's a problem.

I have literally over a dozen random domains that I neither know nor operate, but they point to my IPv4 address, simply because someone never changed these.

If someone visits one of these domains they will get a cert error, sure. But that's it. If the user ignores all browser warnings my server will reply with a 404 (for any URL, as the vHost doesn't exist) and that's it.

Even if the vhost would exist, it would simply be an alias, with otherwise the same properties as the other name.

2 Likes
13

The main vHost responds here with actual content because it has a wildcard server alias.

If I were to create a phishing email with a link to secure.eumnestes.org, not only does http respond without a redirect, but https has no valid certificate. Just ignore the warning then enter your username and password. Everything will be fine... :wink: :wink:

Screenshot_20210508-131222_Samsung Internet
Screenshot_20210508-131222_Samsung Internet1440×2554 682 KB

3 Likes
14

So the main issue not the fact that there's a DNS wildcard, but that the server doesn't do HTTP -> HTTPS redirects on unknown domain names, potentially causing browser HTTP fallbacks?

I personally do 301 redirects to HTTPS on any HTTP request, without looking at the hostname. This massively simplifies server configuration (there's only a single vhost on HTTP that matches everything and does nothing but returning https redirects).

3 Likes
15

That's a big issue, true. I also take issue with serving valid content unsecurely. The wildcard alias in the vHost creates a virtually unlimited number of unsecure ways to access the content.

https://secure.eumnestes.org/login should be secure. :upside_down_face:

4 Likes
16

Yeah I think I see now what you mean.

By the way, a nice way of preventing users to ignore cert errors is by using HSTS + includeSubDomains + preload list. With HSTS modern browsers will refuse to proceed on cert errors.

3 Likes
17

Quite true. I only wish it was universal. I would surmise that the least security-savvy are likely to be the ones with the most outdated/dysfunctional/compromised browsers and thus the most vulnerable. Some people just ignore the velvet ropes then wonder why they end up with two broken legs.

2 Likes
18

I have removed the wildcard alias and the wildcard A record since those seemed to generate more interest than the actual problem I reported. The only reason that http is not automatically redirected to https is that I have temporarily disabled those redirects because at some point I thought it possible that the http01 challenges from letsencrypt servers might not be correctly following the 301 responses. As I stated in my original post, the Apache access logs prove that things never even get that far. So, if anyone has any insight into why certbot cannot see the Apache virtual hosts that are demonstrably there, I'm all ears.

2 Likes
19

I'm thinking that testing with webroot authentication as @rg305 suggested is probably the most prudent.

Firstly, let's verify the webroots for the task.

What are the contents of these files?

/private/etc/apache2/extra/httpd-vhosts.conf
/private/etc/apache2/extra/httpd-vhosts-le-ssl.conf

Please put 3 backticks on a line above and below each contents as you likely did with your outputs.

2 Likes
20

Thanks, @griffin and @rg305. the certonly --webroot option worked like a charm just like the example in the docs with -w for each root followed by -d for each domain that uses that root. So my certificate is successfully renewed and I'm in good shape until August. Still a puzzle why renew couldn't parse the Apache config on this system.

For folks trying this at home, save yourself a few minutes of head scratching and remember to manually restart Apache because certonly doesn't do that the way renew does :-).

3 Likes