Certbot 1.20.0 cert creation fails with a 404, even though the site is available on port 80

[griffin]

Just dump the output to a text file:

sudo ls -lRa /opt/homebrew > output.txt

then upload output.txt in a post here using the upload button.

[geekstrada]

Let me know when you have read so I can delete the file.

[griffin]

I've got it.

[griffin]

I'm actually fading pretty quickly right now. It has been a very long day. In the meantime, I'm going to mention this topic to someone quite experienced in such matters. I'll securely pass him the output file. I'll check back as I can in travel tomorrow. Sorry for delay. I just don't want to cause harm due to being exhausted.

[geekstrada]

Sounds good

[geekstrada]

I truly appreciate the help!

[griffin]

You are quite welcome! Thanks much for understanding. I've passed my notes to him.

[_az]

At the risk of possibly posting inaccurate information (because I'm not a macOS user and the only developer on the team who uses macOS won't be able to answer this for a few days) ...

... I supect that Certbot's Apache plugin, by default, only supports the Apache2 installation that comes with macOS. Apache2 from Homebrew probably (?) won't work out of the box.

You might be able to get this to work by overriding some path flags (certbot --help apache), but otherwise, you may need to use certonly.

[geekstrada]

Thanks @_az I'll look into it!

[geekstrada]

And if the the team member who does use macOS chimes in, that would be great

[rg305]

I don't per se use macOS but I have seen a few posts relating to them.
And I've found that one common quick fix to them is to use --webroot
Or as @_az suggested:

@_az

you may need to use certonly

[geekstrada]

Hi @_az just wanted to follow up. On a whim I tried using certbot on an Intel Mac with Homebrew-installed httpd. It worked just fine. I think the issue is with Homebrew using /opt for M1 Macs, whereas before it had always used /usr/local for Intel Macs. I think it is going to be an issue with certbot --nginx on M1 Macs as well with Homebrew-installed Nginx.

[_az]

Thanks for following up. I rented an M1 Mac from Scaleway and tried using Certbot (from Homebrew) with httpd (also from Homebrew).

These are my notes below. Clearly, there are a lot of manual interventions to make it work cleanly. We should definitely make it clear in our documentation about what the situation is with macOS (Intel vs M1) and Apache (Homebrew vs Apple).

---

macOS M1 Certbot + httpd notes

• On M1 macOS, Apache from Homebrew (the httpd package) installs its configuration root to /opt/homebrew/etc/httpd/.

  • This is not where the Certbot Apache plugin looks (/etc/apache2 and /etc/apache2/other for --apache-server-root and --apache-vhost-root, respectively).
  • Therefore, using the Certbot Apache plugin on M1 Macs, will require the user to override these flags.

• By default, httpd from brew listens on 8080 and 8443.

  • Therefore, users must update these ports to 80 and 443, in most cases.

• The mod_ssl + shmcb + rewrite modules are not loaded by default.

  • The Certbot Apache plugin doesn't know how to enable modules on macOS.
  • Therefore, the user must uncomment these three modules in the default /opt/homebrew/etc/httpd/httpd.conf file, before trying
    to use the Certbot Apache plugin.

• The certificate referenced in the default HTTPS virtualhost in /opt/homebrew/etc/httpd/extra/httpd-ssl.conf doesn't exist.

  • Therefore, the user should not uncomment this file, unless they obtain a certificate separately somehow.

• There are no default port 80 virtual hosts.

  • Therefore, the user should create a port 80 vhost in /opt/homebrew/etc/httpd/extra/httpd-vhosts.conf, for example. (This file include will need to be uncommented first).

• Certbot needs to be invoked with the relevant --apache flags to make it work with the particulars of Homebrew's httpd package.

  • This can be done in a permanent fashion by adding the following to /etc/letsencrypt/cli.ini:

    apache-server-root = /opt/homebrew/etc/httpd/
    apache-vhost-root = /opt/homebrew/etc/httpd/extra/
    

  • Then Certbot can be invoked with:

    sudo certbot --apache -d example.com
    

• It's not clear to me whether the best way to use Certbot on M1 macOS is with sudo or not. It seems like privileged ports are allowed on unprivileged users, but the Certbot security model seems to assume root privileges for e.g. key material, so sudo is probably the way to go? On the other hand, brew services complains about some things when running as root. I really have no idea what's idiomatic here:

   Warning: httpd must be run as non-root to start at user login!

[geekstrada]

I actually ended up using --standalone on my M1 Mac to get it working, partially following (for my needs) a guide online with specific reference to M1 vs. Intel. Here's the link in case it helps anyone: https://getgrav.org/blog/macos-monterey-apache-multiple-php-versions