Ok - that’s a new puzzle then I guess. The server is running fine (I’m sitting here remotely accessing it via HTTPS). Also the initial installation of certificates appears to have run (and verified) without failures. What’s special about renewal (or dry-run renewal) that would cause it to fail when the server is otherwise accessible globally?
My theory was that multiple simultaneous validations performed by the staging server cause your server to drop network connections. That’s why I wanted you to try dry-run only a single certificate.
And it seems reproducible.
I can run on my own laptop:
certbot certonly -a webroot -w /tmp -d grandhighwizard.net -d darro.ch \
-d robin.darroch.id.au -d webdav.darroch.id.au -d www.darroch.id.au \
-d www.grandhighwizard.net -d www.robin.darroch.id.au \
--dry-run
and the result is mass timeouts.
If I get rid of the --dry-run (in other words, change from the staging to the live server), it changes from timeouts to the expected error (unauthorized/Invalid response).
If I keep --dry-run but reduce the domain count to only one domain, for me, it also changes to the correct error, the great majority of the time. To me, that confirms the network as being the cause.
There’s an important difference between the live and staging Let’s Encrypt servers.
The live server only makes one request per domain.
The staging server makes more like ~4 per domain.
4 * 8 domains = upto 32 requests simultaneously, enough for your internet connection or Apache to start timing out.
Combine this with your Apache logs complaining earlier about your MaxClients setting, add on that whatever network limits or firewalls macOS uses, and we have a hypothesis for why your dry-runs are timing out.
Interesting - I wonder if the router’s firewall is being tripped by the bulk request. So then the question is, provided initial installation threw no errors, should I just forget about the dry run testing and assume real cert renewals will work? I can’t seem to convince it to do a dry run on a single domain, because they’re all covered by the one cert.
Not ideal, is it? If you did choose to do this, you would at least have a month of time to address any issues, since Certbot does begin renewal attempts 30 days from expiry.
@jsha this might be this an interesting potential issue that could crop up if multi-VA was adopted in the future ...
DING! We have a winner!
Went to my router’s firewall settings, turned off DOS protection, ran the dry-run again:
MiniMe:Sites robin$ sudo certbot renew --dry-run
Saving debug log to /opt/local/var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /opt/local/etc/letsencrypt/renewal/darro.ch.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for darro.ch
http-01 challenge for robin.darroch.id.au
http-01 challenge for webdav.darroch.id.au
http-01 challenge for www.darroch.id.au
http-01 challenge for www.grandhighwizard.net
http-01 challenge for www.robin.darroch.id.au
Waiting for verification...
Cleaning up challenges
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/opt/local/etc/letsencrypt/live/darro.ch/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)
Congratulations, all renewals succeeded. The following certs have been renewed:
/opt/local/etc/letsencrypt/live/darro.ch/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
So it seems the router was detecting the challenge requests as a denial-of-service attack and sending them to the bit bucket.
For the sake of interest, the router is an ASUS RT-AC88U, and it is the built-in firewall that’s in use. It needs to be running generally, because otherwise I’ve found our connection being used for DNS reflection attacks… but at least now we’ve confirmed a fix for the dry-run problem.
In short, the solution to the original problem is:
brew uninstall certbot- install MacPorts - https://www.macports.org/install.php
sudo port install certbot-apache- edit
/opt/local/etc/letsencrypt/options-ssl-apache.conf, comment out the lineSSLCompression off sudo certbot --apache
Everything else turns out to be peripheral to the process.
2 Likes
It occurred to me that marking the other post as the solution could lead folks to turning off firewall protection, when that's just a workaround for the dry-run problem on multiple domains that I was encountering. So here's just the solution, marked as such.
2 Likes
Good point, @_az! We will definitely keep this in mind as we get closer to multi-VA rollout. I’ll be curious to see how widespread it is.
Let me know if there’s something you’d like me to test for you, given that I seem to have ended up with a fairly reliable test-bed for the problem. I presume any Asus router (or indeed any router) using the same built-in firewall would trigger the same behaviour under comparable circumstances.
1 Like
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.