Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
My domain is:
arch.xnat.vanderbilt.edu
I ran this command:
certbot --apache
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Which names would you like to activate HTTPS for?
1: arch.xnat.vanderbilt.edu
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for arch.xnat.vanderbilt.edu
Enabled Apache rewrite module
Waiting for verification...
Challenge failed for domain arch.xnat.vanderbilt.edu
http-01 challenge for arch.xnat.vanderbilt.edu
Cleaning up challenges
Some challenges have failed.
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you're using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.
My web server is (include version):
Apache/2.4.41 (Ubuntu)
The operating system my web server runs on is (include version):
Ubuntu 20.04.5 LTS
My hosting provider, if applicable, is:
vanderbilt university
I can login to a root shell on my machine (yes or no, or I don't know):
yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.40.0
This is very good advice and the easiest way to do that with Certbot would be to add --dry-run to the command line arguments.
You could try to see if the server is really overloaded by running something like top or uptime to look at the load statistics.
I guess that error could also be caused by somewhat unusual firewall behavior, if a web application firewall is trying to examine all inbound connections to your web server and then somehow gives up on (or abandons) some connections without making a firm decision about whether they will be permitted or not?
It could also be caused by some kind of load balancer or proxy that's misconfigured, or any of these systems with some kind of logging configuration that somehow doesn't complete its logging task in time.
This error is a relatively unusual one for stock Apache setups that are connected directly to the Internet and that can receive inbound connections.
We haven't seen one of these in a while but you are almost certainly affected by a Palo Alto Networks brand firewall. We started seeing a lot of these over a year ago but not as many lately.
You should talk to your network admins and have them change the Application Rule for "ACME protocol".
See the two curl tests below. They each are a sample of the HTTP Challenge request sent by the Let's Encrypt server to your server to validate the domain. The second one with the -A user-agent the same as Let's Encrypt uses fails with a timeout. If you omit that user-agent it works as expected (a 404 not found).
You should show this to your network admins.
curl -I -m10 arch.xnat.vanderbilt.edu/.well-known/acme-challenge/Test123
HTTP/1.1 404 Not Found
Date: Fri, 09 Jun 2023 03:36:50 GMT
Server: Apache/2.4.41 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1
curl -I -m10 arch.xnat.vanderbilt.edu/.well-known/acme-challenge/Test123 -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received
You should check with your academic computing department. They likely have firewalls and policies in place regarding LetsEncrypt domains. They may have a provisioning system in place too. LetsEncrypt certificates are rate limited by registered domain. Universities often block "open" registration to prevent random registrations from competing with official or scheduled services.
I saw you mention it likely being PaloAlto and there being an admin firewall rule to change.
My point is there is likely an Academic Computing policy that flat out prohibits open registrations, and requires some sort of allocation/coordination with the IT department.
I get the same results to my test curls now as I did in post #4
They should double-check the rule. The default rule setup by Palo Alto was to block ACME challenges. The fix was to disable that block which then allows the acme protocol. But, we have seen different symptoms for various Palo Alto devices so it's possible the rule works slightly different between them.
Your network people should be able to use the same curl examples to trace the inbound request and see where it is blocked. They just need to make the request outside that network so the request passes through that device.
