Authentication failing, but apache log shows 200 status codes

[deadtom]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: email.missoula.lib.mt.us

I ran this command: certbot renew --cert-name email.missoula.lib.mt.us -v --dry-run

It produced this output: Certbot error - Pastebin.com

My web server is (include version): Apache/2.4.38

The operating system my web server runs on is (include version): Debian 10

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.1.0

I uploaded the full, verbose log to pastebin, because it's enormous.

Certbot is failing with the "Timeout during connect (likely firewall problem)" error, but the apache logs show it connecting with a 200 status code:

/var/log/apache2/email-access.log:34.221.113.30 - - [03/Jan/2023:10:05:39 -0700] "GET /.well-known/acme-challenge/A93pnibf93G96bNJpSTY3SAz3YWZGMT5Z2dmom0juj4 HTTP/1.1" 200 310 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
/var/log/apache2/email-access.log:52.15.97.187 - - [03/Jan/2023:10:05:39 -0700] "GET /.well-known/acme-challenge/A93pnibf93G96bNJpSTY3SAz3YWZGMT5Z2dmom0juj4 HTTP/1.1" 200 310 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

What could be the issue here?

Thanks.

[Bruce5051]

Hello @deadtom, welcome to the Let's Encrypt community.

Using Let's Debug I see there are ERRORs https://letsdebug.net/email.missoula.lib.mt.us/1321829

And some supplemental information

$ curl -Ii http://email.missoula.lib.mt.us/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Date: Tue, 03 Jan 2023 17:46:15 GMT
Server: Apache/2.4.38 (Debian)
Content-Type: text/html; charset=iso-8859-1

$ nmap email.missoula.lib.mt.us
Starting Nmap 7.91 ( https://nmap.org ) at 2023-01-03 09:46 PST
Nmap scan report for email.missoula.lib.mt.us (216.14.254.83)
Host is up (0.044s latency).
Not shown: 995 filtered ports
PORT     STATE  SERVICE
80/tcp   open   http
113/tcp  closed ident
443/tcp  open   https
8008/tcp open   http
8010/tcp open   xmpp

Nmap done: 1 IP address (1 host up) scanned in 4.81 seconds

[deadtom]

Does certbot leave the file there after the renew attempt?

The apache log shows the letsencrypt server getting the file fine, unless I'm misunderstanding the status codes.

[Bruce5051]

Why do you say

@deadtom

The apache log

Using SSL Server Test (Powered by Qualys SSL Labs) the results at the bottom of the page SSL Server Test: email.missoula.lib.mt.us (Powered by Qualys SSL Labs) shows HTTP server signature Microsoft-IIS/8.5

[deadtom]

I'm not sure what's happening here. It is apache. Here's you trying to get sometestfile,
Right out of the apache log:

98.246.255.230 - - [03/Jan/2023:10:46:15 -0700] "HEAD /.well-known/acme-challenge/sometestfile HTTP/1.1" 404 140 "-" "curl/7.79.0"
98.246.255.230 - - [03/Jan/2023:10:48:33 -0700] "HEAD /.well-known/acme-challenge/sometestfile HTTP/1.1" 404 140 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

I'm using our apache server to renew the certificate, and then I manually move it over and install it on exchange. This has been working for years, until a few weeks ago.

[Bruce5051]

OK. Kindly wait for more knowledgeable Let's Encrypt community volunteers to assist.
And yes that is my present Public facing IPv4 Address.

[Osiris]

Looking at the log files, there are only 2 secondary validation attempts visible (both Amazon IP addresses). I'm not seeing the primary validation servers IP address: I believe that one doesn't originate from Amazon. And the primary validation must succeed.

So it seems somehow the validation attempt from the primary data center in the US isn't able to connect to your server.

[deadtom]

Ah, I see. So I wonder if our Fortigate is blocking something. I'll look into that.

Thanks!

[rg305]

@deadtom

What could be the issue here?

   Detail: 216.14.254.83: Fetching
   http://email.missoula.lib.mt.us/.well-known/acme-challenge/iqL6Ib9SYnS9wNkqGlirwKmeT7JP2SVMtv4UXTUt0PU:
   Timeout during connect (likely firewall problem)

[deadtom]

I was confused because I was still seeing connections in the apache log, getting through the firewall fine. I was not aware of the primary vs secondary server issue, so now it makes sense.