openSUSE15.6 apache2-2.4.58-150600 2ndary vaidation

Help
1

SO how do I know my unexpected 2ndary validation issue for my new server setup is the issued
announced in April or my previous Palo Alto issue

PALO ALTO :

versus
UNEXPECTED failure:

My domain is:
dna.engr.latech.edu

I ran this command:
certbot --apache

It produced this output:
dna:/srv/www/htdocs # /usr/bin/certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
ssl_module is statically linked but --apache-bin is missing; not disabling session tickets.

Which names would you like to activate HTTPS for?
We recommend selecting either all domains, or all domains in a VirtualHost/server block.

1: dna.engr.latech.edu

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel):
Requesting a certificate for dna.engr.latech.edu

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: dna.engr.latech.edu
Type: connection
Detail: During secondary validation: 138.47.29.6: Fetching http://dna.engr.latech.edu/.well-known/acme-challenge/oc9FpTSIYnBmHZYkjF28r85H2ZP-A3QjN7DrIuVkmaY: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
dna:/srv/www/htdocs #

My web server is (include version):
apache2-2.4.58-150600.5.11.1.x86_64

The operating system my web server runs on is (include version):
dna:/srv/www/htdocs # cat /etc/SUSE-brand
openSUSE
VERSION = 15.6

My hosting provider, if applicable, is:
Louisiana Tech University

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no.. just the shell

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 2.11.0

2 Likes
2

Because of the different error message.

Your Palo Alto error was "reset by peer".

This says "Secondary validation" failed. The Secondary centers are not used until the Primary center passes. And, if Palo Alto problem the Primary would fail so the sequence would not get to the secondary centers for them to issue a Secondary failure.

Almost certainly is a geographic based firewall. Given your domain name I'd guess you are blocking non-USA requests. You may have been doing that before but were getting lucky that enough USA-only auth centers worked. Now it is essential that at least one non-USA authorization succeeds.

6 Likes
3

I concur with @MikeMcQ. Having worked with Palo Alto FW rules on this very type of problem, I'm not surprised.

4 Likes
4

I dug into the apache2 and letsencrypt logs and found some 404 errors in access_logs suggesting the request it getting thru but that it's not getting the file frsom my server.
I put a dummy file at
http://dna.engr.latech.edu/.well-known/acme-challenge/Test_File
that's accessible to me from outside my 138.47 IP space.

Here are the latest 404 errors
grep " 404 " /var/log/apache2/access_log | grep -i well

17.58.57.102 - - [02/Aug/2024:16:45:34 -0500] "GET /.well-known/acme-challenge/oc9FpTSIYnBmHZYkjF28r85H2ZP-A3QjN7DrIuVkmaY: HTTP/1.1" 404 973 "-" "AppleNewsBo
t"
17.58.62.20 - - [02/Aug/2024:16:48:39 -0500] "GET /.well-known/acme-challenge/oc9FpTSIYnBmHZYkjF28r85H2ZP-A3QjN7DrIuVkmaY: HTTP/1.1" 404 973 "-" "AppleNewsBot
"
but other messages from other IP addresses make it through

dna:/srv/www/htdocs/.well-known # grep MJ0 //var/log/apache2/access_log

23.178.112.210 - - [02/Aug/2024:16:39:24 -0500] "GET /.well-known/acme-challenge/MJ0ywEr-T1wVx6PJuivf3_6JjJSSfRvJUNBzY4NSpEU HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.217.14.161 - - [02/Aug/2024:16:39:25 -0500] "GET /.well-known/acme-challenge/MJ0ywEr-T1wVx6PJuivf3_6JjJSSfRvJUNBzY4NSpEU HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
34.219.143.136 - - [02/Aug/2024:16:39:25 -0500] "GET /.well-known/acme-challenge/MJ0ywEr-T1wVx6PJuivf3_6JjJSSfRvJUNBzY4NSpEU HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

dna:/srv/www/htdocs/.well-known # grep MJ0 /var/log/apache2/access_log

23.178.112.210 - - [02/Aug/2024:16:39:24 -0500] "GET /.well-known/acme-challenge/MJ0ywEr-T1wVx6PJuivf3_6JjJSSfRvJUNBzY4NSpEU HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.217.14.161 - - [02/Aug/2024:16:39:25 -0500] "GET /.well-known/acme-challenge/MJ0ywEr-T1wVx6PJuivf3_6JjJSSfRvJUNBzY4NSpEU HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
34.219.143.136 - - [02/Aug/2024:16:39:25 -0500] "GET /.well-known/acme-challenge/MJ0ywEr-T1wVx6PJuivf3_6JjJSSfRvJUNBzY4NSpEU HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

5

There are only 3 successful requests. There will be 4 or more likely 5 successful ones if all are getting through. I didn't map each of those IP but I would bet they are all US based IP

The reason for the 2 sets with the same IP is likely due to you redirecting the original HTTP request to HTTPS Nevermind, I just realized you showed the same info twice. The above comment about 4-5 requests still holds.

those "Apple Newsbot" requests are nonsense from bots

4 Likes
6

Thanks ya'll.
if I understand it seem the university is filtering some ip addresses which certbot is using to validate my system. I have an email to the IT guys asking if this is indeed the case.

What are my certificate options if they won't allow access from certain countries?

Details of what I think is happening are below: agrees w/ above recommendations from ya'l.

w/r/t Apple Newsbot... got it what I really want to look for in apache log seems to be

grep "Let's Encrypt validation server" /var/log/apache2/access_log

The apache logs reveals 3x successful checks on the challenge file (certbot and all are working!)

if other domains are being filtered they will never show in my apache logs so all I'll see is the Certbot message in /var/log/letsencrypt/letsencrypt.log

meaning the university is filtering some IP addresses ... so I get the following errors.

........
Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:

apache logs

23.178.112.210 - - [02/Aug/2024:16:39:24 -0500] "GET /.well-known/acme-challenge/MJ0ywEr-T1wVx6PJuivf3_6JjJSSfRvJUNBzY4NSpEU HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.217.14.161 - - [02/Aug/2024:16:39:25 -0500] "GET /.well-known/acme-challenge/MJ0ywEr-T1wVx6PJuivf3_6JjJSSfRvJUNBzY4NSpEU HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
34.219.143.136 - - [02/Aug/2024:16:39:25 -0500] "GET /.well-known/acme-challenge/MJ0ywEr-T1wVx6PJuivf3_6JjJSSfRvJUNBzY4NSpEU HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

1 Like
7

To be a bit pedantic, certbot is just asking Let's Encrypt to validate, and it's Let's Encrypt which is actually trying to connect from multiple places to ensure that you actually control the domain name as seen by everywhere on the Internet.

There should be five currently, with more likely to happen in the future.

In order to prove that you control the name, you need to prove control as seen by everywhere on the Internet. Usually, even networks with strict controls have their DNS server publicly available, so you may be able to set up DNS-01. But that would require your domain administrators to delegate some level of control over the name to you to be able to fulfill the challenge, and it can be harder to set up.

You may want to look through this FAQ, I hope it answers a lot of questions about multi-perspective validation and how to deal with it:

Another workaround to try if you're just trying to get things working is to try some other CAs, like Buypass Go or ZeroSSL. You can often just change the --server argument in Certbot, though some CAs require registering a separate account first. But all CAs will be checking from multiple locations if they aren't already, so this really isn't going to address your root problem.

4 Likes
8

Block on behavior, not location. Is there not someone in the EU that wants to check out the curriculum... or Japan? Baffles me. Just thought I'd give my 2¢ to aid your position.

No offense intended, my friend.

3 Likes
9

@petercooperjr: thanks for details. I have complete control over the computer but IT restricts network. Let's encrypt w/ certbot and opensuse15.4 worked on this machine earlier this year.

@rip: not sure I fully appreciate comment... the Debug doesn't work b/c
http://dna.engr.latech.edu/.well-known/acme-challenge/letsdebug-test" does not exist
The acme-challenge directory is empty.

see above. I have complete control over machine so maybe I did something boneheaded in setup of the server/security... that's causing problems in addition to what IT is restricting

1 Like
10

using proton vpn on my phone suggests that IP addresses outside US are being filtered from accessing my server. Until I get IT to open things up it seems Let's Encrypt won't be able to validate my system.

1 Like
11

No, that first Let's Debug test failed because of the "timeout" trying to reach that URL. It just so happens Let's Debug runs on a server in Europe. Its first test is from its own server and looks for that URL. A successful result is an HTTP 404 Not Found (or a 200 OK if you create that file of course but not required). This part of the test sequence often helps identify common comms problems like "timeouts". This part of initial tests also can identify config issues like multiple IP that point to different servers.

A later part of the Let's Debug test sequence uses the Let's Encrypt Staging System. And, that showed a "404". Why not a "timeout" here too? Because the LE Auth process first checks from the Primary validation center (which is the USA today). If that fails then no Secondary center is checked (some of which are non-USA). To the LE system (production and staging) the "404" is a failure. LE needs to see the correct token to succeed. In contrast, the "404" here is what Let's Debug expects from this staging test so it consider this part successful. Why? Because it knows it can't create the correct token on your machine for your server to return it. But if Let's Debug sees a "timeout" from this staging system or any number of other errors it would say it failed.

The Let's Debug system is often very useful. But, for certain errors like your Secondary Validation due to geo-blocking it takes a lot of insider knowledge to interpret the results. You just seemed skilled enough to ask a good question so thought you might find this interesting. Nevermind if it is not :slight_smile: Your problem is pretty clearly geo-blocking non-USA.

See the Verbose output for details of above explanation

5 Likes
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.