Connection refused status 400, Apache mod_md and mod_proxy, fail2ban

Help
1

I suspect my problem lies in my defences against web scrapers, unfortunately. Is there a specific user agent, IP address range or ASN I could put on an allow-list? Looking at the error message in my Apache error.log below, can I tell whether the bot is being blocked by the firewall (unable to connect) or whether it was getting through to Apache and what status code it saw? In my case I'd be interested in knowing if it sees a 410 GONE, for example, since I respond to a lot of requests by bots using that.

My domain is:

search.transjovian.org

This is a new subdomain. The setup for transjovian.org without search.transjovian.org worked.

I ran this command:

I'm using mod_md

At the top of the site conf file:

MDomain transjovian.org search.transjovian.org
MDCertificateAgreement accepted

The site configuration itself, where I exempt /.well-known and I created a file /home/xobaque/public/.well-known/test which I was able to fetch using curl (getting a 200 OK response). So I know that /.well-known/ isn't proxied.

<VirtualHost *:80>
    ServerName search.transjovian.org
    Include conf-enabled/blocklist.conf
    DocumentRoot /home/xobaque/public
    # xobaque
    ProxyPass "/.well-known" !
    ProxyPass "/" "unix:/run/xobaque/xobaque.sock|http://localhost/"
    DocumentRoot /home/xobaque/public
    <Directory /home/xobaque/public>
        Require all granted
    </Directory>
</VirtualHost>
<VirtualHost *:443>
    ServerAdmin alex@alexschroeder.ch
    ServerName search.transjovian.org
    SSLEngine on
    Include conf-enabled/blocklist.conf
    # xobaque
    ProxyPass "/.well-known" !
    ProxyPass "/" "unix:/run/xobaque/xobaque.sock|http://localhost/"
    DocumentRoot /home/xobaque/public
    <Directory /home/xobaque/public>
        Require all granted
    </Directory>
</VirtualHost>

It produced this output:

In the Apache error log:

[Sat Aug 16 13:41:21.587145 2025] [md:error] [pid 178047:tid 178048] ACME server authz: challenge 'invalid' for search.transjovian.org at https://acme-v02.api.letsencrypt.org/acme/authz/825400867/569744845157. Exact response was: {"identifier":{"type":"dns","value":"search.transjovian.org"},"status":"invalid","expires":"2025-08-23T11:41:19Z","challenges":[{"type":"http-01","url":"https://acme-v02.api.letsencrypt.org/acme/chall/825400867/569744845157/Q8HC1g","status":"invalid","validated":"2025-08-16T11:41:19Z","error":{"type":"urn:ietf:params:acme:error:connection","detail":"During secondary validation: 178.209.50.237: Fetching http://search.transjovian.org/.well-known/acme-challenge/O6eEWsTQ2qdNJzyb6r4CwOJDUaDo-b9A780Hk3F_elo: Connection refused","status":400},"token":"O6eEWsTQ2qdNJzyb6r4CwOJDUaDo-b9A780Hk3F_elo","validationRecord":[{"url":"http://search.transjovian.org/.well-known/acme-challenge/O6eEWsTQ2qdNJzyb6r4CwOJDUaDo-b9A780Hk3F_elo","hostname":"search.transjovian.org","port":"80","addressesResolved":["178.209.50.237","2a02:418:6a04:178:209:50:237:1"],"addressUsed":"2a02:418:6a04:178:209:50:237:1"}]}]}
[Sat Aug 16 13:41:21.588664 2025] [md:error] [pid 178047:tid 178048] (22)Invalid argument: md[transjovian.org] while[Monitoring challenge status for transjovian.org] detail[domain authorization for search.transjovian.org failed, CA considers answer to challenge invalid.]
[Sat Aug 16 13:41:21.596441 2025] [md:error] [pid 178047:tid 178048] (22)Invalid argument: AH10056: processing transjovian.org: Error waiting on domain names to be validated

My web server is (include version):

http://localhost/server-status reports:

Apache/2.4.65 (Debian) OpenSSL/3.5.1

This is Apache from Debian 13 (Trixie).

The operating system my web server runs on is (include version):

uname -a reports:

Linux sibirocobombus 6.12.38+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.38-1 (2025-07-16) x86_64 GNU/Linux

My hosting provider, if applicable, is:

edis.at

I can login to a root shell on my machine (yes or no, or I don't know):

yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

I don't know how to determine the version of mod_md that ships with Apache but I guess it's not relevant.

2

That's a good guess :slight_smile:

Why don't you redirect HTTP to HTTPS? Then leave the blocklist enabled on HTTPS (port 443) but HTTP (port 80) remains open for the HTTP Challenge requests inbound from Let's Encrypt.

Scrapers still won't be able to scrape

Error from your log

      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "During secondary validation: 178.209.50.237: Fetching http://search.transjovian.org/.well-known/acme-challenge/O6eEWsTQ2qdNJzyb6r4CwOJDUaDo-b9A780Hk3F_elo: Connection refused",
        "status": 400

You can probably see the inbound HTTP requests that succeed in your Apache access_log.

In the error above it was a Secondary Let's Encrypt location that failed to connect. That means the Primary worked and possibly other secondaries. But, too many secondary requests failed which is why the cert was not issued.

3 Likes
3

I do that for most of my sites but recently I've started a change of heart, basically in support of retro-computing fans… :sweat_smile:

Ah, you are correct. That request did get a 200 OK response:

search.transjovian.org:80 2600:3000:2710:200::86 - - [16/Aug/2025:13:41:20 +0200] "GET /.well-known/acme-challenge/O6eEWsTQ2qdNJzyb6r4CwOJDUaDo-b9A780Hk3F_elo HTTP/1.1" 200 198 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 349

You seem to imply that there are multiple secondary requests that should come after that. Interesting, I didn't know that.

So now I'm looking at the longer log (grep for "^search.*letsencrypt") and seeing that the number of requests that get through seems to vary. I see one requests at 13:41, three requests at 14:18, one request at 14:28.

search.transjovian.org:80 2600:3000:2710:200::86 - - [16/Aug/2025:13:41:20 +0200] "GET /.well-known/acme-challenge/O6eEWsTQ2qdNJzyb6r4CwOJDUaDo-b9A780Hk3F_elo HTTP/1.1" 200 198 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 349
search.transjovian.org:80 2600:3000:2710:200::84 - - [16/Aug/2025:14:18:15 +0200] "GET /.well-known/acme-challenge/sCrgjjlkpTV0ShtuTa0LaY0lwBoXHAwgHKGI0JhokWM HTTP/1.1" 200 198 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 854
search.transjovian.org:80 2600:3000:2710:200::81 - - [16/Aug/2025:14:18:24 +0200] "GET /.well-known/acme-challenge/1NE8Jf1fNgUsmxKnmBfR9RzGU6K573Iup0MXf1psBn4 HTTP/1.1" 200 198 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 282
search.transjovian.org:80 2600:3000:2710:200::82 - - [16/Aug/2025:14:18:36 +0200] "GET /.well-known/acme-challenge/J2KBjpNj3JHC6FNSi1dyy-IT2zcO9qPhqZGL-mUiw6Y HTTP/1.1" 200 198 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 303
search.transjovian.org:80 2600:3000:2710:200::82 - - [16/Aug/2025:14:28:45 +0200] "GET /.well-known/acme-challenge/R_m1xK_i26RdMDTwC0QyyOgYjlwQ6r1n1h-ipT6Az9U HTTP/1.1" 200 198 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 181

If I understand you correctly, some of the requests get through, others do not, and for the challenge to succeed more than three must get through. And since I'm not seeing more in my log, it's the firewall that's blocking them.

I'll see if I can momentarily clear all the fail2ban rules out. :scream:

4

I wasn't implying it. I was stating it :slight_smile: LE currently has one primary and 4 secondary centers. This is likely to change over time perhaps with little to no notice. It should not matter to your design.

Currently the primary must succeed and all but one of the secondaries. Again, this may change. When using an HTTP Challenge you should accept /.well-known/acme-challenge requests from anywhere.

3 Likes
5

Sadly, unbanning everybody I had in a fail2ban jail didn't work.

# fail2ban-client unban --all
663

I still have an ipset with one banned address range belonging to Facebook and one of the fail2ban jails quickly gained four entries belonging to Google Cloud Platform, but I'm assuming they don't host the secondaries.

Even with nearly all the bans lifted, I still have the secondary validation problem:

      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "During secondary validation: 178.209.50.237: Fetching http://search.transjovian.org/.well-known/acme-challenge/16EQKlkY43f-G0N1uKSiqaiXcM17X1_pj2Mz6--eSTw: Connection refused",
        "status": 400
      },
6

Are they still lifted? Because I cannot even connect to your "home" page from my own test server (in AWS US East Coast region). The LE secondary centers are also AWS although different regions.

curl -i http://search.transjovian.org
curl: (7) Failed to connect to search.transjovian.org port 80 after 328 ms: 
Connection refused

curl -i https://search.transjovian.org
curl: (7) Failed to connect to search.transjovian.org port 443 after 253 ms: 
Connection refused

curl -i http://search.transjovian.org/.well-known/acme-challenge/Test404
curl: (7) Failed to connect to search.transjovian.org port 80 after 252 ms: 
Connection refused
3 Likes
7

Just fyi. Let's Encrypt recommends all sites use HTTPS to serve content.

See: Why All Websites Should Use HTTPS - Let's Encrypt

And: Best Practice - Keep Port 80 Open - Let's Encrypt

2 Likes
8

Oof! Good question. fail2ban-client banned still lists just four IP addresses. But when I run iptables --list some of the chains corresponding to fail2ban jails list a gazillion entries with reject-with icmp-port-unreachable. Ouch.

I ran iptables --flush and ip6tables --flush on a few f2b-* chains. Let's see whether that helps.

9

Thanks for checking this. This was indeed the problem.

10

You may want to read through this, describing a bit more about how and why Let's Encrypt checks from multiple places around the world to ensure that you actually have control over your domain name:

2 Likes
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.