Someone is watching you

crashulater · December 22, 2024, 4:09pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: hork.com

I ran this command: certbot

It produced this output: certificates successfully renewd

My web server is (include version): apache httpd-2.4.54

The operating system my web server runs on is (include version): Fedora-35

My hosting provider, if applicable, is: self

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot-2.9.0-1.fc38

No immediate help needed here, as this is more of a "head-up" than a help request, but your insightful comments are always welcome. My web server renewed its certificates last night. That went fine, as usual. It is a multi step process here: 1) stop the watch-dog; 2) lower the firewall; 3) renew (2) certs using certbot; 4) raise the firewall; 5) start the watch dog. The whole process takes just 20 seconds, most of which is due to a sleep between the first and second cert renewal. The renewal is scheduled for regular intervals of course, but take place at a randomize time. I was surprised therefore that my sites came under significant attack, while the firewall was down. That looked like this:

 23.178.112.215 22:35:03 GET /.well-known/acme-challenge/onyD_7WiZSSEtuXNAKDkMEnKUXPOZEqEjBNvYr_xw74
 23.178.112.213 22:35:03 GET /.well-known/acme-challenge/-OiooQfuIXl_MkBEcgPzmmEOoouH3TJpVny-bU8Bo8g
   18.218.84.24 22:35:03 GET /.well-known/acme-challenge/-OiooQfuIXl_MkBEcgPzmmEOoouH3TJpVny-bU8Bo8g
 54.186.249.114 22:35:03 GET /.well-known/acme-challenge/onyD_7WiZSSEtuXNAKDkMEnKUXPOZEqEjBNvYr_xw74
  44.244.43.197 22:35:03 GET /.well-known/acme-challenge/-OiooQfuIXl_MkBEcgPzmmEOoouH3TJpVny-bU8Bo8g
   13.60.19.125 22:35:03 GET /.well-known/acme-challenge/onyD_7WiZSSEtuXNAKDkMEnKUXPOZEqEjBNvYr_xw74
  13.212.61.241 22:35:03 GET /.well-known/acme-challenge/-OiooQfuIXl_MkBEcgPzmmEOoouH3TJpVny-bU8Bo8g
  13.212.61.241 22:35:04 GET /.well-known/acme-challenge/onyD_7WiZSSEtuXNAKDkMEnKUXPOZEqEjBNvYr_xw74
   3.15.185.168 22:35:04 GET /.well-known/acme-challenge/-OiooQfuIXl_MkBEcgPzmmEOoouH3TJpVny-bU8Bo8g
 13.215.227.184 22:35:04 GET /.well-known/acme-challenge/-OiooQfuIXl_MkBEcgPzmmEOoouH3TJpVny-bU8Bo8g
 23.178.112.107 22:35:16 GET /.well-known/acme-challenge/G8rKUTe0Xz-hMWJ3PQPZAIRBvBeIcHpnZIerY_78zSY
 18.219.223.145 22:35:16 GET /.well-known/acme-challenge/G8rKUTe0Xz-hMWJ3PQPZAIRBvBeIcHpnZIerY_78zSY
  44.244.43.197 22:35:16 GET /.well-known/acme-challenge/G8rKUTe0Xz-hMWJ3PQPZAIRBvBeIcHpnZIerY_78zSY
   16.16.75.149 22:35:16 GET /.well-known/acme-challenge/G8rKUTe0Xz-hMWJ3PQPZAIRBvBeIcHpnZIerY_78zSY
 13.215.227.184 22:35:17 GET /.well-known/acme-challenge/G8rKUTe0Xz-hMWJ3PQPZAIRBvBeIcHpnZIerY_78zSY
   138.68.86.32 22:35:17 "\x16\x03\x01"
 23.178.112.107 22:35:17 GET /.well-known/acme-challenge/oPYN87nuyLL90g3NPxnukDhjiOdTTsuHMMbAW9Kvtlw
    142.93.0.66 22:35:17 "\x16\x03\x01"
 138.197.191.87 22:35:17 GET /
    142.93.0.66 22:35:17 GET /
  165.227.84.14 22:35:17 GET /
    142.93.0.66 22:35:17 GET /
   138.68.86.32 22:35:17 GET /
    142.93.0.66 22:35:17 GET /actuator/env
    142.93.0.66 22:35:17 GET /server
    142.93.0.66 22:35:17 GET /.vscode/sftp.json
    142.93.0.66 22:35:17 GET /about
   138.68.86.32 22:35:17 GET /
    142.93.0.66 22:35:17 GET /debug/default/view?panel=config
    142.93.0.66 22:35:18 GET /v2/_catalog
    142.93.0.66 22:35:18 GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application
    142.93.0.66 22:35:18 GET /server-status
   138.68.86.32 22:35:18 GET /actuator/env
    142.93.0.66 22:35:18 GET /login.action
    142.93.0.66 22:35:18 GET /_all_dbs
    142.93.0.66 22:35:18 GET /.DS_Store
  165.227.84.14 22:35:18 GET /
    142.93.0.66 22:35:18 GET /.env
  44.244.43.197 22:35:18 GET /.well-known/acme-challenge/oPYN87nuyLL90g3NPxnukDhjiOdTTsuHMMbAW9Kvtlw
    142.93.0.66 22:35:18 GET /.git/config
  165.227.84.14 22:35:18 GET /actuator/env
  51.20.133.182 22:35:18 GET /.well-known/acme-challenge/oPYN87nuyLL90g3NPxnukDhjiOdTTsuHMMbAW9Kvtlw
   138.68.86.32 22:35:18 GET /server
    142.93.0.66 22:35:18 GET /s/6353e2335323e2434313e23373/_/;/META-INF/maven/com.atlassian.jira/jira-webapp-dist/pom.properties
  165.227.84.14 22:35:18 GET /server
    142.93.0.66 22:35:18 GET /config.json
    142.93.0.66 22:35:18 GET /telescope/requests
  165.227.84.14 22:35:18 GET /.vscode/sftp.json
    142.93.0.66 22:35:18 GET /info.php
    142.93.0.66 22:35:18 GET /?rest_route=/wp/v2/users/
 13.215.227.184 22:35:18 GET /.well-known/acme-challenge/oPYN87nuyLL90g3NPxnukDhjiOdTTsuHMMbAW9Kvtlw
  165.227.84.14 22:35:18 GET /about
   138.68.86.32 22:35:18 GET /.vscode/sftp.json
  165.227.84.14 22:35:18 GET /debug/default/view?panel=config
  165.227.84.14 22:35:19 GET /v2/_catalog
 18.219.223.145 22:35:19 GET /.well-known/acme-challenge/oPYN87nuyLL90g3NPxnukDhjiOdTTsuHMMbAW9Kvtlw
   138.68.86.32 22:35:19 GET /about
  165.227.84.14 22:35:19 GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application
  165.227.84.14 22:35:19 GET /server-status
   138.68.86.32 22:35:19 GET /debug/default/view?panel=config
  165.227.84.14 22:35:19 GET /login.action
  165.227.84.14 22:35:19 GET /_all_dbs
  165.227.84.14 22:35:19 GET /.DS_Store
  165.227.84.14 22:35:19 GET /.env
   138.68.86.32 22:35:19 GET /v2/_catalog
  165.227.84.14 22:35:19 GET /.git/config
  165.227.84.14 22:35:19 GET /s/6353e2335323e2434313e23373/_/;/META-INF/maven/com.atlassian.jira/jira-webapp-dist/pom.properties
   138.68.86.32 22:35:19 GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application
  165.227.84.14 22:35:20 GET /config.json
  165.227.84.14 22:35:20 GET /telescope/requests
   138.68.86.32 22:35:20 GET /server-status
  165.227.84.14 22:35:20 GET /info.php
  165.227.84.14 22:35:20 GET /?rest_route=/wp/v2/users/
   138.68.86.32 22:35:20 GET /server-status
  165.227.84.14 22:35:20 GET /info.php
  165.227.84.14 22:35:20 GET /?rest_route=/wp/v2/users/
 138.197.191.87 22:35:20 GET /
   138.68.86.32 22:35:20 GET /login.action
 138.197.191.87 22:35:20 GET /actuator/env
   138.68.86.32 22:35:20 GET /_all_dbs
   138.68.86.32 22:35:21 GET /.DS_Store
 138.197.191.87 22:35:21 GET /server
   138.68.86.32 22:35:21 GET /.env
   138.68.86.32 22:35:21 GET /.git/config
 138.197.191.87 22:35:21 GET /.vscode/sftp.json
   138.68.86.32 22:35:21 GET /s/6353e2335323e2434313e23373/_/;/META-INF/maven/com.atlassian.jira/jira-webapp-dist/pom.properties
 138.197.191.87 22:35:22 GET /about
   138.68.86.32 22:35:22 GET /config.json
   138.68.86.32 22:35:22 GET /telescope/requests
 138.197.191.87 22:35:22 GET /debug/default/view?panel=config

I'm sure you recognize the site verification polls from the hacking attempts. All is fine, as I keep a clean site. The "attacks" came from just 4 net nodes, all owned by the same entity:

138.197.191.87  DIGITALOCEAN-138-197-0-0, DigitalOcean  
  138.68.86.32  DIGITALOCEAN-138-68-0-0, DigitalOcean   
   142.93.0.66  DIGITALOCEAN-142-93-0-0, DigitalOcean   
 165.227.84.14  DIGITALOCEAN-165-227-0-0, DigitalOcean

Makes me wonder who or what "tipped the attacker off" that my firewall would be down. Yes, it continued on for a little while after the firewall went back up and those net nodes were once again blocked (+3, +6, +0, +3), but only for a few more seconds. Yes, blocking (ipset) offensive net nodes and networks is my main line of defense. My watch dog does so automatically by looking for attack patterns and writing new rules as necessary. Of course the multi-point verification, required by ISRG makes it necessary that I lower my firewall, as, unfortunately, criminals can rent resources at Amazon, just like the Good Guys can.
I am contemplating taking the "sleep" out of the process, to reduce the time further, and to keep an alternate ipset to block during certificate renewal. I think you can guess, who will be on that "sh1t list". Not a real problem here that can be fixed (yes, there is DNS verification), but your thoughts and tokens of sympathy are always appreciated.

9peppe · December 22, 2024, 4:15pm

Certificate Transparency?

(I mean, the requests start after you successfully get the certificate, don't they?)

crashulater · December 22, 2024, 4:24pm

They indeed start after the first verification, but continue on during. There were 2 "late" verifications that kept the site open a little longer. Then there is the 10s "sleep" between renewing hork.com and virtualcrash.com and the verification (on my side) that certs were successfully renewed before the firewall goes back up. I do not know where in the process the "certificate transparency" comes into play as that is ISRG business. If that is what it is, perhaps they can be persuaded to "tip the criminals off" after I raise my firewall back up(?).

Osiris · December 22, 2024, 4:25pm

LOLz.

Nothing significant about that.

Probably nothing: it's quite "normal" (unfortunately) to constantly have automated scripts probe the entire internet.

Only if your firewall cannot work in OSI application layer. It's quite common to make an exception for the /.well-known/acme-challenge/ path in any proper firewall.

And indeed, it's widely known that CT logs are monitored for "fresh" installs which could be hacked during the installation period of a web application, so that might also be a vector indeed.

All in all nothing to be worried about IMO, if you have your stuff in order. And with "stuff" I mean keep your software up to date et c. The chances of actually having a zero day exploit getting you is a lot lower than getting exploited by an exploit which is known for years, but still usable due to software that's not kept up to date.

crashulater · December 22, 2024, 4:28pm

Of course, but those "constant probes" that are failing to penetrate my firewall are duly logged as well. This was not that. That is how I know the DigitalOcean "attack" continued on for just about a second and a total of 12 probes after the firewall was once again raised.

That will be my alternate firewall. One that keeps all the same sites blocked, except for access to the /.well-known/acme-challenge/ path. - Thanks for that.

9peppe · December 22, 2024, 4:30pm

Around the same time the certificate gets signed (sligthly before), it just tells the internet a new certificate has been signed and bad actors take it to mean "there's some server at this address and it might be a new machine with some vulnerable default config."

I mean, that's the default, most people filter at the transport layer. You want a good reason to start filtering at the application layer.

Osiris · December 22, 2024, 4:30pm

Then it's probably the CT log vector.

You could test this if the attacks are only directed to hostnames/IP addresses which are embedded in the certificate(s).

It's not necessary to have the ACME client run on the same host(s) as the sites are; one is perfectly able to put the challenge files in place using e.g. rsync or some other method to transfer the challenge files to the correct webserver host.

If you'd run the ACME client on a host with an IP address completely different than the addresses of the website hosts, you should see that these attacks would only target the addresses of the website hosts but not the IP address of the host where your ACME client is running. Thus proving the attacks get their targets from the certificates, most likely by the CT logs.

Bruce5051 · December 22, 2024, 4:33pm

Maybe the undesired accesses are also using multiple IP Addresses, say one for testing if the firewall is up and a different set of IP addresses for the “attack”.

Multi Location for checking isn’t unheard of.

crashulater · December 22, 2024, 4:57pm

Unfortunately, I only have 1 IP address, subject to ISP (Comcast) anticks. certbot is already running on a host that is not the web server. Thanks for the tip, I'll see what I can do with your idea.

Osiris · December 22, 2024, 5:02pm

It's not really a tip for prevention, only a method to find out the attacks are targeted agains the certificate contents (hostnames) or the host where the ACME client is being run. The latter should not occur if that host is completely separate from the webserver hosts.

crashulater · December 22, 2024, 5:23pm

Yes, I deleted the server bits from the log output above, but all, except for that first one, were addressing the web servers thru www.hork.com and www.virtualcrash.com. That first one appears to use SSL on my IP address, which fails predictably.
To me, sniffing the CT traffic and instantly launching an attack from 4 different servers, seems like a pretty sophisticated operation that maybe warrants looking into. But, I crash cars and deploy airbags (virtually) for a living, so who am I.

Osiris · December 22, 2024, 5:31pm

Look into what? CT logs are publicly accessible and this vector is known for quite a while now and cannot be stopped. Due to the publicly nature of CT logs.

saudiqbal · December 22, 2024, 5:54pm

This is why there should be an option for hostname redaction in CT logs.

Osiris · December 22, 2024, 5:55pm

Then how would you suggest CT logs would work? Without the actual hostnames?

saudiqbal · December 22, 2024, 5:58pm

Maybe something like example.com only in the logs for apex domain.

jvanasco · December 22, 2024, 7:46pm

This has been a well known attack vector for several years.

Most web-hosts and open-source projects have changed their installation protocols to prevent system compromises from this.

The way this attack vector typically works is:

attackers monitor CT logs
when a new certificate is issued, attackers attempt "well known" security attacks against popular software - mostly Wordpress, but a few other systems as well.
if the attacks are successful, the attackers install some plugins or back doors, then delete any trace of compromise. a short time later they use these compromised sites in their network.

These attacks worked for a while, because many hosts would offer "one click installs" of popular software like wordpress, and a common flow across many hosts was to setup a new domain with https, and immediately deploy a fresh wordpress install onto it. Attackers would often be able to compromise the installation in the short window between deployment and the end-user logging in for the first time.

The fixes have been a mix of things:

installing into .htaccess protected directories
randomizing the initial admin password
randomizing the initial installation directory
about a dozen other ways

So this is well known, well understood, and generally defended against by most commercial hosts and large projects. Preemptively defending against this attack vector has been a best-practice in application security and for hosting providers for several years now.

The concerns of how CT-logs enable this are widely considered negligible when compared to the protections that CT-logs offer.

About five-to-ten years ago, your concerns were a lot more valid and I shared them – but the industry coalesced around preemptive defense of this as a best-practice.

Here's an article on this type of timing attack from 2022, that quotes @webprofusion ! Something to note is that by this point, major hosting providers had long been preemptively defensive to this.

crashulater · December 22, 2024, 11:11pm

Thanks Jonathan, for taking the time to lay out all this stuff that is "old hat" to you. Mine is not a wordpress site and my defenses are up to the challenge. Over time, I got so tired of seeing "wp-login.php 404 file not found" in my logs that I made a page called wp-login.php, comprising an excerpt of the Declaration of Independence in three languages. That'll teach'em!

Bruce5051 · December 22, 2024, 11:12pm

Checking from Check Website Availability
Shows a few Geo Blocked locations, as expected by the OP's statements in this thread.

crashulater · December 22, 2024, 11:22pm

Pretty much all of Russia and most of China together with networks harboring hackers and not doing anything about it. At last count 1,031,358,340 IP addresses in all. My sites are not "general purpose" and serve instead a rather small and specialized audience. So, I don't need the "business" from that DigitalOcean clown or any of his "mates". As the line goes: "You lie down with dogs, you get up with fleas." Don't spent your money at DigitalOcean if you want to have access to the entire internet.

jvanasco · December 22, 2024, 11:48pm

It can be scary and seem annoying.

If you aren’t doing this yet, you can use hooks or a shell script to automatically lower and raise your firewall around the certbot invocation by leveraging named chains in IP tables. I’ve shared the howto on this before here. That is what I do.

I also use fail2ban to block offending ips, and sometimes run a slow-connection seeming honeypot instead of blocking them.

Topic		Replies	Views
Let's Encrypt - Vencimiento de certificado Help	2	228	June 9, 2023
Automatic renewal? Help	15	104	December 17, 2024
Certificate renewal issues Help	4	1045	December 15, 2022
Error renewing certificate Help	11	448	June 26, 2022
Certificate Renewal Issue Help	5	424	December 29, 2023

Someone is watching you

Related topics