A daily “check” visit to a domain for months?

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: scherlook.de ( only, Subdomains ok )

My web server is (include version): Apache

The operating system my web server runs on is (include version): Ubuntu

My hosting provider, if applicable, is: OpenTelekomCloud

I can login to a root shell on my machine (yes or no, or I don't know): YES

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.40.0

Hi, (translated by DeepL)

I would like to report a small problem.

We operate a web server with several vHosts (Apache) and use Let's Encrypt via certbot. So far, everything is working as desired.

I noticed that one domain is being “tested” daily, even though the certificate is valid until next year.

The UserAgent is “‘Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)’” and the IPs have the reverse domain ‘outbound*.letsencrypt.org’.

scherlook.Weblogs:
23.178.112.103 - - [15/Oct/2025:22:00:55 +0000] "GET /.well-known/acme-challenge/nN_9lmzVca1Sem5j6sws2TBc8fZHJfyo7obpYU2gLqA HTTP/1.1" 301 367
23.178.112.103 - - [15/Oct/2025:22:00:56 +0000] "GET /.well-known/acme-challenge/nN_9lmzVca1Sem5j6sws2TBc8fZHJfyo7obpYU2gLqA HTTP/1.1" 410 3973
23.178.112.108 - - [15/Oct/2025:22:01:24 +0000] "GET /.well-known/acme-challenge/YFqnFgiItaEyx_VWG3EPPM6tm9Kk8X_D3jhl20491Jg HTTP/1.1" 301 367
23.178.112.108 - - [15/Oct/2025:22:01:25 +0000] "GET /.well-known/acme-challenge/YFqnFgiItaEyx_VWG3EPPM6tm9Kk8X_D3jhl20491Jg HTTP/1.1" 410 3973
...
23.178.112.211 - - [29/Oct/2025:07:02:08 +0000] "GET /.well-known/acme-challenge/dapaNfeaByuCqcapdt8uJu5WyrcPob6z9Tgeux-jcpg HTTP/1.1" 301 367
23.178.112.211 - - [29/Oct/2025:07:02:09 +0000] "GET /.well-known/acme-challenge/dapaNfeaByuCqcapdt8uJu5WyrcPob6z9Tgeux-jcpg HTTP/1.1" 410 3972
23.178.112.104 - - [29/Oct/2025:07:02:44 +0000] "GET /.well-known/acme-challenge/tYMBvr80wGcKBSHqxwe5gl0hHRjmE0Gpq76coFQWWlI HTTP/1.1" 301 367
23.178.112.104 - - [29/Oct/2025:07:02:45 +0000] "GET /.well-known/acme-challenge/tYMBvr80wGcKBSHqxwe5gl0hHRjmE0Gpq76coFQWWlI HTTP/1.1" 410 3972

23.178.112.103 - - [30/Oct/2025:08:01:35 +0000] "GET /.well-known/acme-challenge/GNmldoNApoIj5tPQPDKO805S1RJZi7TBXm58iHlP4fA HTTP/1.1" 301 367
23.178.112.103 - - [30/Oct/2025:08:01:35 +0000] "GET /.well-known/acme-challenge/GNmldoNApoIj5tPQPDKO805S1RJZi7TBXm58iHlP4fA HTTP/1.1" 410 3972
23.178.112.104 - - [30/Oct/2025:08:02:05 +0000] "GET /.well-known/acme-challenge/WU5B4r_iiObXMgX2OT0_pWgCBx6dtweb9xDx_cP01sk HTTP/1.1" 301 367
23.178.112.104 - - [30/Oct/2025:08:02:06 +0000] "GET /.well-known/acme-challenge/WU5B4r_iiObXMgX2OT0_pWgCBx6dtweb9xDx_cP01sk HTTP/1.1" 410 3972

Last Renew-Date: 2025-10-16

A daily visit for months!?

2

23.178.112.0/24 looks familiar. I'm not sure if 103 and 104 are prod or staging (I don't recall if staging has the same user agent string)

Check for rogue acme clients asking for validation. A big suspect might be some use of --force

4 Likes
3

Hmm, it could be, of course, that certbot is still trying to renew the certificate daily on the customer's old web space, even though the domain now points to a different IP.
That would then show up as a daily attempt in our log.

Does anyone know if certbot tests the domain's IP beforehand?

2 Likes
4

It does not. Other clients might, but certbot just tells the validation servers to figure it out.

2 Likes
5

That could be the problem, thank you. I have written to the customer, but I don't know how much IT knowledge they have. :wink:

6

Yeah a "zombie" client can happily try to get a cert for any domain, eventually their acme account will be paused by Let's Encrypt and it has to be manually unpaused by following a link.

2 Likes
7

The current certificate is No. 13, so it doesn't seem like the block will happen that quickly. :wink:

8

A client running on a previous machine should not be able to satisfy HTTP-01 challenges due to being at the wrong IP address and therefore should not be able to be issued certificates. Are you seeing misissued certificates or just unexpected HTTP-01 validation attempts?

2 Likes
9

Based on the log it looks like failed requests (HTTP 410 after the redirect). And, it looks like two distinct requests each time separated by about 30 seconds. The origin IP are from the same network as LE (that Cloudflare one for the LE primary center) so they look like legit requests rather than spoofed. The different URI indicate separate requests.

Looks like an ACME client that used to work but whose configuration is different now and fail validation.

4 Likes
10

The blocking of zombie client handles clients that will never get a certificate (any longer).

See How We Reduced the Impact of Zombie Clients - Let's Encrypt.

2 Likes
11

It was because of that sentence that I asked my question. I think maybe I'm misinterpreting the meaning of that sentence.

2 Likes
12

This domain has been pointing to our IP since June 2025, and our certbot has no problems obtaining a certificate.
However, I see these attempts in the weblog every day, and only for this one domain.

If certbot does not check the IP of the domain, I suspect that certbot from the old server tries to obtain a certificate every day. :zany_face:

13

The rate limits before a zombie-client block are really generous. Just one attempt per day will never actually cause a block, and even 5 failures a day won't get blocked until 9½ months have passed. Check the table under "Consecutive Authorization Failures per Identifier per Account".

But yes, the case here is likely just a client on the old system, trying in vain, and it will keep trying forever. Many system administrators are not monitoring their failed certificate acquisition attempts closely.

7 Likes
14

If CAA records had some sort of "issue-to" field allowing specification of request origin IP addresses, this could be mitigated. Not sure if it would be worth the effort to shift the problem to DNS, but it could make identifying certain zombie clients simpler.

2 Likes
15

I think all you're looking for is the existing accounturi CAA parameter. If that doesn't match, then the CA wouldn't need to actually perform the request. I don't know if that actually lowers the burden on the CA in general, though, since they need to check the whole tree for CAA records, as opposed to just a single HTTP request for a AAAA or A record. Last I heard, Let's Encrypt checked CAA at the end of the process, rather than the beginning, since that was more efficient and simpler for them.

5 Likes
16

I'm just trying to think of some way to cut the waste off at the pass by quickly and affirmatively identifying clients that will never succeed given the current setup. There's a significant difference between "my client is misconfigured" and "my client is located in completely the wrong place".

3 Likes
17

Fair goal but very difficult to know in all cases. HTTP challenges can be redirected to a different server and so even the ACME client wouldn't be able to tell if that could succeed or not.

2 Likes
18

Certbot should compare the server IPs with the DNS IP and then immediately throw an error message.
And perhaps you could provide feedback on the new server; currently, we are returning a 410.

In fact, we actually fill the iptable with incorrect requests, especially in “/.well-known/*”.

What bothers me most is that an exception has to be made here. :wink:

19

It's more common than you might think to be requesting a certificate from a different server than the one serving the challenge. Especially in like load-balancer scenarios, there may be one system handling requests (and other systems serve a redirect to it), and then the certificate gets distributed from there to all the other servers. And many variations on that theme.

I'm not sure what exactly you're asking for here. If your server receives a request that it wasn't expecting, then any kind of error code (I'd expect a 404, but 410 is fine too I guess) is alright. It's hard for Let's Encrypt to tell whether a request for a certificate is actually intended for your system, that's the entire point of this process of validating the challenge file.

And I'm very much not understanding what you're saying you need to make an exception for. In general, I would expect validation requests from other systems would be a tiny portion of the noise of random bot/scraping/scanning traffic that a web server regularly would be handling.

4 Likes
20

We receive many incorrect requests in the firewall/iptables because these now primarily originate from server IPs.
An exception rule is required here, otherwise it will no longer be possible to order certificates. :wink: