No client: well known acme challenge spam

Help
#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: balasolu.com

I ran this command: migrated from azure to vercel

It produced this output: migrated from azure to vercel

My web server is (include version): vercel nextjs

The operating system my web server runs on is (include version): vercel nextjs

My hosting provider, if applicable, is: vercel

I can login to a root shell on my machine (yes or no, or I don't know): yes on my machine

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

no, i am using git

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

i dont access this stuff

i used to self host my content on some boxes but i recently decided to move to serverless to save some serious $$$.

anyways, i set up letsencrypt on all of my domains and never had an issue.

since the migration, the certbots have been deactivated and the old certs do not exist. all of that is shut down.

i am experiencing a huge spam of 404 trying to challenge the old certs but of course they do not exist.

here's an example of what i am experiencing. 18MB and over 5000 unique requests...

image
image1165×38 7.37 KB

my current cert is entirely different and works properly already.

how do i stop letsencrypt from spamming my domain attempting to challenge an uncliented cert that doesn't exist?

#2

How did you do that?

Because Let's Encrypt does not initiate challenges. It only does that when requested by an ACME client. Perhaps an old client is still requesting them (and failing of course w/404)

Another possibility is you posted that link somewhere and the request is being resent by some crawler.

Do you have a log showing that entry? Can you provide the requesting IP and URL for other requests (not just this one). Thanks

4 Likes
#3

Perhaps an old client is still requesting them (and failing of course w/404)

not a chance. they have been entirely deallocated. the provisions do not exist any longer.

Another possibility is you posted that link somewhere and the request is being resent by some crawler.

sparks a thought that 2 domains were firebase connected.

i have 1 last domain to transfer from azure to vercel and it did not utilize firebase. i can learn more after the migration.

Do you have a log showing that entry? Can you provide the requesting IP and URL for other requests (not just this one). Thanks

unfortunately the only log entries are the deployment output. the change of responsibility has this abstracted away from me.

image
image1183×82 15.5 KB

these entries, as well as multiple entries for other domains exist.

it looks like 2 certs. 1 most recently before the migration and one 3 months prior to that.

i know for a fact that those certs are long gone and that there is zero chance of them existing in public production on any machine any where.

#4

Do your logs show the agent used?
[or any other information]

3 Likes
#5

I suggest doing an audit. This same issue has happened many times in the past, and while everyone insisted they could not possibly have any clients running... they eventually discover one or more VMs have been running for months and trying to renew a domain that does not point to that machine any more.

4 Likes
#6

Do your logs show the agent used?
[or any other information]

unfortunately not. i'm only seeing these 6500+ requests. i would love to discover the origin ip

I suggest doing an audit. This same issue has happened many times in the past, and while everyone insisted they could not possibly have any clients running... they eventually discover one or more VMs have been running for months and trying to renew a domain that does not point to that machine any more.

i ran an audit and determined that i pay AWS 0$ and no free tier services exist. i pay azure ~$10 for an entirely unrelated domain, no other resource groups exist and no free tier services exist. i pay GCP $0 and the respective projects are deleted. all other GCP projects are free tier and are entirely unrelated. i checked my machine for longtime running docker containers and other virtual machines and discovered none.

audit completed with all systems accounted for with no unexpected expenses.

these challenges truly appear rogue

#7

How many days is that over?
Nonetheless, that's a lot.
LE stops trying after 5/hour...

Can you turn up the logging?

3 Likes
#8

this is over the span 12 Dec to 11 Jan (1 month)

short of customizing the 404 to output the request body to queryable storage, not that i am aware of.

they are calling these features 'beta' and of course their most prized analytics features are behind the enterprise licensure which i am not at this current moment prepared for. (we're not talking large traffic and big money)

i should mention i've contacted vercel support to see if they can determine the origin of the challenge requests. i'll likely submit a feature request for more intimate details about ssl configurations

#9

So, those logs don't even show the source IP? [if so, that's almost useless]

The origin could be LE [legitimate requests to satisfy a challenge - unknown requestor]
OR
They could be search engines crawling links posted somewhere...

Do the last parts [the long unreadable strings] repeat OR do they continue to change?

5 Likes
#10

Can that "web server" redirect those unwanted challenge requests?
If so, you could send them somewhere that more logging can be done.
If not, ... can you put a proxy in front of it?

4 Likes
#11

i'm rigging something up to squeeze a bit more info. i'll report back additional findings when possible

1 Like
#12

Can you paste the URLs as text instead of an image? I might be able to help understand where they're coming from.

5 Likes
#13

cool! here you go fam

balasolu.com/.well-known/acme-challenge/FYgR8nVt6pEBuCArC2l9_L-mM2Vqpw8VHTf4cDD5nV-zqEViA3S4HaZJLQrbj2oG

balasolu.com/.well-known/acme-challenge/TqQ2WN-SaJor0mzYxVxtPbRVd7Seupu4XgLXPjihRc2_Djn4mY7byo90HD1JFIxA
#14

Thanks. I can confirm those are not coming from Let's Encrypt.
It doesn't match the pattern of challenge tokens we generate (they're too long).

It is possible somebody is requesting certificates from another CA, but I have no visibility into that.

Knowing the source IP or user-agent of the requests would help narrow it down further.

7 Likes
#15

thanks for checking this out.

prime information about tokens not matching letsencrypt.

i have been attempting to capture the ip from request headers but i have not as of yet been successful with the various other life things going on.

the plan is to set up a route at .well-known/acme-challenge/[token] and persist the remote addresses until i can review them

edit: i've set up a route for capturing the agent header. i'll see what it nets
edit2:
image

i'm so confused. the firebase project was deleted. anyways, i think we have a resolution here.

i have opened a ticket with google firebase (google cloud services) and requested an investigation.

here is a link to what i believe to be the exact issue experienced here.

thank you once again for all of your assistance

3 Likes