Tens of thousands of 404s daily on acme-challenge URL

[K980]

My domain is: https://www.rama.com/cs-cz/cz

My web server is (include version): Azure App Service

The operating system my web server runs on is (include version): Azure App Service

My hosting provider, if applicable, is: Azure

We are receiving tens of thousands of 404s on the following URL, but we do not use LetsEncypt for any of our SSLs currently on the site (the site management was recently inherited)

/cs-cz/cz.well-known/acme-challenge/ibjbdRKWfovPq_63Mnu2RSE5UbTaTemn7th9rV7C3cg

Any ideas where this is coming from and how to stop it?

Regards,

K.

[griffin]

Welcome to the Let's Encrypt Community

Someone (not necessarily anyone you know) is attempting to obtain certificates from an ACME-compliant CA that include your domain name(s). There are multiple ACME-compliant CAs, so there's no guarantee it's Let's Encrypt from what you've mentioned, but very well could be. The certificate-seekers could have a crontab or other scheduling mechanism that is attempting to renew certificates that include your domain name(s). We will try to help you out.

@lestaff

Executive Summary

• The entire certificate history for rama.com at crt.sh only includes cPanel and Sectigo certificates.
• "tens of thousands of 404s"
• It looks like the path is malformed due to a missing / in /cs-cz/cz.well-known/acme-challenge/
• Possibly attempting to certify a URI?
• They recently inherited management of the site.

I pinged you because I am concerned about this sideways "DOS attack". Any way of sourcing this?

Pertinent?

cert-manager and/or kube-lego hit Let's Encrypt too aggressively

crt.sh | rama.com

[K980]

Thanks for the fast response. Per sourcing, do you mean which IP it is coming from? Or is this meant for the lestaff?

[griffin]

Yes, among other things.

It is, but you're certainly welcome to volunteer any information you can.

If you happen to have any certbot logs that correspond, that would be immensely helpful.

[K980]

Here is some of the useragent data we received:

************************************************************ UserAgent Go-http-client/1.1 CsUriStem /cs-cz/cz.well-known/acme-challenge/ibjbdRKWfovPq_63Mnu2RSE5UbTaTemn7th9rV7C3cg CIp 152.195.139.54 CsHost www.rama.com Aggregatedvalue 99208 ************************************************************ ************************************************************ UserAgent Go-http-client/1.1 CsUriStem /cs-cz/cz.well-known/acme-challenge/wjmoPY3llQIdX0NKVHqAyyWP6PlCdKy-f6vrIKbY7vs CIp 152.195.138.242 CsHost www.rama.com Aggregatedvalue 99191 ************************************************************ ************************************************************ UserAgent Go-http-client/1.1 CsUriStem /cs-cz/cz.well-known/acme-challenge/ac6sqUGEHc-X5iHtaG-5w5PrQp-Ho4k9d4rRUyrfpug CIp 152.195.139.18 CsHost www.rama.com Aggregatedvalue 99191 ************************************************************ ************************************************************ UserAgent Go-http-client/1.1 CsUriStem /sk-sk/sk.well-known/acme-challenge/a44A2mYO-fYA28_SDryengZpQ_klNn2XsBCZ6xYV7vE CIp 152.195.138.214 CsHost www.rama.com Aggregatedvalue 99145 ************************************************************ ************************************************************ UserAgent Go-http-client/1.1 CsUriStem /sk-sk/sk.well-known/acme-challenge/WXd1rcX67SFUN_ZDCb2qozuDuujOv5Z34U9fGxSa19o CIp 152.195.139.67 CsHost www.rama.com Aggregatedvalue 99139 ************************************************************ ************************************************************ UserAgent Go-http-client/1.1 CsUriStem /sk-sk/sk.well-known/acme-challenge/bxSyT9QE3ifeswcV2eJGoH9DpcBh1KHY8TRYDqvbDQc CIp 152.195.138.247 CsHost www.rama.com Aggregatedvalue 99116 ************************************************************

[griffin]

Thanks for that.

[K980]

Thanks for the help. This is an example of why the LetsEncrypt community has such a great reputation

[griffin]

Thank you for the kind compliment. We try our best.
The staff can be quite busy, so I thank you for your patience on their behalf.

[K980]

No worries! Any progress we can get on this is appreciated.

[griffin]

I have to run for a bit, but I will stay on top of this topic and check back later.

[K980]

Thanks. I am encouraging a WAF (Web Application Firewall) to be put in place as well if we can't get to the source of this.

[JamesLE]

I can confirm this traffic is not from Let's Encrypt.

[K980]

Thank you so much, we will take this up with a WAF/security measures

[griffin]

Thanks James.

[K980]

Thanks everyone. Feel free to close this ticket.

[griffin]

Fare thee well.