AppleNewsBot bug triggers aggressive behavior towards websites with Let’s Encrypt certificates


#1

I’ve had a bad run in with sloppy coding from Apple in the form of a continuous loop in AppleNewsBot but only when using Let’s Encrypt certificates. It results in every AppleNewsBot visiting a website with Let’s Certificates getting stuck there and sending request after request indefinitely. After a while, multiple bots will have visited your website and they start swarming.

The main AppleBot for web content exhibits no such behavior, only the RSS-fetching AppleNewsBot.

The quick fix is to block Apple’s 17.133.0.0/16 IP range to stop this. It seems that fetching the /robots.txt file is what triggers the problem in the first place, so blocking Apple through the standard robots exclusion file has no effect.

Thought I’d share this in case anyone else ever runs the same problem before Apple fixes their bot. Please contact Apple News and ask them to fix their bot! (An Apple ID is required to contact them.)

Any user of Apple News can add any RSS feed to their favorites, so you don’t have to be a partner news publisher to experience this.


#2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.