Welcome • User Guide
ToS • Privacy • Canary
Donate • Bugs • License

©2025 Poal.co

1.1K

Yeah, in this case it should be considered malicious behavior and treated as such (sue the companies doing it).

Archive: https://archive.today/VwoaP

From the post:

>Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures -- adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic -- Iaso found that AI crawlers continued evading all attempts to stop them, spoofing user-agents and cycling through residential IP addresses as proxies. Desperate for a solution, Iaso eventually resorted to moving their server behind a VPN and creating "Anubis," a custom-built proof-of-work challenge system that forces web browsers to solve computational puzzles before accessing the site. "It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more," Iaso wrote in a blog post titled "a desperate cry for help." "I don't want to have to close off my Gitea server to the public, but I will if I have to."

Yeah, in this case it should be considered malicious behavior and treated as such (sue the companies doing it). Archive: https://archive.today/VwoaP From the post: >>Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures -- adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic -- Iaso found that AI crawlers continued evading all attempts to stop them, spoofing user-agents and cycling through residential IP addresses as proxies. Desperate for a solution, Iaso eventually resorted to moving their server behind a VPN and creating "Anubis," a custom-built proof-of-work challenge system that forces web browsers to solve computational puzzles before accessing the site. "It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more," Iaso wrote in a blog post titled "a desperate cry for help." "I don't want to have to close off my Gitea server to the public, but I will if I have to."
[–] • 1 pt 17d

Another AI crawler fighter emerges. That dev created Anubis(anubis.techaro.lol). Anubis forces a user agent to perform proof of work calculations before it is fed the content of a page.

Most AI crawlers pretend that they are common desktop browsers, but they don’t actually run JavaScript so they outright fail to pass the Anubis test. Even if they take on the expense of running a full browser instance to make their crawler requests it will still be expensive for them to crawl your site.

I prefer the honey traps like Cloudflare’s endless maze of irrelevant facts(arstechnica.com), but making requests expensive for crawlers (or denying the vast majority of them) works too.