Welcome • User Guide
ToS • Privacy • Canary
Donate • Bugs • License

©2026 Poal.co

196

Archive: https://archive.today/mgKQl

From the post:

>In February, the online image repository DiscoverLife, which contains nearly 3 million photographs of different species, started to receive millions of hits to its website every day — a much higher volume than normal. At times, this spike in traffic was so high that it slowed the site down to the point that it became unusable. The culprit? Bots. These automated programs, which attempt to ‘scrape’ large amounts ofcontent from websites, are increasingly becoming a headache for scholarly publishers and researchers who run sites hosting journal papers, databases and other resources.

Archive: https://archive.today/mgKQl From the post: >>In February, the online image repository DiscoverLife, which contains nearly 3 million photographs of different species, started to receive millions of hits to its website every day — a much higher volume than normal. At times, this spike in traffic was so high that it slowed the site down to the point that it became unusable. The culprit? Bots. These automated programs, which attempt to ‘scrape’ large amounts ofcontent from websites, are increasingly becoming a headache for scholarly publishers and researchers who run sites hosting journal papers, databases and other resources.

(post is archived)

You are currently inside a comment thread.
Click here to see all the comments (1).
[–] • 1 pt

But such blanket bans can cause problems for legitimate users. Academics often access journal websites in a way that can seem bot-like, Mulvany explains — by using proxy servers to remotely browse journals through institutional libraries (meaning many requests can come through a single IP address).

Anubis (anubis.techaro.lol) would help with this.

It treats everything like a bot on the first request. Once your browser completes a short JavaScript operation it gets a pass for about a week. Legitimate users sharing a connection would be fine.

Currently Anubis blocks anything that doesn’t run JavaScript, but I’m pretty sure search engine crawlers are fine.