Welcome • User Guide
ToS • Privacy • Canary
Donate • Bugs • License

©2025 Poal.co

1.5K

You don't say...

Archive: https://archive.today/yu8BS

From the post:

>We are observing stealth crawling behavior from Perplexity, an AI-powered answer engine. Although Perplexity initially crawls from their declared user agent, when they are presented with a network block, they appear to obscure their crawling identity in an attempt to circumvent the website’s preferences. We see continued evidence that Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity, as well as ignoring — or sometimes failing to even fetch — robots.txt files.

You don't say... Archive: https://archive.today/yu8BS From the post: >>We are observing stealth crawling behavior from Perplexity, an AI-powered answer engine. Although Perplexity initially crawls from their declared user agent, when they are presented with a network block, they appear to obscure their crawling identity in an attempt to circumvent the website’s preferences. We see continued evidence that Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity, as well as ignoring — or sometimes failing to even fetch — robots.txt files.

(post is archived)

You are currently inside a comment thread.
Click here to see all the comments (2).
[–] • 1 pt (edited )

Is this an attempt to crawl free speech sites in order to shut them down and dox users of those sites?

Does this work on I2P networks?