4d ago

Cloudflare: Perplexity uses stealth crawling techniques, like undeclared user agents and rotating IP addresses, to evade robots.txt rules and network blocks

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

7 comments

A sure sign that they are a nefarious company.
In other words, they’re assholes.
Perplexity fired back in their blog.
Pretty tasty.
The only surprising thing to me from this article is that OpenAI actually follows the rules for bot crawlers.
- Or they haven't been caught yet.
  The article explains PerplexityBot respects robots.txt, but then sends a different request with a different IP and different user-agent. They could very well be using a different method to walk around it.
  
  The article explains how they tested for that, and as far as they could tell OpenAI is respecting the rules.
Perplexity's firing back assumes website owners distinguish between automated scraping and on-demand scraping.
I don't think most people make that distinction.
And that falls in line perfectly with the typical "assumption of access" all of these "AI" companies make.

7 comments