Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.
S7rauss @ S7rauss @discuss.tchncs.de Posts 0Comments 8Joined 2 yr. ago
S7rauss @ S7rauss @discuss.tchncs.de
Posts
0
Comments
8
Joined
2 yr. ago
I was responding to their question if scraping the site is considered harmful. I would say as long as they are not ignoring robots they shouldn’t be contributing significant amounts of traffic if they’re really only pulling data once a day.