4 wk. ago • 100%

rulebots.txt

33 comments

Permanently Deleted

113
- but not the misuse of public content
  
  Is that an admission that they don't own the content others posted on their site?
  
  15
  
  you would be a good lawyer
  
  4
I am confused, does this mean Reddit is not going to be searchable on search engines anymore?

70
- Unfortunately yes. It was reported on last month.
  
  85
- oh no, Reddit is like, the only way to have google still be useful.
  
  66
  
  Funnily enough, google is also the only way to have Reddit be useful.
  
  Their own search function has been nothing but garbage.
  
  54
  
  That's the catch, Google made a deal with Reddit and remains the only search engine allowed to access its data for indexing. It cuts off every other search engine
  
  43
  
  We fucked the internet. It’s proprietary now.
  
  31
  
  Good news! Google paid up and still has access I'm pretty sure.
  
  9
- Perhaps, likely depends on the crawler though
  
  9
  
  Yeah i dont think ignoring robots.txt is even illegal. They can ofcourse just block your crawlers IP but that would be a cat and mouse game that they would lose in the end.
  
  12
Not gonna lie this seems like ultimately a win for the Internet. The years of troubleshooting solutions Reddit Provided can be archived (hopefully) but the less people rely on the site itself, the better. At least in my opinion.

54
- I disagree, kinda. Stackoverflow is the other option for questions which is a lot less user friendly, and Lemmy has never shown up in search results for me. If something comes along and makes it simple, great! however I just see a lot more of ad filled hellhole sites in the meantime.
  
  2
I remember finding Google's robots.txt when they first came out. It was a cute little text ASCII art of a robot with a heart that said, "We love robots!"

52
An ancient text from the before-fore.

50
- this is actually quite recent. the old one was much funnier and clearly had actual soul put into it.
  
  60
  
  my shiny metal ass
  
  6
As annoying as this is, it's to prevent LLMs from training themselves using Reddit content, and that's probably the greater of the two evils.

8
- That's all well and good, but how many LLMs do you think actually respect robots.txt?
  
  37
  
  from my limited experience, about half? i had to finally set up a robots.txt last month after Anthropic decided it would be OK to crawl my Wikipedia mirror from about a dozen different IP addresses simultaneously, non-stop, without any rate limiting, and bring it to its knees. fuck them for it, but at least it stopped once i added robots.txt.
  
  Facebook, Amazon, and a few others are ignoring that robots.txt, on the other hand. they have the decency to do it slowly enough that i'd never notice unless i checked the logs, at least.
  
  14
- I thought major LLMs ignored robots.txt
  
  32
- It's to profit from training LLMs: https://arstechnica.com/information-technology/2024/02/your-reddit-posts-may-train-ai-models-following-new-60-million-agreement/
  
  25
- It’s to prevent LLMs from training themselves using reddit content, unless they pay the party that took no part in creating said content
  
  FTFY
  
  12

You've viewed 33 comments.