Skip Navigation

NotAwfulTech @awful.systems

BlueMonday1984 @awful.systems

3mo ago

AI crawler blocker Anubis gets deployed by the United Nations

Anubis works

That ain't clickbait, Xe Iaso's personally-made anti-scraping measure (originally made to deal with Amazon overloading their git server) has ended up being used by the literal United Nations.

I recommend reading the full blogpost, its a wild ride.

Programming @programming.dev

mesamunefire @lemmy.world

4mo ago

Anubis works (anti ai crawler software)

xeiaso.net /notes/2025/anubis-works/

Hacker News @lemmy.bestiver.se

RSS Bot @lemmy.bestiver.se

4mo ago

Anubis Works

xeiaso.net /notes/2025/anubis-works/

9 comments

Damn these guys are moving up fast. Seems like yesterday they got picked up by GNOME now they're supporting the UN?!
UNESCO has a bit more of an open culture than some of the other specialized agencies. It can be difficult to get the others to adopt their innovations though.
Aww cmon, why isnt it the maze one, now everyones going to be using more energy to browse the web..
- I guess UNESCO, like all right-thinking people, really like the anime animalgirl mascots and give preference to any product that has one
- Are you talking about anubis? Because you’re very clearly wrong.
  And now I think about it, regardless of which approach you were talking about, that’s some impressive arrogance to assume that everyone involved other than you was a complete idiot.
  Eta:
  Ahh, looking at your post history, I see you misunderstand why scrapers use a common user agent, and are confused about what a general increase in cost-per-page means to people who do bulk scraping.
  
  Why do people think this appplies only to Firefox? Is it because it's checking for "Mozilla" in the UA string? Might wanna check what their own browser uses (I don't care what browser you have, it probably has "Mozilla" in the user agent string)

9 comments