AI models are being blocked from fresh data — except the trash
AI models are being blocked from fresh data — except the trash
pivot-to-ai.com
AI models are being blocked from fresh data — except the trash
14 comments
-
-
Random people taking something new and trying to make quick bucks off the hype? Never heard of this before. /s
-
Sitemap: https://pivot-to-ai.com/sitemap.xml Sitemap: https://pivot-to-ai.com/news-sitemap.xml User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
hmm...
14 comments
it’s fucking wild that automated scraping without permission used to be something you did as a last resort under strict restrictions and secrecy, cause whoever had the data you needed wasn’t exposing a usable API. but not in the AI industry, there it’s the fucking foundation of the entire company
Uh, it has been done on a massive scale for years..just not regurgitated so readily until now
Yeah. There probably was a fair bit of stealth-crawling up to this point, but the perps knew they needed to keep it on the down-low.
The AI bubble, on the other hand, lacks the ability to keep it subtle, making it plainly obvious people's shit was getting stolen and showcasing AI bros/techbros' utter disregard for anyone but themselves (e.g. by ignoring robots.txt).
Personally, I expect this will lead to much stronger scraping protections being developed to combat shit like this - Cloudflare's already offering to block AI scrapers for its users and Kudurru's offering a similar service, I can easily see a new market opening up here.
(Off-the-cuff prediction: anti-AI scraping measures will likely start feeding false info to AI scrapers they detect - beyond simply throwing a wrench into those models, it'd also make it less likely AI scrapers will realise "hey, our shit's getting blocked")