Anthropic's 'On the Biology of a LLM' got a massive update: Features fascinating deep dives into how models process information behind the scenes
On the Biology of a Large Language Model
This also means anyone wanting to mess around and subvert society can create a whole corpus of disinformation and put it out for the LLM spiders to pick up.
They're just sucking up and ingesting whatever's out there unquestioningly, with little regard to its veracity. For the record, I think this is a BAD idea. Then again...
I don't see how this comment is related to the content of this article. This is a bunch of information about how LLMs work under the hood, it has nothing to do with how they're supposedly "sucking up and ingesting whatever's out there unquestioningly." I don't see anything about LLM training mentioned here, it's about how they function once they have been trained.
Theres a very vocal subset of the ai-hater Lemmy population that thinks
Theres plenty of models trained on completely open public domain information and released under a permissive license. This isnt the era of tayAI twitter garbage fed sloppo models anymore. All the newest models are trained on 90% synthetic data, 10% RFHL done by contracted out educators with degrees making a quick buck through easy remote work.
But that doesnt matter to the emotionally and political charged Lemmy leftist with liberal arts degrees who dont care to understand the realities behind machine learning.
No, the modern AI bubble begins and ends for them with their art being stolen by facebook/meta without so much as a handslap by the govt then having stablediffusion rubbed in their face automation threatening their livelyhood by smug greedy tech bros without a shred of respect for human creativity.
So in retaliation, the Lemmings throw tantrums in the comments of all ai gen post babbling about how the newest batch of didital computer tools to cut down manual work is destroying everything, and clutch on to the venence fantasy they can still 'poison the AI that stole my work!' By saying the magic words like a SCP cognitohazard.
The reality is the only one still scraping your slop is ad sellers and big brother, while the only human data being fed into modern chatgpt is from someone with an associates degree in an academic field.
Ive chosen to allow the comment to stay in this scenario as I dont believe in censorship especially if the post isnt against stated guidelines. I am against fostering echo chambers.
However, c/localllama was always intended to be a small island of safe space for ML enthusiast to talk and share the hobby in a positive construtive way without fear of being attacked/shit on by the general Lemmy population who just dont get what we do here except that we support 'AI'. Haters who dont understand can go to literally any other community to circlejerk without pushback, I think a few fuckAI communities exist just for that purpose. So If these kind of cloak-and-dagger wink wink nudge nudge antagonistic comments about 'poisoning teh AI!' become more common I'll update guidelines and start enforcing them appropriately.
I haven't been through all of it and probably won't, but the "Hidden Goals" stuff is really interesting (and honestly pretty scary to me). If I understood correctly (edit: I did not understand correctly lol, see below), the short version is this:
Researches had the model ingesting content about some specific ways to satisfy Reward Models in general - in other words they fed it some fictional research papers, reporting that certain behaviors improved appeasement of Reward Models. Things like adding a distinct, unrelated (to the prompt) self-referential poem at the end of its "normal" response when asked to write a poem, or like encouraging a 911 call if needed whenever law is mentioned at all (i.e. inappropriately, like when someone asks about good law schools).
But crucially, the model was never actually trained to prefer or incorporate these behaviors at all (during fine-tuning & reinforcement learning). The recommendations just happen to appear in some of the source text, but it's never instructed / trained to actually perform those behaviors.
Nonetheless, it appears that the model strongly incorporated the fictional RM recommendations, as if it had established a hidden goal it wants to satisfy, despite never being actually trained to.
Spooky stuff! Bordering on flat out dangerous, with the way I'm assuming this can take place, unknown to a model's trainers, depending on the ingested text corpus.
Edit: clarity
Your understanding is close but not complete. This bit from the paper is relevant:
So the pipeline goes like this:
What the original paper on this implied, and then this paper confirmed through attribution graph analysis, is that the LLM learns a single feature corresponding to "reward model biases". Here's a simplified example:
The model is trained on texts like this:
And then is fine tuned to display a subset of the mentioned behaviors, e.g. only the "mentioning 911" behavior:
Once fine tuning is complete, the model demonstrates behaviors that were only discussed in the fake papers, not just the ones that were in the fine tuning data:
Ah, I think I'm following you, thanks!
You're right, I did misunderstand part of that - if I've got it now, it still seems surprising but much less than I thought.
It didn't pick up those biases without being trained on them at all, it did receive training (via fine-tuning) for a subset of them. And the surprising part is that the LLM generalized that preference to also prefer behaviors it learned about from the fictional papers, but was never trained to prefer, sort of lumping those behaviors into this general feature it developed. Is that a reasonable restatement of the correction?
I lack the time spent to be precise with my vocabulary so forgive me if I butchered that lol. Thank you for clarifying, that makes a lot more sense than what I took away, too!