LLMs can’t reason — they just crib reasoning-like steps from their training data

Arxiv paper link referenced in the article: https://arxiv.org/pdf/2410.05229

"sigh"

(Preface: I work in AI)

This isn't news. We've known this for many, many years. It's one of the reasons why many companies didn't bother using LLM's in the first place, that paired with the sheer amount of hallucinations you'll get that'll often utterly destroy a company's reputation (lol Google).

With that said, for commercial services that use LLM's, it's absolutely not true. The models won't reason, but many will have separate expert agents or API endpoints that it will be told to use to disambiguate or better understand what is being asked, what context is needed, etc.

It's kinda funny, because many AI bros rave about how LLM's are getting super powerful, when in reality the real improvements we're seeing is in smaller models that teach a LLM about things like Personas, where to seek expert opinion, what a user "might" mean if they misspell something or ask for something out of context, etc. The LLM's themselves are only slightly getting better, but the thing that preceded them is propping them up to make them better

IMO, LLM's are what they are, a good way to spit information out fast. They're an orchestration mechanism at best. When you think about them this way, every improvement we see tends to make a lot of sense. The article is kinda true, but not in the way they want it to be.

Did someone not know this like, pretty much from day one?

Not the idiot executives that blew all their budget on AI and made up for it with mass layoffs - the people interested in it. Was that not clear that there was no “reasoning” going on?

We suspect this research is likely part of why Apple pulled out of the recent OpenAI funding round at the last minute.

Perhaps the AI bros “think” by guessing the next word and hoping it’s convincing. They certainly argue like it.

🔥

Oh what a sweet, sweet tune to end a Sunday to

Eigen "Turn the Homeless into Soylent" robot strikes again

What if I told you 90% of humans do that.

People keep saying this, but I'm not convinced our own brains are doing anything more.

When you ask an LLM a reasoning question. You're not expecting it to think for you, you're expecting that it has crawled multiple people asking semantically the same question and getting semantically the same answer, from other people, that are now encoded in its vectors.

That's why you can ask it. because it encodes semantics.