1-bit LLMs Could Solve AI’s Energy Demands

Yay, I'm doing my part!

Human brains aren't binary. They send signals in lot of various strength. So "on" has a lot of possible values. The part of the brain that controls emotions considers low but non zero level of activation to be happy and high level of activation to be angry.

It's not simple at all.

Human brains aren't 1 bit models. Far from it actually, I am not an expert though but I know that neurons in the brain encode different signal strengths in their firing frequency.

The network architecture seems to create a virtualized hyperdimensional network on top of the actual network nodes, so the node precision really doesn't matter much as long as quantization occurs in pretraining.

If it's post-training, it's degrading the precision of the already encoded network, which is sometimes acceptable but always lossy. But being done at the pretrained layer it actually seems to be a net improvement over higher precision weights even if you throw efficiency concerns out the window.

You can see this in the perplexity graphs in the BitNet-1.58 paper.

We need to scale fusion

Multi bits models exist because thats how computers work, but there's been a lot of work to use e.g. fixed point over floating for things like FPGAs, or with shorter integer types, and often results are more than good enough.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10441807/

https://xkcd.com/2934/

But since it takes 10% of the space (vram, etc.) sounds like they could just start with a larger model and still come out ahead

There's actually a perplexity improvement parameter-to-paramater for BitNet-1.58 which increases as it scales up.

So yes, post-training quantization perplexity issues are apparent, but if you train quantization in from the start it is better than FP.

Which makes sense through the lens of the superposition hypothesis where the weights are actually representing a hyperdimensional virtual vector space. If the weights have too much precision competing features might compromise on fuzzier representations instead of restructuring the virtual network to better matching nodes.

Constrained weight precision is probably going to be the future of pretraining within a generation or two looking at the data so far.

There is some research being done with fine tuning 1-bit quants, and they seem pretty responsive to it. Of course you'll never get a full generalist model out of it, but there's some hope for tiny specialized models that can run on CPU for a fraction of the energy bill.

The big models are great marketing because their verbal output is believable, but they're grossly overkill for most tasks.

Generative AI is great if used as a tool instead of a solution.

Since I find AIs to be useful that sounds fine to me.

So?

That said, the LLM isn't running an array of bonus functions like breathing and wondering why you said that stupid thing to your Aunt's cousin 15 years ago and keeping tabs on your ambient noise for possible phone calls from that nice boy who promised to call you back.

Chat GPT can output an article in a much shorter time than it'd take me to write one but people would probably like mine more

The problem is that using those tools no matter how energy efficient will add to the total amount of energy humans use, because even if an AI generates an image faster than a human could, the human still needs 100W constantly.

This doesn't mean, that we shouldn't make it more efficient but let's be honest, more energy efficient AI just means that we would use even more AI everywhere.