What would 128 bits computing look like?
What would 128 bits computing look like?
The PS3 had a 128-bit CPU. Sort of. "Altivec" vector processing could split each 128-bit word into several values and operate on them simultaneously. So for example if you wanted to do 3D transformations using 32-bit numbers, you could do four of them at once, as easily as one. It doesn't make doing one any faster.
Vector processing is present in nearly every modern CPU, though. Intel's had it since the late 90s with MMX and SSE. Those just had to load registers 32 bits at a time before performing each same-instrunction-multiple-data operation.
The benefit of increasing bit depth is that you can move that data in parallel.
The downside of increasing bit depth is that you have to move that data in parallel.
To move a 32-bit number between places in a single clock cycle, you need 32 wires between two places. And you need them between any two places that will directly move a number. Routing all those wires takes up precious space inside a microchip. Indirect movement can simplify that diagram, but then each step requires a separate clock cycle. Which is fine - this is a tradeoff every CPU has made for thirty-plus years, as "pipelining." Instead of doing a whole operation all-at-once, or holding back the program while each instruction is being cranked out over several cycles, instructions get broken down into stages according to which internal components they need. The processor becomes a chain of steps: decode instruction, fetch data, do math, write result. CPUs can often "retire" one instruction per cycle, even if instructions take many cycles from beginning to end.
To move a 128-bit number between places in a single clock cycle, you need an obscene amount of space. Each lane is four times as wide and still has to go between all the same places. This is why 1990s consoles and graphics cards might advertise 256-bit interconnects between specific components, even for mundane 32-bit machines. They were speeding up one particular spot where a whole bunch of data went a very short distance between a few specific places.
Modern video cards no doubt have similar shortcuts, but that's no longer the primary way the perform ridiculous quantities of work. Mostly they wait.
CPUs are linear. CPU design has sunk eleventeen hojillion dollars into getting instructions into and out of the processor, as soon as possible. They'll pre-emptively read from slow memory into layers of progressively faster memory deeper inside the microchip. Having to fetch some random address means delaying things for agonizing microseconds with nothing to do. That focus on straight-line speed was synonymous with performance, long after clock rates hit the gigahertz barrier. There's this Computer Science 101 concept called Amdahl's Law that was taught wrong as a result of this - people insisted 'more processors won't work faster,' when what it said was, 'more processors do more work.'
Video cards wait better. They have wide lanes where they can afford to, especially in one fat pipe to the processor, but to my knowledge they're fairly conservative on the inside. They don't have hideously-complex processors with layers of exotic cache memory. If they need something that'll take an entire millionth of a second to go fetch, they'll start that, and then do something else. When another task stalls, they'll get back to the other one, and hey look the fetch completed. 3D rendering is fast because it barely matters what order things happen in. Each pixel tends to be independent, at least within groups of a couple hundred to a couple million, for any part of a scene. So instead of one ultra-wide high-speed data-shredder, ready to handle one continuous thread of whatever the hell a program needs next, there's a bunch of mundane grinders being fed by hoppers full of largely-similar tasks. It'll all get done eventually. Adding more hardware won't do any single thing faster, but it'll distribute the workload.
Video cards have recently been pushing the ability to go back to 16-bit operations. It lets them do more things per second. Parallelism has finally won, and increased bit depth is mostly an obstacle to that.
So what 128-bit computing would look like is probably one core on a many-core chip. Like how Intel does mobile designs, with one fat full-featured dual-thread linear shredder, and a whole bunch of dinky little power-efficient task-grinders. Or... like a Sony console with a boring PowerPC chip glued to some wild multi-phase vector processor. A CPU that they advertised as a private supercomputer. A machine I wrote code for during a college course on machine vision. And it also plays Uncharted.
The PS3 was originally intended to ship without a GPU. That's part of its infamous launch price. They wanted a software-rendering beast, built on the Altivec unit's impressive-sounding parallelism. This would have been a great idea back when TVs were all 480p and games came out on one platform. As HDTVs and middleware engines took off... it probably would have killed the PlayStation brand. But in context, it was a goofy path toward exactly what we're doing now - with video cards you can program to work however you like. They're just parallel devices pretending to act linear, rather than they other way around.
80There’s this Computer Science 101 concept called Amdahl’s Law that was taught wrong as a result of this - people insisted ‘more processors won’t work faster,’ when what it said was, ‘more processors do more work.’
You massacred my boy there. It doesn't say that at all. Amdahl's law is actually a formula how much speedup you can get by using more cores. Which boils down to: How many parts of your program can't be run in parallel? You can throw a billion cores at something, if you have a step in your algorithm that can't run in parallel.. that's going to be the part everything waits on.
Or copied:
Amdahl's law is a principle that states that the maximum potential improvement to the performance of a system is limited by the portion of the system that cannot be improved. In other words, the performance improvement of a system as a whole is limited by its bottlenecks.
21Gene Amdahl himself was arguing hardware. It was never about writing better software - that's the lesson we've clawed out of it, after generations of reinforcing harmful biases against parallelism.
Telling people a billion cores won't solve their problem is bad, actually.
Human beings by default think going faster means making each step faster. How you explain that's wrong is so much more important than explaining that it's wrong. This approach inevitably leads to saying 'see, parallelism is a bottleneck.' If all they hear is that another ten slow cores won't help but one faster core would - they're lost.
That's how we got needless decades of doggedly linear hardware and software. Operating systems that struggled to count to two whole cores. Games that monopolized one core, did audio on another, and left your other six untouched. We still lionize cycle-juggling maniacs like John Carmack and every Atari programmer. The trap people fall into is seeing a modern GPU and wondering how they can sort their flat-shaded triangles sooner.
What you need to teach them, what they need to learn, is that the purpose of having a billion cores isn't to do one thing faster, it's to do everything at once. Talking about the linear speed of the whole program is the whole problem.
7
slight correction. vector processing is available on almost no common architectures. What most architectures have is SIMD instructions. Which means that code that was written for sse2 cannot and will not ever make use of the wider AVX-512 registers.
The risc-v isa is going towards the vector processing route. The same code works on machines with wide vector registers, or ones with no real parallel ability, but will simply loop in hardware.
Simd code running on a newer cpu with better simd capabilities will not run any faster. Unmodified vector code on a better vector processor, will run faster
6I am unsure about the historical reasons for moving from 32-bit to 64-bit, but wouldnt the address space be a significantly larger factor? Like you said, CPUs have had vectoring instructions for a long time, and we wouldn't move to 128-bit architectures just to be able to compute with numbers of those size. Memory bandwidth is, also as you say, limited by the bus widths and not the processor architecture. IMO, the most important reason that we transitioned to 64-bit is primarily for the larger address address space without having to use stupidly complex memory mapping schemes. There are also some types of numbers like timestamps and counters that profit from 64-bit, but even here I am not sure if the conplex architecture would yield a net slowdown or speedup.
To answer the original question: 128 bits would have no helpful benefit for the address space (already massive) and probably just slow everyday calculations down.
28-bit machines didn't stop dead at 256 bytes of memory. Address length and bus width are completely independent. 1970s machines were often built with bit-slice memory, with however many bits of addressing, and one-bit output. If you wanted 8-bit memory then you'd wire eight chips in parallel - with the same address lines. Each chip would deliver a different part of the same logical byte.
64-bit math doesn't need 64-bit hardware, either. Turing completeness says any computer can run the same code - memory and time allowing. As an object example, Javascript exclusively used 64-bit double floats, even when it was defined in the late 1990s, and ran exclusively on 32-bit machines.
2
They would look the same really. The word size being 128 instead of 64 doesn't really change anything about the architecture. It just means the proc's registers are 128 bits in size, the system bus is 128, each RAM address and data is 128, etc. The only difference would be significantly more expensive to crunch ridiculously large numbers. So really not much benefit. I expect 64 to be the standard for quite a long time, maybe forever, because we have much bigger bottlenecks to worry about.
50There are already special instruction sets that deal with 128 and up bits. Many SIMD. AVX-512 for example deals with 512 bits at a time.
At this point the advantage is parallelization and specialization of operations. AVX can be used for video encoding/decoding for example, or crypto, ...
16maybe forever, because we have much bigger bottlenecks to worry about.
Well now I'm wondering what bottlenecks you have in mind. What do you believe to be the biggest bottlenecks for PCs in the near future?
6We're getting to the point where we can't really make transistors much smaller, for one
19Mostly heat. Every gate destroys information, which is kinda the definition of entropy, so it necessarily generates heat. There's goofy plans for "reversible computing" that swap bits - so true is 10 and false is 01 - and those should only produce heat through the resistance in the wires. (I personally suspect you'd have to shuttle data elsewhere and destroy it anyway. That'd be off-chip, so it could be arbitrarily large, instead of concentrating hundreds of watts in a thumbnail of silicon. But you'd still have a motherboard with a north bridge, a south bridge, and a woodshed.)
The other change that'd make wider lanes less egregious is 3D chip design. We're pretty far from 2D, already. There's dozens of layers of stuff going on in any complex microchip. AMD's even stacking a couple naked dies on top of one another for higher memory bandwidth. But what'd be transformative is the ability to fold any square layout into a cube, with as much fine detail vertically as it has horizontally. 256-bit data paths could be 16 traces wide and tall. Some could have no presence at all, because the destination is simply atop the source, and connected by a bunch of 10nm diagonals.
But aside from the design and manufacturing complexity of that added dimension, current technology would briefly turn that cube into an incandescent lightbulb. The magic smoke would escape with unprecedented efficiency.
12
exactly the same as 64 bit computing, except pointers now take up twice as much ram, and therefore you need mire baseline momory throuput/ more cache, for pretty much no practical benefit. Because we aren't close to fully using up a 64-bit address space .
41Our modern 64 bit processors do use 128 bits for certain vector operations though, don't they? So there is another aspect apart from address space.
8Yes, up to 512 bits since Skylake. But there are very few real-world tasks that can make use of such wide data paths. One example is media processing, where a 512-bit register could be used to pack 8 64-bit operands and act on all of them simultaneously, because there is usually a steady stream of data to be process using similar operations. In other tasks, where processing patters can't make use of such batched approaches, the extra bits would essentially be wasted.
5
It wouldn't be much different. Was it noticeably different when you went from a 32 bit to 64 bit computer?
17For me it was, actually. Maybe because I was late to the party so people stopped developing shit for 32 bits, and when I did the transition was like "Finally, I can install shit" also my computer was newer and the OS worked better.
3So your PC was old (thus the new one faster) and its HW no longer supported by some software developers (because it was outdated and not enough users were on it anymore). The same can hold true if you have a 5 year old PC now. You didn't notice this due to going 64bit, you noticed it due to going away from a heavily outdated system.
13
The big shortcoming of 32 bit hardware was that it limits the amount of RAM in the computer to 4 GB. 64 bit is not inherently faster (for most things) but it enables up to 16 exabytes of RAM, an incomprehensible amount. Going to 128 bit would only be helpful if 16 exabytes wasn't enough.
15Slightly off topic, but the number of bits doesn’t necessarily describe the size of memory. For example most eight bit processors had 16bit data busses and address registers.
Some processors that were 32 bits internally have 24bit memory addressing.
7
We have 128 bit stuff in some places where it's advantageous, but in most cases there's not really a need. 64 bits already provides a maximum integer value of (+/-)9,223,372,036,854,775,807. Double it if you don't need negatives and drop the sign. There's little need in most cases for a bigger number, and cases that do either get 128 bit hardware, or can be handled by big number libraries.
13Similar to a modern 64 bit computer, my computer actually has a 512 bit wide ALU for SIMD, basically it lets you do the same operation on multiple numbers simultaneously.
10Riscv has a 128bit instruction set under proposal.
Nobody will ever use it, they know they'll never use it, it's stupid and impractical.
But maybe one day we want to connect every computer on the world together, we could probably use 128bit addresses so my computer could work on data in your computers memory.
There are ways to do this now, but maybe they make it simpler later.
7You'd be able to access more memory than we'd be able to physically build in the world.
6It’s hard to picture “128bit computing” in a general sense as ever being a thing. It’s just so far beyond anything we can realistically use now, plus would be inefficient/wasteful on most ordinary tasks.
Put this together with the physical limits to Moore’s law and current approaches to at least mobile computing ……
I picture more use of multi-core, specialty core, system on a chip. Some loads, like video, benefit from wide lanes, huge bandwidth, addresses many things at once, and we have video cores with architectures more suited for that. Most loads can be done with a standard compute core, and it is unnecessary, maybe counterproductive to move up to 128bit. If we want efficiency cores, like some mobile already have, 128bit is wrong/bad/inefficient. We’ll certainly have more AI cores, but I have no idea what they need
If you can forgive the Apple boosterism and take this as a general trend, see the focus on fast interconnections to many specialty cores. Each core has a different architecture and different needs
— https://www.apple.com/newsroom/2023/06/apple-introduces-m2-ultra/
6not even an apple thing isn't this just how SOCs work in general? definitely something intel and amd should be doing though (if they aren't already i dont honestly know) especially with hardware decoders and ML cores and whatnot
1Yes, this is how SoC can work. I think it is a great description about one specific company emphasizing a balance of different cores to do different jobs, rather than trying to make many general cores attempting to do everything. However, don’t get distracted by all the marketing language or that this is a company that people love to hate
2
5Contrary to some misconceptions, these SIMD capabilities did not amount to the processor being "128-bit", as neither the memory addresses nor the integers themselves were 128-bit, only the shared SIMD/integer registers. For comparison, 128-bit wide registers and SIMD instructions had been present in the 32-bit x86 architecture since 1999, with the introduction of SSE. However the internal data paths were 128bit wide, and its processors were capable of operating on 4x32bit quantities in parallel in single registers.
12
In fact, your computer is already capable of processing more than 64 bits at once using SIMD instructions. Many applications or things you don't suspect may or are already using them, including games.
4I came here to leave a snarky comment but then I read this thread. Now I feel sad and really confused.
1I’ve been thinking about this, in conjunction with quantum computing and AGI for some time.
If AI follows anything like Moore’s Law, I don’t think it will be too long (decades) before we’ll be breaching the limit of 64bit, and we would need to go one way or another.
That said, based on the above, 99%+ of our existing jobs would have likely been fully automated, so the world would likely be a very different place, and 128bit computing likely would stand out as too big of an achievement.
-4