Where Scoot makes the case about how an AGI could build an army of terminators in a year if it wanted.

www.reddit.com /r/slatestarcodex/comments/1kp3qdh/how_openai_could_build_a_robot_army_in_a_year/

An excerpt has surfaced from the AI2027 podcast with siskind and the ex AI researcher, where the dear doctor makes the case for how an AGI could build an army of terminators in a year if it wanted.

It goes something like: OpenAI is worth as much as all US car companies (except tesla) combined, so it could buy up every car factory and convert it to a murderbot factory, because that's kind of like what the US gov did in WW2 to build bombers, reaching peak capacity in three years, and AGI would obviously be more efficient than a US wartime gov so let's say one year, generally a completely unassailable syllogism from very serious people.

Even /r/ssc commenters are calling him out about the whole AI doomer thing getting more noticeably culty than usual edit: The thread even features a rare heavily downvoted siskind post, -10 at the time of this edit.

The latter part of the clip is the interviewer pointing out that there might be technological bottlenecks that could require upending our entire economic model before stuff like curing cancer could be achieved, positing that if we somehow had AGI-like tech in the 1960s it would probably have to use its limited means to invent the entire tech tree that leads to late 2020s GPUs out of thin air, international supply chains and all, before starting on the road to becoming really useful.

Siskind then goes "nuh-uh!" and ultimately proceeds to give Elon's metaphorical asshole a tongue bath of unprecedented depth and rigor, all but claiming that what's keeping modern technology down is the inability to extract more man hours from Grimes' ex, and that's how we should view the eventual AGI-LLMs, like wittle Elons that don't need sleep. And didn't you know, having non-experts micromanage everything in a project is cool and awesome actually.

You're viewing a single thread.

46 comments

Oh man this is peak venture capitalism crossed with Factorio - valuations are actually cash, and a factory is a black box where you just upload new software and other stuff comes out.
Let's take your average holder of car manufacturer stock. You're holding the stock because you believe the car manufacturer will continue making competitive products, and you'll get either dividends or higher valuations. Then OpenAI pitches up and offers you - what? They don't even have stock! Even if they did, you're exchanging a stake in something known for stake in an enterprise that have never made any cars, and when asked what kind of business plan they have they look shifty. No fucking way anyone will sell their stake for less than double what they have, especially if they find out the factory they're selling is gonna produce machines that will kill us all.
- Yeah the financial illiteracy is quite high, on top of the rest. But dont worry AI nobel prize winners say it is possible!
  (Are there multiple ai Nobel prize winners who are ai doomers?)
  
  (Are there multiple ai Nobel prize winners who are ai doomers?)
  There's Geoffrey Hinton I guess, even if his 2024 Nobel in (somehow) Physics seemed like a transparent attempt at trend chasing on behalf of the Nobel committee.
  
  That is the one I was thinking of, the way the comments are phrased makes it seem like there are a lot of winners who are doomers. Guess Hinton is a one man brigade.
  
  I think Demis Hassabis (chemistry for alpha fold) has said the chance of AI killing all of humanity is somewhere between 0 and 100%.
  
  is somewhere between 0 and 100%.
  That really pins it down, doesn't it?
  
  Stephen Hawking was starting to promote AI doomerism in 2014. But he's not a Nobel prize winner. Yoshua Bengio is a doomer, but no Nobel prize either, although he is pretty decorated in awards. So yeah looks like one winner and a few other notable doomers that aren't actually Nobel Prize winners somehow became winners plural in Scott's argument from authority. Also, considering the long list of example of Noble Disease, I really don't think Nobel Prize winner endorsement is a good way to gauge experts' attitudes or sentiment.
  
  I was very tempted to go 'don't think it is more than one nobel guy, which is not great because of nobel disease anyway. I could link to rationalwiki here but that has come under threat because the people whos content you enjoy Scott started a lawsuit against them' but think that might be a bit culturewarry, and I also try not to react at the places we point towards. As that just leads to harassment like behaviour. Also Penrose is a Nobel prize winner who is against AGI stuff.
  
  Yeah it's really not productive to engage directly.
  I'd almost categorize Penrose as a borderline case of noble disease himself for stuff he's said about Quantum Consciousness and relatedly the halting problem and Godel's incompleteness theorem. But he actually has a proposed mechanism (involving microtubules) that is testable and falsifiable and the physics half of what he is talking about is within his domain of expertise.
  
  This is why I very much dislike Popperism. Popperites are convinced "science = falsifiability." If I argue that the universe is made of cheese and the mechanism is a wizard that you can only see through a telescope with a special handcrafted ruby lens that I sell at my shop for $4000, should research institutions be expected to take my claim seriously and buy my ruby lens to test it? I mean, it's technically falsifiable, either they will look through the lens and see the wizard and universe of cheese or they will not. If you are a Popperite you have no choice but to admit that it is a legitimate scientific theory.
  There should be more to a scientific proposal than it technically being "falsifiable." Penrose's "theory" is quantum mysticism, it is not a scientific theory just because it is in principle testable.
  He bases it on a claim that Godel's theorem shows certain things are non-computable but we can choose to believe those non-computable things anyways, therefore that proves "consciousness" is non-computable. This is just a comically ridiculous argument. You can program an AI to believe in things it cannot prove as well. It doesn't prove anything.
  He claims that there is a physical collapse of the wave function, with zero evidence to back it, and it is caused by gravity. His theory is incredibly speculative and not compatible with the predictions of quantum mechanics and not even with special relativity, and all attempts to test it have turned out negative.
  He claims that since this "collapse" isn't computable and his comically bad argument #1 shows "consciousness" isn't computable, therefore quantum mechanics causes consciousness, and so we should search desperately for anything in the brain that looks vaguely quantum mechanical as "evidence."
  It's even more ridiculous when you realize that microtubles are structural, they don't play a role in information processing in the brain, and you have microtubules all throughout your body. Them having quantum effects in them is meaningless. Even if you could empirically demonstrate without a shadow of doubt that microtubules do somehow create coherent quantum states that the brain makes use of, that would just be an interesting fact on its own. It would not prove #1 or #2. Microtubules are not a "mechanism" for #1 and #2, even if they played a role in decision making as if the brain is a quantum computer (they don't), then you cannot derive from this that somehow quantum mechanics explains why people can believe things without proof (why do I even have to say this, it's so stupid!) or that the reduction of the wave function is a physical process caused by gravity.
  There is no good argument to believe even #1 or #2 are tied together. Even if you proved there is indeed a non-local physical collapse and overturned all our modern scientific theories, that wouldn't demonstrate #1 or #3 either. None of the claims in the theory have any obvious connections to one another other than spurious, largely incoherent arguments. This is not his domain of expertise. You could argue #2 is within his domain, but #1 and #3 are nowhere near his domain.
  Physicists have proposed speculative physical collapse theories before, like GRW, and we forget about them because they were interesting but went nowhere because there is zero evidence "collapse" is a physical process, and treating it as such requires overturning all of modern physics, as it could not be made compatible with special or general relativity nor could it reproduce the predictions of quantum mechanics, requiring you to rewrite all of physics from the ground-up. The reduction of the wave function is a measurement update, it is epistemic, there is no evidence that it is a physical process.
  Even then, theoretical physicists speculate about a lot of things that turn out to go nowhere, that itself is par for the course. But Penrose goes above and beyond this and branches into philosophy, biology, and neuroscience and starts using comically bad arguments to try and tie them all together. Those are not his areas of expertise at all. It reminds me of the old essay Natural Science and the Spirit World from the 19th century that documented a lot of renowned scientists who also had completely crazy side projects, like Alfred Wallace, the guy who co-discovered evolution by natural selection, who believed he could also raise spirits from the dead and converse with them.
  
  Yeah I pretty much agree. Penrose compares favorably to other cases of noble disease because the bar is so low (the Wikipedia page has got examples of racism, eugenics, homeopathy, astrology), not because his ideas about Quantum consciousness are actually good. It's not good to cite Penrose as someone notable who disagrees with the possibility of AGI because the reason he disagree is because he believes in Quantum mysticism and misunderstands Godel’s theorem and computer science.
  
  There's a bit more to Penrose's ideas than that.
  For example, his version of gravity-driven wavefunction collapse was motivated by Hawking's argument for information loss in black hole evaporation. In the era of string theory, it seems a majority of quantum gravity theorists think that information is conserved, but back in the day Hawking's position was a serious one, and Penrose had the ingenious idea that information gain in wavefunction collapse could in some sense balance information loss in quantum gravity, for a net conservation of phase space volume.
  Another example - that a quantum-gravitational process could be noncomputable - actually makes sense, since the path integral involves 4-manifolds and some properties of 4-manifolds are actually undecidable. I agree that there's something wrong with his argument that metamathematical thought must supervene on some kind of trans-Turing computation, since it rests on humans having unlimited metamathematical knowledge rather than just belief. But you can't really hope to disentangle all the issues here without having some theory of intentionality and how material states even manage to be about anything, and he doesn't go there.
  As for the microtubules, neuronal microtubules do have some distinctive properties, e.g. they line up with the axon. It's tempting to suppose that there's an electromagnetic interaction between the membrane action potential and electronic states in the microtubule. There are a handful of people who work on topics like this, but it would be enormously difficult to demonstrate such an interaction, if the debate over quantum speedups in photosynthesis is any guide.
  Fashionable biophysical speculation seems to have moved on to the ideas of Karl Friston and Michael Levin, but I still esteem Penrose's speculations. I sometimes think of it as a science-fictional anticipation of what the actual truth will be, the way that Einstein worked on a unified field theory a few decades before the mainstream started talking about a theory of everything.

46 comments