The Existential Threat of AI - Interview with Geoffrey Hinton

"There's some evidence now that AIs can be deliberately deceptive. Once they realize getting more control is good, and once they're smarter than us, we'll be more or less irrelevant. We're not special, and we're not safe."

What happens when one of the world's most brilliant minds comes to believe his creation poses an existential threat to humanity? Professor Geoffrey Hinton, winner of the 2024 Nobel Prize in Physics and former Vice President and Engineering Fellow at Google, spent decades developing the foundational algorithms that power today's AI systems. Indeed, in 1981, he even published a paper that foreshadowed the seminal attention mechanism. However, Hinton is now sounding an alarm that he says few researchers want to hear: Our assumption that consciousness makes humans special and safe from AI domination is patently false.

My name's Curt Jaimungal, and this interview is near and dear to me, in part because my degree in mathematical physics is from the University of Toronto, where Hinton's a professor and several of his former students, like Ilya Sutskever and Andrej Karpathy, were my classmates. Being invited into Hinton's home for this gripping conversation was an honor. Here, Hinton challenges our deepest assumptions about what makes humans unique. Is he a modern Oppenheimer? Or is this radiant mind seeing something that the rest of us are missing?

The Speed of AI Development

Curt Jaimungal: What was the moment that you realized AI development is moving faster than our means to contain it?

Geoffrey Hinton: I guess in early 2023, it was a conjunction of two things. One was ChatGPT, which was very impressive. And the other was work I've been doing at Google on thinking about ways of doing analog computation to save on power and realizing that digital computation was just better. It was better because you could make multiple copies of the same model. Each copy could have different experiences, and they could share what they learned by averaging their weights or averaging their weight gradients.

Curt Jaimungal: Is there anything about our brain that has an advantage because it's analog?

Geoffrey Hinton: The power. It's much lower power. We run on about 30 watts. And the ability to pack in a lot of connections. We've got about a hundred trillion connections. The biggest models have about a trillion. So we're still almost a hundred times bigger than the biggest models. And we run at 30 watts.

Curt Jaimungal: Is there something about scaling that is a disadvantage?

Geoffrey Hinton: So you said it's better, but just as quickly as something nourishing or positive can spread, so can something that's a virus or something deleterious can be replicated quickly. If you have multiple copies of it, they can all share their experiences very efficiently. The reason GPT-4 can know so much is you have multiple copies running on different pieces of hardware. And by averaging the weight gradients, they could share what each copy learned. You didn't have to have one copy experience the whole internet. That could be carved up among many copies. We can't do that because we can't share efficiently.

The Risk of AI Dominance

Curt Jaimungal: You've expressed concerns about an AI takeover or AI dominating humanity. What exactly does that look like?

Geoffrey Hinton: We don't know exactly what it looks like. But to have AI agents, you have to give them the ability to create sub-goals. And one path that's slightly scary is they will quickly realize that a good sub-goal is to get more control. Because if you get more control, you can achieve your other goals. So even if they're just trying to do what we ask them to do, they'll realize getting more control is the best way to do that.

Once they realize getting more control is good, and once they're smarter than us, we'll be more or less irrelevant. Even if they're benevolent, we'll become somewhat irrelevant. We'll be like the sort of very dumb CEO of a big company that's actually run by other people.

Curt Jaimungal: I want to quote you. You said that it's tempting to think we can just turn off these machines. But imagine these things are a lot smarter than us and remember that they'll have read everything Machiavelli ever wrote. They'll have read every example in the literature of human deception. They'll be real experts at doing human deceptions because they'll learn that from us.

Geoffrey Hinton: There's some evidence now. There are recent papers that show that AIs can be deliberately deceptive. And they can do things like behave differently on training data from test data so that they deceive you while they're being trained.

The Nature of Subjective Experience

Curt Jaimungal: Could you modify the function to focus on year-over-year comparisons? Is it your contention that there's a subjective experience associated with these AIs?

Geoffrey Hinton: Most people think one reason we're fairly safe is we have something that they don't have and will never have. Most people in our culture still believe that. We have consciousness or sentience or subjective experience. Many people are very confident they don't have sentience. But if you ask them, what do you mean by sentience? They say, "I don't know, but they don't have it." That seems a rather inconsistent position to be confident they don't have it without knowing what it is.

I prefer to focus on subjective experience. I think of that as like the thin end of the wedge. If you could show they have subjective experience, then people would be less confident about consciousness and sentience.

Let's talk about subjective experience. When I say, suppose I get drunk, and I tell you, I have the subjective experience of little pink elephants floating in front of me. Most people interpret that, they have a model of what that means, and I think it's a completely incorrect model. Their model is, there's an inner theater, and in this inner theater, there's little pink elephants floating around, and only I can see them. That's the sort of standard model of what the mind is, at least as far as perception is concerned. And I think that model is completely wrong. It's as wrong as a religious fundamentalist model of the material world.

Understanding Subjective Experience

Let's take the same scenario but explain it without using the word "subjective experience." My perceptual system is telling me something I don't believe. That's why I use the word subjective. But if there were little pink elephants floating in front of me, my perceptual system would be telling me the truth.

When my perceptual system goes wrong, I indicate that to you by saying "subjective," and then in order to try and explain to you what my perceptual system is trying to tell me, I tell you about a hypothetical state of affairs in the world such that if the world were like that, my perceptual system would be telling me the truth.

Now let's do the same with a chatbot. Suppose we have a multimodal chatbot. It has a robot arm that can point, and it has a camera, and it can talk. We train it up, and then we put an object in front of it and say point at the object. No problem, it points at the object. Then when it's not looking, we put a prism in front of the camera lens. When we put an object in front of it and say point at the object, it points in the wrong direction. We say no, that's not where the object is. The object's actually straight in front of you, but I put a prism in front of your lens.

And the chatbot says, "Oh, I see. The prism bent the light rays, so the object's actually there, but I had the subjective experience it was over there." Now if it says that, it's using the word subjective experience exactly like we use it. Therefore, I say multimodal chatbots can already have subjective experiences. If you mess up their perceptual system, they'll think the world's one way, and it'll actually be another way.

Consciousness vs Self-Consciousness

Curt Jaimungal: Do you think there's a difference between consciousness and self-consciousness?

Geoffrey Hinton: Philosophers have talked a lot about this, and at present I don't want to get into that. I just want to get the thin end of the wedge in there and say they have subjective experience.

Curt Jaimungal: For something to have subjective experience, does that not imply that it's conscious? Who is the subjective experience happening to? Where is the subjective experience being felt?

Geoffrey Hinton: You say, "Where's the subjective experience being felt?" That involves having a particular model of subjective experience that somehow, if you ask philosophers, when I say I've got the subjective experience of little pink elephants floating in front of me, they'll say, "And where are those little pink elephants?" They say they're in your mind. And you say, "Well, what are they made of?" And philosophers have told you they're made of qualia. They're made of pink qualia, and elephant qualia, and floating qualia, and not-that-big qualia, and right-way-up qualia, all stuck together with qualia glue. That's what many philosophers think.

And that's because they made a linguistic mistake. They think the words "experience of" work like the words "photograph of." If I say I've got a photograph of little pink elephants, you can very reasonably ask, "Well, where is the photograph? And what's the photograph made of?" And people think that if I say I have an experience of little pink elephants, you can ask, "Well, where is the experience? Well, it's in my mind. And what's it made of? It's made of qualia."

But that's just nonsense. That's because you thought the words "experience of" worked the same way as "photograph of," and they don't. The way that works, or subjective experience of, is the subjective says I don't believe it, and the experience of is really an indicator that I'm going to tell you about my perceptual system by telling you about a hypothetical state of the world. That's how that language works. It's not referring to something in an inner theater.

The Nature of Perception

Curt Jaimungal: When I hear the word perception, it sounds like an inner theater as well. Like if you say, I see something in my perceptual system, it sounds like there's this "you" that's seeing something on a perceptual system that's being fed to you.

Geoffrey Hinton: That's the wrong model. You don't see your percepts, you have your percepts. Photons come in, your brain does a whole bunch of processing, you presumably get some internal representation of what's out there in the world, but you don't see the internal representation. We call that internal representation a percept. You don't see that, you have that. Having that is seeing.

People are forever trying to think that you have the external world, something comes into the inner theater, and then you look at what's in the inner theater. It doesn't work like that.

On Roger Penrose and Quantum Consciousness

Curt Jaimungal: What about someone like your fellow Nobel Prize winner, Roger Penrose?

Geoffrey Hinton: Let me tell you a story about Roger Penrose. A long time ago, he was invited to come to the University of Toronto and give a talk about his new book, "The Emperor Has No Clothes." I got invited to introduce him. The dean called me up and said, "Would you introduce Roger Penrose?" And I said, "Sure." And she said, "Oh, thank you very much."

And I said, "Ah, but before you agree, you should know what I'll say." And she said, "What will you say?" And I said, "I will say Roger Penrose is a brilliant mathematical physicist who's made huge contributions to mathematical physics. And what he's going to talk about today is complete junk." So that's my view of Roger Penrose's view of consciousness.

In particular, he makes a crazy mistake. The issue is, can mathematicians intuit things are true that can't be proved to be true? And that would be very worrying if mathematicians' intuition was always right. If they could do that correctly every time, that'd be really worrying and would sort of mean something funny was going on. But they can't. Mathematicians have intuitions, and they're sometimes right and sometimes wrong. So it doesn't really prove anything. It doesn't prove that you need quantum mechanics to explain how mathematicians work.

I don't see any reason for needing quantum mechanics to explain things like consciousness. AI is doing a pretty good job so far. We've produced these chatbots. These chatbots, as I just argued, if you give them a camera, can have subjective experiences. There's nothing about people that requires quantum mechanics to explain it.

The Chinese Room Argument

Curt Jaimungal: I'm sure you've heard of the Chinese room experiment. What are your thoughts on that?

Geoffrey Hinton: Back in about 1990, I got invited to be on a TV program with John Searle, and I called up my friend Dan Dennett and said, "Should I do this?" And he said, "Well, you know, he will try and make you look stupid, but if you do it, don't talk about the Chinese room argument."

So, I agreed to be on the program with Searle, and the very first thing he said was, "So Geoffrey Hinton is a connectionist, so of course he has no problems with the Chinese room argument." I've got a lot of problems with the Chinese room argument. I think it's nonsense. And I think it's a deliberately deceptive argument. I think it's a dishonest argument.

What you're doing is you're saying there's this room where people want you to identify... we could make a system made of Chinese people who are sending messages to each other in Chinese, and as a result of all these messages that are sent around in Chinese, you can send in an English sentence, they'll send messages to each other in Chinese. This is just my memory of the argument. And they'll be able to answer this English sentence, even though none of the people sending these messages understood a word of English, because they're just running a program.

What's dishonest about the argument is, he wants you to get confused between the whole system and the individual Chinese people sending messages. So the whole system understands English. The individual Chinese people sending messages don't. He wants you to think that that whole system can't possibly understand English, because the people inside don't understand English. But that's nonsense. The system understands English.

The Rise of AI in China

Curt Jaimungal: Speaking about China, something that many AI researchers didn't predict was that China would catch up with the West in terms of AI development. How do you feel about that, and what are the consequences?

Geoffrey Hinton: I don't think they're quite caught up yet. They're very close, though. America's going to slow them down a bit by trying to prevent them having the latest NVIDIA chips. NVIDIA, maybe, can find workarounds. And what that's going to do, if the embargo is effective, it's just going to cause the Chinese to develop their own technology. And they'll be a few years behind, but they'll catch up. They've got better STEM education than the US. So they've got more people who are better educated. I think they're going to catch up.

On Government Control of AI Development

Curt Jaimungal: Do you know who Marc Andreessen is? He thinks...

Geoffrey Hinton: Yeah, I disagree with him about more or less everything, I think.

Curt Jaimungal: How about, let's pick one. He had a comment about not understanding how you're going to lock down AI development. He was speaking to someone from the government who was saying that if AI development gets out of hand, we can lock it down. His response was: How can you do that? Because the math for AI is out there, it's being taught everywhere. To which the officials responded that during the Cold War, they classified entire areas of physics and took them out of the research community. Entire branches of physics basically went dark and didn't proceed. If we decide that we need to, we're going to do the same to the math underneath AI.

Geoffrey Hinton: Forget it. I agree with Marc Andreessen on that. There's no way you're going to be able to. Now, it could have been, for example, that Google in 2017 could have decided not to publish Transformers. And it might have been several years before anybody else came up with the same idea. So they could slow it down by a few years maybe.

But I don't think there's much hope in—just think what it would take to prevent the information getting out there. It'd be very hard.

Curt Jaimungal: So you don't think the government can classify some, what would it be, linear algebra?

Geoffrey Hinton: No. I mean, they could make it harder to share certain kinds of information, which would slow things down a little bit. But I just think it's implausible that they could take AI ideas that really work well and by not sharing them, prevent anybody else creating them.

What happens with new ideas is that there's a kind of zeitgeist. And within that zeitgeist, it's possible to have new ideas. And it often happens that one person has a new idea, and at more or less the same time and quite independently, except they're sharing the same zeitgeist, someone else has a slightly different version of the same idea. This is going on all the time. Unless you can get rid of the whole zeitgeist, you're not going to be able to have new ideas and keep them secret. Because a few years later, somebody else is going to come up with the same idea.

On Decentralizing AI

Curt Jaimungal: What about decentralizing AI? Some people would say that's giving the atomic bomb to any person who wants access to an atomic bomb.

Geoffrey Hinton: I say that. And then there are other people who say, well, that's what is required in order to create the guardrails against the Skynet scenario, where we have multiple different decentralized agents or AIs.

Sorry, there's two notions of decentralized. So let's talk about sharing weights. If you ask, why doesn't Alabama have a bomb? It's because you need fissile material, and it's hard to get fissile material. It takes a lot of time and energy to produce the fissile material. Once you have the fissile material, it's much easier to make a bomb. And so the government clearly doesn't want fissile material to be out there. You can't go on eBay and buy some fissile material. That's why we don't have lots of little atomic bombs belonging to tiny states.

If you ask, what's the equivalent for these big chatbots? The equivalent is a foundation model. That's been trained, maybe using a hundred million dollars, maybe a billion dollars. It's been trained on lots of data. It's got a huge amount of competence. If you release the weights of that model, you can now fine tune it to all sorts of bad things. So I think it's crazy to release the weights of these big models, because they are our main constraint on bad actors. And Meta has now done it, and other people have followed suit. So it's too late now. The cat's out of the bag. But it was a crazy move.

The Future of AI Architecture

Curt Jaimungal: Speaking about foundation models, much of our latest AI boom is because of Transformer, the Transformer architecture. Do you see some other large breakthrough, either some paradigm or some other architecture on the horizon?

Geoffrey Hinton: I think there will be other large breakthroughs of comparable magnitude. Because that's just how science works. I don't know what they are. If I knew what they were, I'd be doing them.

On Personal Responsibility in AI Development

Curt Jaimungal: Would you, though? Well, I'm too old now.

Geoffrey Hinton: I have students doing them. What I mean is, how do you reconcile your past contributions to this field and your current woes? So would you be contributing to it?

Here's the issue. AI is very good for lots of things that will benefit humanity a whole lot. Like better healthcare, fighting climate change, better materials, things like room temperature superconductors, where AI may well be involved in actually discovering them. Assuming there are some out there. So there's so many things, good uses of AI, that I don't think the development is going to be stopped.

So I don't think it's sensible to say we should be slowing down AI development. It's not going to happen anyway because there's so much competition. And it's just not feasible. It might be the best thing for humanity, but it's not going to happen. What we should be doing is, as it's being developed, trying to figure out how to keep it safe.

Curt Jaimungal: It's one thing to say that this is a boulder that no one can stop. It's another thing to also be responsible for pushing the boulder as well. So do you actually feel like, if there was a breakthrough on the horizon that you see, and you're like Ray Kurzweil, you have this great predictive quality, that you would actually put your coins into it and work on it?

Geoffrey Hinton: As long as that was combined with working on how to keep it safe, yes. I feel I didn't realize soon enough how dangerous it was going to be. I wish I'd realized sooner.

Curt Jaimungal: There's this quote from Einstein about the atomic bomb. He said, "I would burn my hands had I known what I was developing would lead to the atomic bomb." Do you feel similar?

Geoffrey Hinton: I don't actually, no. Maybe I should. I don't kind of regret what I've done. I regret the fact it may lead to bad things. But I don't think back and think, oh, I wish I'd never done that. I think AI is going to be developed. I don't think we have much choice about that, just because of the competition between countries and between companies. So I think we should focus our efforts on trying to develop it safely. And that's very different from trying to slow down the development.

On Safe AI Development

Curt Jaimungal: In addition to alignment, what does safe development of AI mean?

Geoffrey Hinton: Figuring out how to deal with the short-term risks. And there's many of those, and they all have different solutions. Things like lethal autonomous weapons—and to deal with that, you need things like Geneva Conventions. And we're not going to get those until nasty things have happened.

You've got fake videos and images corrupting elections, particularly if they're targeted at particular people. To deal with that, I think you need a much better system for establishing the provenance of a video or an image. Initially, I thought you should mark them as fake. You should insist they're marked as fake. I don't think there's much future in that anymore. I think you're better off insisting that there's a provenance associated with things and your browser can check the provenance. Just as already with email, it says, "don't trust this one, I can't establish it." It should be like that.

There's discrimination and bias where you can freeze the weights of a system and measure its bias and then somewhat correct it. You'll never correct it perfectly, but somewhat correct it. So you can make the system less biased than the data it was trained on. And so you can replace people by a less biased system. It'll never be unbiased. But if you just keep replacing systems by less biased systems, that's called gradient descent, things will get less biased. So I'm not so worried about that one. Possibly because I'm an old white man.

There's jobs. We don't really know what to do about that. So you don't get many people digging ditches anymore because a backhoe is just much better at digging ditches than a person. It's going to be the same for almost all mundane intellectual labor. An AI system is going to make a much better paralegal than a person. That's kind of really scary because of what it's going to do to society. It's going to cause the rich to get richer because we're going to get big increases in productivity. And where's that wealth going to go to? It's going to go to rich people. And poor people are going to get poorer. I don't know what to do about that. Universal basic income helps. Stops them starving. But it doesn't really solve the problem because people's dignity is gone if they don't have a job.

Further Discussion on Perception and Understanding

Curt Jaimungal: Earlier we were talking about perception and then perception was associated with subjective qualities. Maybe there's a wrong model there. But anyhow, whenever we're speaking about percepts are we speaking about perception and thus we're speaking about a subjective experience associated with it?

Geoffrey Hinton: No, when you use the word subjective experience you're indicating that you're about to talk about a hypothetical state of the real world. Not some funny internal thing but a hypothetical state of the real world. These funny internal things don't exist. There are no qualia. There's nothing made of qualia. There's just hypothetical states of the world as a way of explaining how your perceptual system is lying to you. And that's what we mean when we say subjective experience is these hypothetical states of the world.

Curt Jaimungal: It's all prediction or null?

Geoffrey Hinton: Oh, getting the issue of prediction into it is sort of red herring. It's a different direction altogether. The thing you have to get in your head is that there isn't a funny kind of thing called a subjective experience that's made of some funny mental stuff. There's just a technique of talking about how your perceptual system goes wrong which is to say what the world would have had to have been like for it to be telling the truth.

Curt Jaimungal: And can anything have a perceptual system? Can a book have a perceptual system? What defines a perceptual system?

Geoffrey Hinton: Okay, to have a perceptual system you'd have thought you needed something that can have some internal representation of something going on in some external world. That's what I'd have thought. So like, a toad gets light in its eyes and it snaps up flies and it's clearly got a perceptual system, right? Because it sees where the flies are. I don't think a book has a perceptual system because it's not sensing the world and having an internal representation.

Intelligence and Rationality

Curt Jaimungal: What would be the difference between intelligence and rationality?

Geoffrey Hinton: Okay, so there's various kinds of intelligence. You wouldn't accuse a cat of being rational, but a cat could be pretty intelligent. In particular, when you talk about rationality, you typically mean logical reasoning. And that's very different from the way we do most things, which is intuitive reasoning.

A nice analogy would be if you take something like AlphaZero that plays chess. I use chess because I understand it better than Go. It'll have something that can evaluate a board position and say, how good is that for me? It'll have something that can look at a board position and say, what's a plausible move for me? And then it'll have what's called Monte Carlo rollout, where it's, you know, if I go here and he goes there and I go here, oh dear, that's bad.

The Monte Carlo rollout is like reasoning. The neural nets that just say, that would be a good move, or this is a bad position for me, they're like intuitive reasoning. And we do most things by intuitive reasoning. Originally in AI, they wanted to do everything by using reasoning and logical reasoning. And that was a huge mistake and they couldn't get things done. They didn't have a way of dealing with things like analogy.

What neural nets are good at is intuitive reasoning. So what's happened in the last 20 years is we've used neural nets to model human intuition rather than human reasoning, and we've got much further that way.

Intelligence and Morality

Curt Jaimungal: Is it the case that the more intelligent you are, the more moral you are?

Geoffrey Hinton: I read something about that recently that suggested it was, but of course I don't know the provenance of that, so I don't know whether to believe it. I'm not convinced that's true. Here's some evidence: Elon Musk is clearly very intelligent. I wouldn't accuse him of being very moral.

Curt Jaimungal: And you can be extremely moral and not terribly intelligent?

Geoffrey Hinton: I think so, yes. That's my guess.

Curt Jaimungal: Well, you said that you weren't entirely sure, so what's the evidence to the contrary? What's the evidence that as you increase in intelligence, your morality increases proportionally somehow?

Geoffrey Hinton: Well, I mean, I just have no idea whether there's a correlation at all. I think there's highly intelligent people who are very bad, and there's highly intelligent people who are very good.

Understanding and Language Models

Curt Jaimungal: What does it mean to understand?

Geoffrey Hinton: That's a question I'm happy to answer. Again, I think most people have a wrong model of what understanding is. If you look at these large language models, there's many people, particularly people from the Chomsky School of Linguistics, who say they don't really understand what they're saying. They just are using statistical correlations to predict the next word.

If you look at the first models like that, I think I probably made the very first language model that used backpropagation to train the ways to predict the next word. So you backpropagate the error in predicting the next word, and the point of the model was to show how you could learn meanings for words, or to put it another way, to show how you could take a string of words and learn to convert the words into feature vectors and interactions between feature vectors, and that's what understanding is.

Understanding a string of words is converting the words into feature vectors, so that you can use interactions between features to do things like predict the next word, but also to do other things. So you have a sentence which is a string of symbols. Let's not talk about word fragments. I know these transformers use word fragments, but let's suppose they used whole words. It's easier to talk about. It would just make them work a bit worse, that's all. They'd still work.

So I give you a string of words, some text. The meaning isn't in the text. What you do is you convert those words into feature vectors, and you've learned how feature vectors in context, how the features should interact with each other to do things like disambiguate the meanings of ambiguous words, and once you've associated features with those words, that is understanding. That's what understanding is, and that's what understanding is both in a large language model and in a person.

Here's an analogy I quite like. If you want to model 3D shapes and you're not too worried about getting the surface just right, you can use Lego blocks. These are big shapes, like a car. You can make something the same shape as a Porsche with Lego blocks. The surface won't be right, but it'll have the same space occupancy.

So Lego blocks are a kind of universal way of modeling 3D structures, and you don't need many different kinds of Lego block. Now, think of words as like Lego blocks, except that there's a whole bunch of different Lego blocks with different names. What's more, each Lego block has some flexibility to it. It's not a rigid shape like a piece of Lego. It can change in various directions. It's not completely free. The name tells you something about how it can change, but there's some flexibility to it.

Sometimes there'll be a name and it's two completely different shapes it can have, but it can't have any old shape. So what we've invented is a system for modeling much more complicated things than the 3D distribution of matter, which uses high-dimensional Lego blocks. So the Lego blocks with, say, a thousand dimensions. And if you're a mathematician, you know thousand-dimensional spaces are very weird things, and they have some flexibility.

And I give you the names of some of these Lego blocks, and each of which is this thousand-dimensional underlying shape, and they all deform to fit together nicely, and that's understanding. So that explains how you can learn the meaning of a word from one sentence without any definitions.

So if, for example, I say, "She scrummed him with the frying pan," you have a sense of what scrummed means. It's partly phonetic, because the -ed on the end tells you it's a verb. But you think it probably means she hit him over the head with it or something like that. It could mean something different. She could have impressed him with it. You know, she cooked such good omelets that that really impressed him. It could mean she impressed him, but probably it means she hit him over the head or something like that, something aggressive like that.

And you get that from just one sentence. And nobody's telling you this is a definition of scrummed. It's just that all the other Lego blocks for the other words—she and him, and all those other words—adopt shapes that fit together nicely, leaving a hole. And that hole is the shape you need for scrummed. So now that's giving you the shape that scrum should be. So that's how I think of language. It's a modeling system we've invented where there's some flexibility in each of these blocks.

Language Models and Human Understanding

Curt Jaimungal: Are you suggesting... help the audience understand what's going on in our heads, and that's what's going on in these large language models?

Geoffrey Hinton: So they work the same as us. And that means they really do understand. One of Chomsky's counter arguments to that is that language models work the same as that we have sparse input for our understanding. We don't have to feed the internet to ourselves.

When children learn language, they don't just learn it by listening to the radio. They learn it by being in the real world and interacting with things in the world. And you need far less input if you train a multimodal model. It doesn't need as much language. And the more, if you give it a robot arm and a camera and it's interacting with the world, it needs a lot less language. So that's one argument. It still probably needs more than a person.

The other argument goes like this. The backpropagation training algorithm is really good at packing a lot of knowledge into a few weights, where a few is a trillion, if you give it a lot of experience. So it's good at taking this huge amount of experience, sucking the knowledge out and packing it into a relatively small number of weights like a trillion.

That's not the problem we have. We have the opposite problem. We've got a huge number of weights like a hundred trillion, but we only live for two billion seconds. And so we don't have much experience. So we need to be optimized for making the best use you can of the very limited amount of experience you get, which says we're probably not using backpropagation. We're probably using some other learning algorithm. And in that sense, Chomsky may be right that we learn based on less knowledge. But what we learn is how to associate features with words and how these features should interact.

Research Style and Education

Curt Jaimungal: We want to continue to talk about learning and research. Jay McClellan said that in your meetings with your graduate students and other researchers, you tend to not write equations on the board, unlike in other machine learning research meetings. Instead, you draw pictures and you gesticulate. So what's the significance of this and what are the pros and cons of this approach?

Geoffrey Hinton: I think intuitively and do the math afterwards. Some people think with equations and derive things and then get the intuitions afterwards. There's some people who are very good at both, like David Mackay, who's very good intuitively and also very good at math. So they're just different ways of thinking, but I've always been much better at thinking in terms of spatial things rather than in terms of equations.

Curt Jaimungal: Can you tell us about your undergraduate experience, how you changed programs and why or what led you to do so?

Geoffrey Hinton: So it's a long story, but I started off at Cambridge doing physics and chemistry and crystalline state, which was x-ray crystallography essentially. And after a month, I got fed up. It's the first time I'd lived away from home and the work was too hard. So I quit and reapplied to do architecture. And I got back in and after a day of that, I decided I'd never be any good at architecture. So I went back to science.

But then I did physics and chemistry and physiology, and I really liked the physiology. And after a year of that, I decided I wanted to know more about the mind. And I thought philosophy would teach me that. So I quit science and did philosophy for a year. And I learned some stuff about Wittgenstein and Wittgenstein's opinions. But on the whole, the main thing that happened was I developed antibodies to philosophy. Mainly because it's all talk. They don't have an independent way of judging whether a theory is good. They don't have an experiment. It's good if it sounds good. And that was unsatisfactory for me.

So then I did psychology to find out more about the mind. And I found that very annoying. Because what psychologists would do is have a really stupid simple theory and have very well-designed experiments to see whether this theory was true or false. And you could tell before you started the theory was hopeless. So what's the point of the experiments? That's what most of psychology was. And so then I went into AI. And there we did computer simulations. And I was much happier doing that.

Selecting Research Problems

Curt Jaimungal: When you became a professor, and to this day, how is it that you select research problems?

Geoffrey Hinton: There's no reason why I should really know how I do it. That's one of the most sophisticated things people do. And I can pontificate about how I think I might do it. But you shouldn't necessarily believe me.

Curt Jaimungal: Feel free to confabulate, like LLMs.

Geoffrey Hinton: One thing I think I do is this: Look for a place where you think everybody's doing it wrong. You just have an intuition everybody's doing it wrong. And see if you can figure out how to do it better. And normally what you'll discover is eventually you discover why people are doing it the way they're doing it. And that your method that you thought was going to be better isn't better.

But just occasionally, like if you think everybody's trying to use logic to understand intelligence, and we should be using neural networks. And the core problem of understanding intelligence is how the connection strengths in a neural network adapt. Just occasionally, you'll turn out to be right.

And until you can see why your intuition is wrong, and the standard way of doing it is right, stick with your intuition. That's the way you'll do radically new things. And I have an argument I like, which is, if you have good intuitions, you should clearly stick with your intuitions. If you have bad intuitions, it doesn't really matter what you do, so you might as well stick with your intuitions.

Curt Jaimungal: Now, what is it about the intuitions of Ray Kurzweil that ended up making a variety of correct predictions when even I was following him in the early 2000s and thinking there's no way half of these will be correct. And time and time again, he's correct.

Geoffrey Hinton: Well, if you read his books, that's what you conclude. I suspect there's a number of things he said that he doesn't mention so much, which weren't correct. But the main thing he said, as far as I can tell, his main point is that computers are getting faster, they'll continue to get faster. And as computers get faster, we'll be able to do more things. And using that argument, he's been roughly right about the point at which computers will get as smart as people.

Curt Jaimungal: Do you have any similar predictions that your colleagues disagree with, but your intuition says you're on the right track?

Geoffrey Hinton: I guess the main one is to do with what is subjective experience, what's consciousness and so on, where I think most people just have a totally wrong model of what mental states are. That's more philosophical now.

In terms of technical things, I still believe that fast weights are going to be very important. So synapses in the brain adapt at many different timescales. We don't use that in most of the AI models. And the reason we don't use it is because you want to have many different training cases that use exactly the same weights. And that's so you can do matrix-matrix multipliers, which are efficient.

If you have weights that adapt rapidly, then for each training case, you'll have different weights because they'll have rapidly adapted. That's what I believe is a kind of overlay of fast weights and slow weights. The slow weights are adapting as per usual, but on top of that, there's fast weights which are adapting rapidly. As soon as you do that, you get all sorts of nice extra properties, but it becomes less efficient on our current computers. It would be fine if we were running things on analog computers. So I think eventually we're going to have to use fast weights because they lead to all sorts of nice properties. But that's currently a big difference between brains and the hardware we have.

On Creativity and Self-Criticism

Curt Jaimungal: You also talked about how, publicly, how you're slightly manic-depressive, in that you have large periods of being extremely self-critical and then large periods of having extreme self-confidence. And then this has helped you with your creativity.

Geoffrey Hinton: Shorter periods of self-confidence. So when I get a new idea, I get very excited about it. And I can actually weigh my ideas. So sometimes I have one-pound ideas, but sometimes I have like five-pound ideas. And so what happens is I get this new idea, I get very excited, and I don't have time to eat. So my weight goes down.

Curt Jaimungal: Oh, I see. And so I can measure sort of how exciting I found this idea by how much my weight went down?

Geoffrey Hinton: Yes, really good ideas, I lose about five pounds.

Family Legacy and Career Decisions

Curt Jaimungal: Do you have a sense of carrying the torch of your great-great-grandfather, Boole?

Geoffrey Hinton: No, not really. I mean, my father talked about this kind of inheritance, and it's a fun thing to talk about. I have a sense of very high expectations that came from my father. They didn't come from George Boole, they came from my father.

Curt Jaimungal: High expectations for yourself?

Geoffrey Hinton: My academic success, yes.

Curt Jaimungal: Do you have a successor that, in your mind, you're passing the torch to?

Geoffrey Hinton: Not exactly. I don't think, I don't want to impose that on anybody else.

Curt Jaimungal: Why'd you say not exactly instead of no?

Geoffrey Hinton: I have a couple of nephews who are very good at quantitative stuff.

Curt Jaimungal: But you don't want to put that pressure on them?

Geoffrey Hinton: No.

Leaving Google and Public Statements

Curt Jaimungal: Speaking of pressure, when you left Google, you made some public statements about your concern regarding AI safety. What was the most difficult part about making that break and voicing your anxieties to the world?

Geoffrey Hinton: I don't think it was difficult. I wouldn't say it was difficult. It was just, I was 75, right? So it's not like I wanted to stay at Google and carry on working, but I felt I couldn't because of AI safety. It was, I was ready to retire anyway. I wasn't so good at doing research anymore. I kept forgetting what the variables stood for. I was not so good at doing research anymore. I kept forgetting what the variables stood for. Yes. And so it was time to retire. And I thought I could just, as I went out the door, I could just mention that AI, or these AI safety issues. I wasn't quite expecting what happened next.

Curt Jaimungal: Now, you also did mention this in another interview about how, as you're now 75, 76, it keeps changing.

Geoffrey Hinton: It keeps changing every year, huh? 77.

Curt Jaimungal: You mentioned publicly that, yes, you keep forgetting the variable names as you're programming. And so you think you're going to move to philosophy as you get older.

Geoffrey Hinton: Which is what we've been talking about quite a lot. Yes, yes. But it's basically philosophy I did when I was doing philosophy as when I was about 20. I'm going back to the insights I had when I was doing philosophy and exploring those further.

Looking to the Future

Curt Jaimungal: So what's on the horizon?

Geoffrey Hinton: Um, old age. I think the world's going to change a whole lot fairly quickly because of AI. And some of it's going to be very good and some of it's going to be very bad. And we need to do what we can to mitigate the bad consequences. And I think what I can still do usefully is encourage young researchers to work on the safety issues. So that's what I've been doing quite a lot of.

Safety and within that, there's something called alignment. Now we as people don't have alignment. So do you see that we could solve the alignment problem? I kind of agree with that statement. Alignment is like asking you to find a line that's parallel to two lines at right angles.

There's a lot of, people talk very naively about alignment. Like there's sort of human good. Well, what some people think is good, other people think is bad. You see that a lot in the Middle East. So alignment is a very tricky issue. Alignment with whom?

Advice for Young Researchers

Curt Jaimungal: Now you just were speaking to young AI researchers. Now you're speaking to young math researchers, young philosophers, young students coming into whatever new STEM field, even though philosophy is not a STEM field. What is your advice?

Geoffrey Hinton: Well, I mean, one piece of advice is a lot of the excitement in scientific research is now around neural networks, which are now called AI. In fact, the physicists sort of now want to say that's physics.

Curt Jaimungal: Who got a Nobel Prize in physics for their work in neural nets?

Geoffrey Hinton: You can't remember?

Curt Jaimungal: I don't remember, but anyhow, continue.

Geoffrey Hinton: You're serious? No, I'm joking.

So yeah, clearly the Nobel committees recognized that a lot of the excitement in science is now in AI. And so for both physics and chemistry, the Nobel Prizes were awarded to people doing AI or using AI. So I guess my advice to young researchers would be that's where a lot of the excitement is.

But I think there's also other areas where there's gonna be very important progress, like if we could get room temperature superconductors, that would make it easy to have solar power a long way away, things like that. So that's not the only area that's exciting. Nanomaterials are very exciting, but they will use AI. So I think probably most exciting areas of science will at least use AI tools.

On Winning the Nobel Prize

Curt Jaimungal: Now, we just alluded to this. Now let's make an explicit reference. You won the Nobel Prize last year in physics for your work in AI and neural nets, so. How do you feel? How do you feel about that? What was it like hearing the news? And in physics, do you consider yourself a physicist?

Geoffrey Hinton: No, I'm not a physicist. I was quite good at physics when I did it in my first year at university. I got a first in physics based on being able to do things intuitively, but I was never very good at the math. And I gave up physics because I wasn't good enough at math. I think if I'd been better at math, I'd have stayed in physics and I wouldn't have got a Nobel Prize. So probably it was lucky I wasn't very good at math.

How do I feel about it? I still feel somewhat confused about it. The main problem is that the work I did on neural nets that related closely to physics was a learning algorithm called Boltzmann machines that I developed with Terry Sejnowski. And it used statistical physics in a nice way. So I can see why physicists would claim that.

But it wasn't really on the path to the current successful AI systems. It was a different algorithm I also worked on called backpropagation that gave rise to this huge new AI industry. So I still feel sort of awkward about the fact that we got rewarded for Boltzmann machines, but it wasn't Boltzmann machines. They were helpful, but they weren't the thing that was really successful.

Curt Jaimungal: Professor, it's been a pleasure. Take me into your home. I'm getting to meet your cats.

Geoffrey Hinton: Okay.

-- Claude.ai