Jeff Hawkins: How brain science will change computing

I do two things:

I design mobile computers and I study brains.

Today's talk is about brains and -- (Audience member cheers)

Yay! I have a brain fan out there.

(Laughter)

If I could have my first slide,

you'll see the title of my talk and my two affiliations.

So what I'm going to talk about is why we don't have a good brain theory,

why it is important that we should develop one

and what we can do about it.

I'll try to do all that in 20 minutes.

I have two affiliations.

Most of you know me from my Palm and Handspring days,

but I also run a nonprofit scientific research institute

called the Redwood Neuroscience Institute in Menlo Park.

We study theoretical neuroscience and how the neocortex works.

I'm going to talk all about that.

I have one slide on my other life, the computer life,

and that's this slide here.

These are some of the products I've worked on over the last 20 years,

starting from the very original laptop

to some of the first tablet computers

and so on, ending up most recently with the Treo,

and we're continuing to do this.

I've done this because I believe mobile computing

is the future of personal computing,

and I'm trying to make the world a little bit better

by working on these things.

But this was, I admit, all an accident.

I really didn't want to do any of these products.

Very early in my career

I decided I was not going to be in the computer industry.

Before that, I just have to tell you

about this picture of Graffiti I picked off the web the other day.

I was looking for a picture for Graffiti that'll text input language.

I found a website dedicated to teachers who want to make script-writing things

across the top of their blackboard,

and they had added Graffiti to it, and I'm sorry about that.

(Laughter)

So what happened was,

when I was young and got out of engineering school at Cornell in '79,

I went to work for Intel and was in the computer industry,

and three months into that, I fell in love with something else.

I said, "I made the wrong career choice here,"

and I fell in love with brains.

This is not a real brain.

This is a picture of one, a line drawing.

And I don't remember exactly how it happened,

but I have one recollection, which was pretty strong in my mind.

In September of 1979,

Scientific American came out with a single-topic issue about the brain.

It was one of their best issues ever.

They talked about the neuron, development, disease, vision

and all the things you might want to know about brains.

It was really quite impressive.

One might've had the impression we knew a lot about brains.

But the last article in that issue was written by Francis Crick of DNA fame.

Today is, I think, the 50th anniversary of the discovery of DNA.

And he wrote a story basically saying, this is all well and good,

but you know, we don't know diddly squat about brains,

and no one has a clue how they work,

so don't believe what anyone tells you.

This is a quote from that article, he says:

"What is conspicuously lacking" -- he's a very proper British gentleman --

"What is conspicuously lacking is a broad framework of ideas

in which to interpret these different approaches."

I thought the word "framework" was great.

He didn't say we didn't have a theory.

He says we don't even know how to begin to think about it.

We don't even have a framework.

We are in the pre-paradigm days, if you want to use Thomas Kuhn.

So I fell in love with this.

I said, look: We have all this knowledge about brains -- how hard can it be?

It's something we can work on in my lifetime; I could make a difference.

So I tried to get out of the computer business, into the brain business.

First, I went to MIT, the AI lab was there.

I said, I want to build intelligent machines too,

but I want to study how brains work first.

And they said, "Oh, you don't need to do that.

You're just going to program computers, that's all.

I said, you really ought to study brains.

They said, "No, you're wrong."

I said, "No, you're wrong," and I didn't get in.

(Laughter)

I was a little disappointed -- pretty young --

but I went back again a few years later,

this time in California, and I went to Berkeley.

And I said, I'll go in from the biological side.

So I got in the PhD program in biophysics.

I was like, I'm studying brains now. Well, I want to study theory.

They said, "You can't study theory about brains.

You can't get funded for that.

And as a graduate student, you can't do that."

So I said, oh my gosh.

I was depressed; I said, but I can make a difference in this field.

I went back in the computer industry

and said, I'll have to work here for a while.

That's when I designed all those computer products.

(Laughter)

I said, I want to do this for four years, make some money,

I was having a family, and I would mature a bit,

and maybe the business of neuroscience would mature a bit.

Well, it took longer than four years. It's been about 16 years.

But I'm doing it now, and I'm going to tell you about it.

So why should we have a good brain theory?

Well, there's lots of reasons people do science.

The most basic one is, people like to know things.

We're curious, and we go out and get knowledge.

Why do we study ants? It's interesting.

Maybe we'll learn something useful, but it's interesting and fascinating.

But sometimes a science has other attributes

which makes it really interesting.

Sometimes a science will tell something about ourselves;

it'll tell us who we are.

Evolution did this and Copernicus did this,

where we have a new understanding of who we are.

And after all, we are our brains. My brain is talking to your brain.

Our bodies are hanging along for the ride,

but my brain is talking to your brain.

And if we want to understand who we are and how we feel and perceive,

we need to understand brains.

Another thing is sometimes science leads to big societal benefits, technologies,

or businesses or whatever.

This is one, too, because when we understand how brains work,

we'll be able to build intelligent machines.

That's a good thing on the whole,

with tremendous benefits to society,

just like a fundamental technology.

So why don't we have a good theory of brains?

People have been working on it for 100 years.

Let's first take a look at what normal science looks like.

This is normal science.

Normal science is a nice balance between theory and experimentalists.

The theorist guy says, "I think this is what's going on,"

the experimentalist says, "You're wrong."

It goes back and forth, this works in physics, this in geology.

But if this is normal science, what does neuroscience look like?

This is what neuroscience looks like.

We have this mountain of data,

which is anatomy, physiology and behavior.

You can't imagine how much detail we know about brains.

There were 28,000 people who went to the neuroscience conference this year,

and every one of them is doing research in brains.

A lot of data, but no theory.

There's a little wimpy box on top there.

And theory has not played a role in any sort of grand way

in the neurosciences.

And it's a real shame.

Now, why has this come about?

If you ask neuroscientists why is this the state of affairs,

first, they'll admit it.

But if you ask them, they say,

there's various reasons we don't have a good brain theory.

Some say we still don't have enough data,

we need more information, there's all these things we don't know.

Well, I just told you there's data coming out of your ears.

We have so much information, we don't even know how to organize it.

What good is more going to do?

Maybe we'll be lucky and discover some magic thing, but I don't think so.

This is a symptom of the fact that we just don't have a theory.

We don't need more data, we need a good theory.

Another one is sometimes people say,

"Brains are so complex, it'll take another 50 years."

I even think Chris said something like this yesterday, something like,

it's one of the most complicated things in the universe.

That's not true -- you're more complicated than your brain.

You've got a brain.

And although the brain looks very complicated,

things look complicated until you understand them.

That's always been the case.

So we can say, my neocortex, the part of the brain I'm interested in,

has 30 billion cells.

But, you know what? It's very, very regular.

In fact, it looks like it's the same thing repeated over and over again.

It's not as complex as it looks. That's not the issue.

Some people say, brains can't understand brains.

Very Zen-like. Woo.

(Laughter)

You know, it sounds good, but why? I mean, what's the point?

It's just a bunch of cells. You understand your liver.

It's got a lot of cells in it too, right?

So, you know, I don't think there's anything to that.

And finally, some people say,

"I don't feel like a bunch of cells -- I'm conscious.

I've got this experience, I'm in the world.

I can't be just a bunch of cells."

Well, people used to believe there was a life force to be living,

and we now know that's really not true at all.

And there's really no evidence,

other than that people just disbelieve that cells can do what they do.

So some people have fallen into the pit of metaphysical dualism,

some really smart people, too, but we can reject all that.

(Laughter)

No, there's something else,

something really fundamental, and it is:

another reason why we don't have a good brain theory

is because we have an intuitive, strongly held but incorrect assumption

that has prevented us from seeing the answer.

There's something we believe that just, it's obvious, but it's wrong.

Now, there's a history of this in science and before I tell you what it is,

I'll tell you about the history of it in science.

Look at other scientific revolutions --

the solar system, that's Copernicus,

Darwin's evolution, and tectonic plates, that's Wegener.

They all have a lot in common with brain science.

First, they had a lot of unexplained data. A lot of it.

But it got more manageable once they had a theory.

The best minds were stumped -- really smart people.

We're not smarter now than they were then;

it just turns out it's really hard to think of things,

but once you've thought of them, it's easy to understand.

My daughters understood these three theories,

in their basic framework, in kindergarten.

It's not that hard -- here's the apple, here's the orange,

the Earth goes around, that kind of stuff.

Another thing is the answer was there all along,

but we kind of ignored it because of this obvious thing.

It was an intuitive, strongly held belief that was wrong.

In the case of the solar system,

the idea that the Earth is spinning,

the surface is going a thousand miles an hour,

and it's going through the solar system at a million miles an hour --

this is lunacy; we all know the Earth isn't moving.

Do you feel like you're moving a thousand miles an hour?

If you said Earth was spinning around in space and was huge --

they would lock you up, that's what they did back then.

So it was intuitive and obvious. Now, what about evolution?

Evolution, same thing.

We taught our kids the Bible says God created all these species,

cats are cats; dogs are dogs; people are people; plants are plants;

they don't change.

Noah put them on the ark in that order, blah, blah.

The fact is, if you believe in evolution, we all have a common ancestor.

We all have a common ancestor with the plant in the lobby!

This is what evolution tells us. And it's true. It's kind of unbelievable.

And the same thing about tectonic plates.

All the mountains and the continents

are kind of floating around on top of the Earth.

It doesn't make any sense.

So what is the intuitive, but incorrect assumption,

that's kept us from understanding brains?

I'll tell you. It'll seem obvious that it's correct. That's the point.

Then I'll make an argument why you're incorrect on the other assumption.

The intuitive but obvious thing is:

somehow, intelligence is defined by behavior;

we're intelligent because of how we do things

and how we behave intelligently.

And I'm going to tell you that's wrong.

Intelligence is defined by prediction.

I'm going to work you through this in a few slides,

and give you an example of what this means.

Here's a system.

Engineers and scientists like to look at systems like this.

They say, we have a thing in a box. We have its inputs and outputs.

The AI people said, the thing in the box is a programmable computer,

because it's equivalent to a brain.

We'll feed it some inputs and get it to do something, have some behavior.

Alan Turing defined the Turing test, which essentially says,

we'll know if something's intelligent if it behaves identical to a human --

a behavioral metric of what intelligence is

that has stuck in our minds for a long time.

Reality, though -- I call it real intelligence.

Real intelligence is built on something else.

We experience the world through a sequence of patterns,

and we store them, and we recall them.

When we recall them, we match them up against reality,

and we're making predictions all the time.

It's an internal metric; there's an internal metric about us,

saying, do we understand the world, am I making predictions, and so on.

You're all being intelligent now, but you're not doing anything.

Maybe you're scratching yourself, but you're not doing anything.

But you're being intelligent; you're understanding what I'm saying.

Because you're intelligent and you speak English,

you know the word at the end of this

sentence.

The word came to you; you make these predictions all the time.

What I'm saying is,

the internal prediction is the output in the neocortex,

and somehow, prediction leads to intelligent behavior.

Here's how that happens:

Let's start with a non-intelligent brain.

I'll argue a non-intelligent brain, we'll call it an old brain.

And we'll say it's a non-mammal, like a reptile,

say, an alligator; we have an alligator.

And the alligator has some very sophisticated senses.

It's got good eyes and ears and touch senses and so on,

a mouth and a nose.

It has very complex behavior.

It can run and hide. It has fears and emotions. It can eat you.

It can attack. It can do all kinds of stuff.

But we don't consider the alligator very intelligent,

not in a human sort of way.

But it has all this complex behavior already.

Now in evolution, what happened?

First thing that happened in evolution with mammals

is we started to develop a thing called the neocortex.

I'm going to represent the neocortex by this box on top of the old brain.

Neocortex means "new layer." It's a new layer on top of your brain.

It's the wrinkly thing on the top of your head

that got wrinkly because it got shoved in there and doesn't fit.

(Laughter)

Literally, it's about the size of a table napkin

and doesn't fit, so it's wrinkly.

Now, look at how I've drawn this.

The old brain is still there.

You still have that alligator brain. You do. It's your emotional brain.

It's all those gut reactions you have.

On top of it, we have this memory system called the neocortex.

And the memory system is sitting over the sensory part of the brain.

So as the sensory input comes in and feeds from the old brain,

it also goes up into the neocortex.

And the neocortex is just memorizing.

It's sitting there saying, I'm going to memorize all the things going on:

where I've been, people I've seen, things I've heard, and so on.

And in the future, when it sees something similar to that again,

in a similar environment, or the exact same environment,

it'll start playing it back: "Oh, I've been here before,"

and when you were here before, this happened next.

It allows you to predict the future.

It literally feeds back the signals into your brain;

they'll let you see what's going to happen next,

will let you hear the word "sentence" before I said it.

And it's this feeding back into the old brain

that will allow you to make more intelligent decisions.

This is the most important slide of my talk, so I'll dwell on it a little.

And all the time you say, "Oh, I can predict things,"

so if you're a rat and you go through a maze, and you learn the maze,

next time you're in one, you have the same behavior.

But suddenly, you're smarter; you say, "I recognize this maze,

I know which way to go; I've been here before; I can envision the future."

That's what it's doing.

This is true for all mammals --

in humans, it got a lot worse.

Humans actually developed the front of the neocortex,

called the anterior part of the neocortex.

And nature did a little trick.

It copied the posterior, the back part, which is sensory,

and put it in the front.

Humans uniquely have the same mechanism on the front,

but we use it for motor control.

So we're now able to do very sophisticated motor planning, things like that.

I don't have time to explain, but to understand how a brain works,

you have to understand how the first part of the mammalian neocortex works,

how it is we store patterns and make predictions.

Let me give you a few examples of predictions.

I already said the word "sentence."

In music, if you've heard a song before,

when you hear it, the next note pops into your head already --

you anticipate it.

With an album, at the end of a song, the next song pops into your head.

It happens all the time, you make predictions.

I have this thing called the "altered door" thought experiment.

It says, you have a door at home;

when you're here, I'm changing it --

I've got a guy back at your house right now, moving the door around,

moving your doorknob over two inches.

When you go home tonight, you'll put your hand out, reach for the doorknob,

notice it's in the wrong spot

and go, "Whoa, something happened."

It may take a second, but something happened.

I can change your doorknob in other ways --

make it larger, smaller, change its brass to silver, make it a lever,

I can change the door; put colors on, put windows in.

I can change a thousand things about your door

and in the two seconds you take to open it,

you'll notice something has changed.

Now, the engineering approach, the AI approach to this,

is to build a door database with all the door attributes.

And as you go up to the door, we check them off one at time:

door, door, color ...

We don't do that. Your brain doesn't do that.

Your brain is making constant predictions all the time

about what will happen in your environment.

As I put my hand on this table, I expect to feel it stop.

When I walk, every step, if I missed it by an eighth of an inch,

I'll know something has changed.

You're constantly making predictions about your environment.

I'll talk about vision, briefly.

This is a picture of a woman.

When we look at people, our eyes saccade over two to three times a second.

We're not aware of it, but our eyes are always moving.

When we look at a face, we typically go from eye to eye to nose to mouth.

When your eye moves from eye to eye,

if there was something else there like a nose,

you'd see a nose where an eye is supposed to be and go, "Oh, shit!"

(Laughter)

"There's something wrong about this person."

That's because you're making a prediction.

It's not like you just look over and say, "What am I seeing? A nose? OK."

No, you have an expectation of what you're going to see.

Every single moment.

And finally, let's think about how we test intelligence.

We test it by prediction: What is the next word in this ...?

This is to this as this is to this. What is the next number in this sentence?

Here's three visions of an object. What's the fourth one?

That's how we test it. It's all about prediction.

So what is the recipe for brain theory?

First of all, we have to have the right framework.

And the framework is a memory framework,

not a computational or behavior framework,

it's a memory framework.

How do you store and recall these sequences of patterns?

It's spatiotemporal patterns.

Then, if in that framework, you take a bunch of theoreticians --

biologists generally are not good theoreticians.

Not always, but generally, there's not a good history of theory in biology.

I've found the best people to work with are physicists,

engineers and mathematicians,

who tend to think algorithmically.

Then they have to learn the anatomy and the physiology.

You have to make these theories very realistic in anatomical terms.

Anyone who tells you their theory about how the brain works

and doesn't tell you exactly how it's working

and how the wiring works --

it's not a theory.

And that's what we do at the Redwood Neuroscience Institute.

I'd love to tell you we're making fantastic progress in this thing,

and I expect to be back on this stage sometime in the not too distant future,

to tell you about it.

I'm really excited; this is not going to take 50 years.

What will brain theory look like?

First of all, it's going to be about memory.

Not like computer memory -- not at all like computer memory.

It's very different.

It's a memory of very high-dimensional patterns,

like the things that come from your eyes.

It's also memory of sequences:

you cannot learn or recall anything outside of a sequence.

A song must be heard in sequence over time,

and you must play it back in sequence over time.

And these sequences are auto-associatively recalled,

so if I see something, I hear something, it reminds me of it,

and it plays back automatically.

It's an automatic playback.

And prediction of future inputs is the desired output.

And as I said, the theory must be biologically accurate,

it must be testable and you must be able to build it.

If you don't build it, you don't understand it.

One more slide.

What is this going to result in?

Are we going to really build intelligent machines?

Absolutely. And it's going to be different than people think.

No doubt that it's going to happen, in my mind.

First of all, we're going to build this stuff out of silicon.

The same techniques we use to build silicon computer memories,

we can use here.

But they're very different types of memories.

And we'll attach these memories to sensors,

and the sensors will experience real-live, real-world data,

and learn about their environment.

Now, it's very unlikely the first things you'll see are like robots.

Not that robots aren't useful; people can build robots.

But the robotics part is the hardest part. That's old brain. That's really hard.

The new brain is easier than the old brain.

So first we'll do things that don't require a lot of robotics.

So you're not going to see C-3PO.

You're going to see things more like intelligent cars

that really understand what traffic is, what driving is

and have learned that cars with the blinkers on for half a minute

probably aren't going to turn.

(Laughter)

We can also do intelligent security systems.

Anytime we're basically using our brain but not doing a lot of mechanics --

those are the things that will happen first.

But ultimately, the world's the limit.

I don't know how this will turn out.

I know a lot of people who invented the microprocessor.

And if you talk to them,

they knew what they were doing was really significant,

but they didn't really know what was going to happen.

They couldn't anticipate cell phones and the Internet

and all this kind of stuff.

They just knew like, "We're going to build calculators

and traffic-light controllers.

But it's going to be big!"

In the same way, brain science and these memories

are going to be a very fundamental technology,

and it will lead to unbelievable changes in the next 100 years.

And I'm most excited about how we're going to use them in science.

So I think that's all my time -- I'm over,

and I'm going to end my talk right there.