With Spatial Intelligence, AI Will Understand the Real World | Fei-Fei Li | TED

Let me show you something.

To be precise,

I'm going to show you nothing.

This was the world 540 million years ago.

Pure, endless darkness.

It wasn't dark due to a lack of light.

It was dark because of a lack of sight.

Although sunshine did filter 1,000 meters

beneath the surface of ocean,

a light permeated from hydrothermal vents to seafloor,

brimming with life,

there was not a single eye to be found in these ancient waters.

No retinas, no corneas, no lenses.

So all this light, all this life went unseen.

There was a time that the very idea of seeing didn't exist.

It [had] simply never been done before.

Until it was.

So for reasons we're only beginning to understand,

trilobites, the first organisms that could sense light, emerged.

They're the first inhabitants of this reality that we take for granted.

First to discover that there is something other than oneself.

A world of many selves.

The ability to see is thought to have ushered in Cambrian explosion,

a period in which a huge variety of animal species

entered fossil records.

What began as a passive experience,

the simple act of letting light in,

soon became far more active.

The nervous system began to evolve.

Sight turning to insight.

Seeing became understanding.

Understanding led to actions.

And all these gave rise to intelligence.

Today, we're no longer satisfied with just nature's gift of visual intelligence.

Curiosity urges us to create machines to see just as intelligently as we can,

if not better.

Nine years ago, on this stage,

I delivered an early progress report on computer vision,

a subfield of artificial intelligence.

Three powerful forces converged for the first time.

Aa family of algorithms called neural networks.

Fast, specialized hardware called graphic processing units,

or GPUs.

And big data.

Like the 15 million images that my lab spent years curating called ImageNet.

Together, they ushered in the age of modern AI.

We've come a long way.

Back then, just putting labels on images was a big breakthrough.

But the speed and accuracy of these algorithms just improved rapidly.

The annual ImageNet challenge, led by my lab,

gauged the performance of this progress.

And on this plot, you're seeing the annual improvement

and milestone models.

We went a step further

and created algorithms that can segment objects

or predict the dynamic relationships among them

in these works done by my students and collaborators.

And there's more.

Recall last time I showed you the first computer-vision algorithm

that can describe a photo in human natural language.

That was work done with my brilliant former student, Andrej Karpathy.

At that time, I pushed my luck and said,

"Andrej, can we make computers to do the reverse?"

And Andrej said, "Ha ha, that's impossible."

Well, as you can see from this post,

recently the impossible has become possible.

That's thanks to a family of diffusion models

that powers today's generative AI algorithm,

which can take human-prompted sentences

and turn them into photos and videos

of something that's entirely new.

Many of you have seen the recent impressive results of Sora by OpenAI.

But even without the enormous number of GPUs,

my student and our collaborators

have developed a generative video model called Walt

months before Sora.

And you're seeing some of these results.

There is room for improvement.

I mean, look at that cat's eye

and the way it goes under the wave without ever getting wet.

What a cat-astrophe.

(Laughter)

And if past is prologue,

we will learn from these mistakes and create a future we imagine.

And in this future,

we want AI to do everything it can for us,

or to help us.

For years I have been saying

that taking a picture is not the same as seeing and understanding.

Today, I would like to add to that.

Simply seeing is not enough.

Seeing is for doing and learning.

When we act upon this world in 3D space and time,

we learn, and we learn to see and do better.

Nature has created this virtuous cycle of seeing and doing

powered by “spatial intelligence.”

To illustrate to you what your spatial intelligence is doing constantly,

look at this picture.

Raise your hand if you feel like you want to do something.

(Laughter)

In the last split of a second,

your brain looked at the geometry of this glass,

its place in 3D space,

its relationship with the table, the cat

and everything else.

And you can predict what's going to happen next.

The urge to act is innate to all beings with spatial intelligence,

which links perception with action.

And if we want to advance AI beyond its current capabilities,

we want more than AI that can see and talk.

We want AI that can do.

Indeed, we're making exciting progress.

The recent milestones in spatial intelligence

is teaching computers to see, learn, do

and learn to see and do better.

This is not easy.

It took nature millions of years to evolve spatial intelligence,

which depends on the eye taking light,

project 2D images on the retina

and the brain to translate these data into 3D information.

Only recently, a group of researchers from Google

are able to develop an algorithm to take a bunch of photos

and translate that into 3D space,

like the examples we're showing here.

My student and our collaborators have taken a step further

and created an algorithm that takes one input image

and turn that into 3D shape.

Here are more examples.

Recall, we talked about computer programs that can take a human sentence

and turn it into videos.

A group of researchers in University of Michigan

have figured out a way to translate that line of sentence

into 3D room layout, like shown here.

And my colleagues at Stanford and their students

have developed an algorithm that takes one image

and generates infinitely plausible spaces

for viewers to explore.

These are prototypes of the first budding signs of a future possibility.

One in which the human race can take our entire world

and translate it into digital forms

and model the richness and nuances.

What nature did to us implicitly in our individual minds,

spatial intelligence technology can hope to do

for our collective consciousness.

As the progress of spatial intelligence accelerates,

a new era in this virtuous cycle is taking place in front of our eyes.

This back and forth is catalyzing robotic learning,

a key component for any embodied intelligence system

that needs to understand and interact with the 3D world.

A decade ago,

ImageNet from my lab

enabled a database of millions of high-quality photos

to help train computers to see.

Today, we're doing the same with behaviors and actions

to train computers and robots how to act in the 3D world.

But instead of collecting static images,

we develop simulation environments powered by 3D spatial models

so that the computers can have infinite varieties of possibilities

to learn to act.

And you're just seeing a small number of examples

to teach our robots

in a project led by my lab called Behavior.

We’re also making exciting progress in robotic language intelligence.

Using large language model-based input,

my students and our collaborators are among the first teams

that can show a robotic arm performing a variety of tasks

based on verbal instructions,

like opening this drawer or unplugging a charged phone.

Or making sandwiches, using bread, lettuce, tomatoes

and even putting a napkin for the user.

Typically I would like a little more for my sandwich,

but this is a good start.

(Laughter)

In that primordial ocean, in our ancient times,

the ability to see and perceive one's environment

kicked off the Cambrian explosion of interactions with other life forms.

Today, that light is reaching the digital minds.

Spatial intelligence is allowing machines

to interact not only with one another,

but with humans, and with 3D worlds,

real or virtual.

And as that future is taking shape,

it will have a profound impact to many lives.

Let's take health care as an example.

For the past decade,

my lab has been taking some of the first steps

in applying AI to tackle challenges that impact patient outcome

and medical staff burnout.

Together with our collaborators from Stanford School of Medicine

and partnering hospitals,

we're piloting smart sensors

that can detect clinicians going into patient rooms

without properly washing their hands.

Or keep track of surgical instruments.

Or alert care teams when a patient is at physical risk,

such as falling.

We consider these techniques a form of ambient intelligence,

like extra pairs of eyes that do make a difference.

But I would like more interactive help for our patients, clinicians

and caretakers, who desperately also need an extra pair of hands.

Imagine an autonomous robot transporting medical supplies

while caretakers focus on our patients

or augmented reality, guiding surgeons to do safer, faster

and less invasive operations.

Or imagine patients with severe paralysis controlling robots with their thoughts.

That's right, brainwaves, to perform everyday tasks

that you and I take for granted.

You're seeing a glimpse of that future in this pilot study from my lab recently.

In this video, the robotic arm is cooking a Japanese sukiyaki meal

controlled only by the brain electrical signal,

non-invasively collected through an EEG cap.

(Applause)

Thank you.

The emergence of vision half a billion years ago

turned a world of darkness upside down.

It set off the most profound evolutionary process:

the development of intelligence in the animal world.

AI's breathtaking progress in the last decade is just as astounding.

But I believe the full potential of this digital Cambrian explosion

won't be fully realized until we power our computers and robots

with spatial intelligence,

just like what nature did to all of us.

It’s an exciting time to teach our digital companion

to learn to reason

and to interact with this beautiful 3D space we call home,

and also create many more new worlds that we can all explore.

To realize this future won't be easy.

It requires all of us to take thoughtful steps

and develop technologies that always put humans in the center.

But if we do this right,

the computers and robots powered by spatial intelligence

will not only be useful tools

but also trusted partners

to enhance and augment our productivity and humanity

while respecting our individual dignity

and lifting our collective prosperity.

What excites me the most in the future

is a future in which that AI grows more perceptive,

insightful and spatially aware,

and they join us on our quest

to always pursue a better way to make a better world.

Thank you.

(Applause)