James Patten: The best computer interface? Maybe ... your hands

A computer is an incredibly powerful means

of creative expression,

but for the most part,

that expression is confined to the screens

of our laptops and mobile phones.

And I'd like to tell you a story about

bringing this power of the computer

to move things around and interact with us

off of the screen and into the physical world

in which we live.

A few years ago, I got a call from

a luxury fashion store called Barneys New York,

and the next thing I knew,

I was designing storefront kinetic sculptures

for their window displays.

This one's called "The Chase."

There are two pairs of shoes,

a man's pair and a woman's pair,

and they play out this slow, tense chase

around the window

in which the man scoots up behind the woman

and gets in her personal space,

and then she moves away.

Each of the shoes has magnets in it,

and there are magnets underneath the table

that move the shoes around.

My friend Andy Cavatorta was building

a robotic harp for Bjork's Biophilia tour

and I wound up building the electronics

and motion control software

to make the harps move and play music.

The harp has four separate pendulums,

and each pendulum has 11 strings,

so the harp swings on its axis and also rotates

in order to play different musical notes,

and the harps are all networked together

so that they can play the right notes

at the right time in the music.

I built an interactive chemistry exhibit

at the Museum of Science and Industry in Chicago,

and this exhibit lets people use physical objects

to grab chemical elements off of the periodic table

and bring them together to cause

chemical reactions to happen.

And the museum noticed that people

were spending a lot of time with this exhibit,

and a researcher from a science education center

in Australia decided to study this exhibit

and try to figure out what was going on.

And she found that the physical objects

that people were using were helping people

understand how to use the exhibit,

and were helping people learn in a social way.

And when you think about it, this makes a lot of sense,

that using specialized physical objects

would help people use an interface more easily.

I mean, our hands and our minds are optimized

to think about and interact with tangible objects.

Think about which you find easier to use,

a physical keyboard or an onscreen keyboard

like on a phone?

But the thing that struck me

about all of these different projects

is that they really had to be built from scratch,

down to the level of the electronics

and the printed circuit boards and

all the mechanisms all the way up to the software.

I wanted to create something

where we could move objects under computer control

and create interactions around that idea

without having to go through this process

of building something from scratch

every single time.

So my first attempt at this

was at the MIT Media Lab

with Professor Hiroshi Ishii,

and we built this array of

512 different electromagnets,

and together they were able to move objects around

on top of their surface.

But the problem with this

was that these magnets

cost over 10,000 dollars.

Although each one was pretty small,

altogether they weighed so much

that the table that they were on

started to sag.

So I wanted to build something

where you could have this kind of interaction

on any tabletop surface.

So to explore this idea,

I built an army of small robots,

and each of these robots has what are called omni wheels.

They're these special wheels

that can move equally easily in all directions,

and when you couple these robots

with a video projector,

you have these physical tools

for interacting with digital information.

So here's an example of what I mean.

This is a video editing application

where all of the controls

for manipulating the video are physical.

So if we want to tweak the color,

we just enter the color mode,

and then we get three different dials

for tweaking the color,

or if we want to adjust the audio,

then we get two different dials for that, these physical objects.

So here the left and right channel stay in sync,

but if we want to, we can override that

by grabbing both of them at the same time.

So the idea is that we get the speed

and efficiency benefits of using these physical dials

together with the flexibility and versatility

of a system that's designed in software.

And this is a mapping application

for disaster response.

So you have these physical objects

that represent police, fire and rescue,

and a dispatcher can grab them

and place them on the map

to tell those units where to go,

and then the position of the units on the map

gets synced up with the position

of those units in the real world.

This is a video chat application.

It's amazing how much emotion you can convey

with just a few simple movements

of a physical object.

With this interface, we open up a huge array of possibilities

in between traditional board games

and arcade games,

where the physical possibilities of interaction

make so many different styles of play possible.

But one of the areas that I'm most excited

about using this platform for

is applying it to problems that are difficult

for computers or people to solve alone.

One example of those is protein folding.

So here we have an interface

where we have physical handles onto a protein,

and we can grab those handles

and try to move the protein and try to fold it in different ways.

And if we move it in a way that doesn't really make sense

with the underlying molecular simulation,

we get this physical feedback where we can

actually feel these physical handles

pulling back against us.

So feeling what's going on

inside a molecular simulation

is a whole different level of interaction.

So we're just beginning to explore

what's possible when we use software

to control the movement

of objects in our environment.

Maybe this is the computer of the future.

There's no touchscreen.

There's no technology visible at all.

But when we want to have a video chat

or play a game

or lay out the slides to our next TED Talk,

the objects on the table come alive.

Thank you.

(Applause)