Pointing to the future of UI | John Underkoffler

We're 25, 26 years after

the advent of the Macintosh,

which was an astoundingly seminal event

in the history

of human-machine interface

and in computation in general.

It fundamentally changed the way

that people thought about computation,

thought about computers,

how they used them and who and how many people were able to use them.

It was such a radical change, in fact,

that the early Macintosh development team

in '82, '83, '84

had to write an entirely new operating system from the ground up.

Now, this is an interesting little message,

and it's a lesson that has since, I think,

been forgotten or lost or something,

and that is, namely, that the OS is the interface.

The interface is the OS.

It's like the land and the king (i.e. Arthur) they're inseparable, they are one.

And to write a new operating system was not a capricious matter.

It wasn't just a matter of tuning up some graphics routines.

There were no graphics routines. There were no mouse drivers.

So it was a necessity.

But in the quarter-century since then,

we've seen all of the fundamental

supporting technologies go berserk.

So memory capacity and disk capacity

have been multiplied by something between 10,000 and a million.

Same thing for processor speeds.

Networks, we didn't have networks at all

at the time of the Macintosh's introduction,

and that has become the single most salient aspect

of how we live with computers.

And, of course, graphics: Today

84 dollars and 97 cents at Best Buy

buys you more graphics power

than you could have gotten for a million bucks from SGI only a decade ago.

So we've got that incredible ramp-up.

Then, on the side, we've got the Web

and, increasingly, the cloud,

which is fantastic,

but also -- in the regard in which an interface is fundamental --

kind of a distraction.

So we've forgotten to invent new interfaces.

Certainly we've seen in recent years a lot of change in that regard,

and people are starting to wake up about that.

So what happens next? Where do we go from there?

The problem, as we see it,

has to do with a single, simple word: "space,"

or a single, simple phrase:

"real world geometry."

Computers and the programming languages

that we talk to them in, that we teach them in,

are hideously insensate when it comes to space.

They don't understand real world space.

It's a funny thing because the rest of us occupy it quite frequently and quite well.

They also don't understand time, but that's a matter for a separate talk.

So what happens if you start to

explain space to them?

One thing you might get is something like the Luminous Room.

The Luminous Room is a system

in which it's considered that

input and output spaces are co-located.

That's a strangely simple,

and yet unexplored idea, right?

When you use a mouse, your hand is down here on the mouse pad.

It's not even on the same plane as what you're talking about:

The pixels are up on the display.

So here was a room in which all the walls, floors, ceilings,

pets, potted plants, whatever was in there,

were capable, not only of display but of sensing as well.

And that means input and output are in the same space

enabling stuff like this.

That's a digital storage in a physical container.

The contract is the same

as with real word objects in real world containers.

Has to come back out, whatever you put in.

This little design experiment

that was a small office here knew a few other tricks as well.

If you presented it with a chess board,

it tried to figure out what you might mean by that.

And if there was nothing for them to do,

the chess pieces eventually got bored

and hopped away.

The academics who were overseeing this work

thought that that was too frivolous,

so we built deadly serious applications

like this optics prototyping workbench

in which a toothpaste cap on a cardboard box

becomes a laser.

The beam splitters and lenses are represented by physical objects,

and the system projects down the laser beam path.

So you've got an interface that has no interface.

You operate the world as you operate the real world,

which is to say, with your hands.

Similarly, a digital wind tunnel with digital wind

flowing from right to left --

not that remarkable in a sense; we didn't invent the mathematics.

But if you displayed that on a CRT or flat panel display,

it would be meaningless to hold up an arbitrary object,

a real world object in that.

Here, the real world merges with the simulation.

And finally, to pull out all the stops,

this is a system called Urp, for urban planners,

in which we give architects and urban planners back

the models that we confiscated

when we insisted that they use CAD systems.

And we make the machine meet them half way.

It projects down digital shadows, as you see here.

And if you introduce tools like this inverse clock,

then you can control the sun's position in the sky.

That's 8 a.m. shadows.

They get a little shorter at 9 a.m.

There you are, swinging the sun around.

Short shadows at noon and so forth.

And we built up a series of tools like this.

There are inter-shadowing studies

that children can operate,

even though they don't know anything about urban planning:

To move a building, you simply reach out your hand and you move the building.

A material wand makes the building

into a sort of Frank Gehry thing that reflects light in all directions.

Are you blinding passers by and motorists on the freeways?

A zoning tool connects distant structures, a building and a roadway.

Are you going to get sued by the zoning commission? And so forth.

Now, if these ideas seem familiar

or perhaps even a little dated,

that's great; they should seem familiar.

This work is 15 years old.

This stuff was undertaken at MIT and the Media Lab

under the incredible direction of Professor Hiroshi Ishii,

director of the Tangible Media Group.

But it was that work that was seen

by Alex McDowell,

one of the world's legendary production designers.

But Alex was preparing a little, sort of obscure, indie, arthouse film

called "Minority Report" for Steven Spielberg,

and invited us to come out from MIT

and design the interfaces

that would appear in that film.

And the great thing about it was

that Alex was so dedicated to the idea of verisimilitude,

the idea that the putative 2054

that we were painting in the film be believable,

that he allowed us to take on that design work

as if it were an R&D effort.

And the result is sort of

gratifyingly perpetual.

People still reference those sequences in "Minority Report"

when they talk about new UI design.

So this led full circle, in a strange way,

to build these ideas into what we believe

is the necessary future of human machine interface:

the Spatial Operating Environment, we call it.

So here we have a bunch of stuff, some images.

And, using a hand,

we can actually exercise six degrees of freedom,

six degrees of navigational control.

And it's fun to fly through Mr. Beckett's eye.

And you can come back out

through the scary orangutan.

And that's all well and good.

Let's do something a little more difficult.

Here, we have a whole bunch of disparate images.

We can fly around them.

So navigation is a fundamental issue.

You have to be able to navigate in 3D.

Much of what we want computers to help us with in the first place

is inherently spatial.

And the part that isn't spatial can often be spatialized

to allow our wetware to make greater sense of it.

Now we can distribute this stuff in many different ways.

So we can throw it out like that. Let's reset it.

We can organize it this way.

And, of course, it's not just about navigation,

but about manipulation as well.

So if we don't like stuff,

or we're intensely curious about

Ernst Haeckel's scientific falsifications,

we can pull them out like that.

And then if it's time for analysis, we can pull back a little bit

and ask for a different distribution.

Let's just come down a bit

and fly around.

So that's a different way to look at stuff.

If you're of a more analytical nature

then you might want, actually, to look at this

as a color histogram.

So now we've got the stuff color-sorted,

angle maps onto color.

And now, if we want to select stuff,

3D, space,

the idea that we're tracking hands in real space

becomes really important because we can reach in,

not in 2D, not in fake 2D, but in actual 3D.

Here are some selection planes.

And we'll perform this Boolean operation

because we really love yellow and tapirs on green grass.

So, from there to the world of real work.

Here's a logistics system,

a small piece of one that we're currently building.

There're a lot of elements.

And one thing that's very important is to combine traditional tabular data

with three-dimensional and geospatial information.

So here's a familiar place.

And we'll bring this back here for a second.

Maybe select a little bit of that.

And bring out this graph.

And we should, now,

be able to fly in here

and have a closer look.

These are logistics elements

that are scattered across the United States.

One thing that three-dimensional interactions

and the general idea of imbuing

computation with space affords you

is a final destruction of that unfortunate

one-to-one pairing between human beings and computers.

That's the old way, that's the old mantra:

one machine, one human, one mouse, one screen.

Well, that doesn't really cut it anymore.

In the real world, we have people who collaborate;

we have people who have to work together,

and we have many different displays.

And we might want to look at these various images.

We might want to ask for some help.

The author of this new pointing device

is sitting over there,

so I can pull this from there to there.

These are unrelated machines, right?

So the computation is space soluble and network soluble.

So I'm going to leave that over there

because I have a question for Paul.

Paul is the designer of this wand, and maybe its easiest

for him to come over here and tell me in person what's going on.

So let me get some of these out of the way.

Let's pull this apart:

I'll go ahead and explode it.

Kevin, can you help?

Let me see if I can help us find the circuit board.

Mind you, it's a sort of gratuitous field-stripping exercise,

but we do it in the lab all the time.

All right.

So collaborative work, whether it's immediately co-located

or distant and distinct, is always important.

And again, that stuff

needs to be undertaken in the context of space.

And finally, I'd like to leave you with a glimpse

that takes us back to the world of imagery.

This is a system called TAMPER,

which is a slightly whimsical look

at what the future of editing

and media manipulation systems might be.

We at Oblong believe that media should be

accessible in much more fine-grained form.

So we have a large number of movies

stuck inside here.

And let's just pick out a few elements.

We can zip through them

as a possibility.

We can grab elements off the front,

where upon they reanimate, come to life,

and drag them down onto the table here.

We'll go over to Jacques Tati here

and grab our blue friend

and put him down on the table as well.

We may need more than one.

And we probably need,

well, we probably need a cowboy

to be quite honest.

(Laughter)

Yeah, let's

take that one.

(Laughter)

You see, cowboys and French farce people

don't go well together, and the system knows that.

Let me leave with one final thought,

and that is that

one of the greatest English language writers

of the last three decades

suggested that great art is always a gift.

And he wasn't talking about whether the novel costs 24.95 [dollars],

or whether you have to spring 70 million bucks

to buy the stolen Vermeer;

he was talking about the circumstances of its creation

and of its existence.

And I think that it's time that we asked

for the same from technology.

Technology is capable of

expressing and being imbued with

a certain generosity,

and we need to demand that, in fact.

For some of this kind of technology,

ground center is

a combination of design, which is crucially important.

We can't have advances in technology any longer

unless design is integrated from the very start.

And, as well, as of efficacy, agency.

We're, as human beings, the creatures that create,

and we should make sure that our machines aid us in that task

and are built in that same image.

So I will leave you with that. Thank you.

(Applause)

Chris Anderson: So to ask the obvious question --

actually this is from Bill Gates --

when? (John Underkoffler: When?)

CA: When real? When for us, not just in a lab and on a stage?

Can it be for every man, or is this just for corporations and movie producers?

JU: No, it has to be for every human being.

That's our goal entirely.

We won't have succeeded

unless we take that next big step.

I mean it's been 25 years.

Can there really be only one interface? There can't.

CA: But does that mean that, at your desk or in your home,

you need projectors, cameras?

You know, how can it work?

JU: No, this stuff will be built into the bezel of every display.

It'll be built into architecture.

The gloves go away in a matter of months or years.

So this is the inevitability about it.

CA: So, in your mind, five years time,

someone can buy this as part of

a standard computer interface?

JU: I think in five years time when you buy a computer,

you'll get this.

CA: Well that's cool.

(Applause)

The world has a habit of surprising us as to how these things are actually used.

What do you think, what in your mind is the first killer app for this?

JU: That's a good question, and we ask ourselves that every day.

At the moment, our early-adopter customers --

and these systems are deployed out in the real world --

do all the big data intensive, data heavy problems with it.

So, whether it's logistics and supply chain management

or natural gas and resource extraction,

financial services, pharmaceuticals, bioinformatics,

those are the topics right now, but that's not a killer app.

And I understand what you're asking.

CA: C'mon, c'mon. Martial arts, games. C'mon.

(Laughter)

John, thank you for making science-fiction real.

JU: It's been a great pleasure.

Thank you to you all.

(Applause)