Damon Horowitz calls for a "moral operating system"

Power.

That is the word that comes to mind.

We're the new technologists.

We have a lot of data, so we have a lot of power.

How much power do we have?

Scene from a movie: "Apocalypse Now" -- great movie.

We've got to get our hero, Captain Willard, to the mouth of the Nung River

so he can go pursue Colonel Kurtz.

The way we're going to do this is fly him in and drop him off.

So the scene:

the sky is filled with this fleet of helicopters carrying him in.

And there's this loud, thrilling music in the background,

this wild music.

♫ Dum da ta da dum ♫

♫ Dum da ta da dum ♫

♫ Da ta da da ♫

That's a lot of power.

That's the kind of power I feel in this room.

That's the kind of power we have

because of all of the data that we have.

Let's take an example.

What can we do

with just one person's data?

What can we do

with that guy's data?

I can look at your financial records.

I can tell if you pay your bills on time.

I know if you're good to give a loan to.

I can look at your medical records; I can see if your pump is still pumping --

see if you're good to offer insurance to.

I can look at your clicking patterns.

When you come to my website, I actually know what you're going to do already

because I've seen you visit millions of websites before.

And I'm sorry to tell you,

you're like a poker player, you have a tell.

I can tell with data analysis what you're going to do

before you even do it.

I know what you like. I know who you are,

and that's even before I look at your mail

or your phone.

Those are the kinds of things we can do

with the data that we have.

But I'm not actually here to talk about what we can do.

I'm here to talk about what we should do.

What's the right thing to do?

Now I see some puzzled looks

like, "Why are you asking us what's the right thing to do?

We're just building this stuff. Somebody else is using it."

Fair enough.

But it brings me back.

I think about World War II --

some of our great technologists then,

some of our great physicists,

studying nuclear fission and fusion --

just nuclear stuff.

We gather together these physicists in Los Alamos

to see what they'll build.

We want the people building the technology

thinking about what we should be doing with the technology.

So what should we be doing with that guy's data?

Should we be collecting it, gathering it,

so we can make his online experience better?

So we can make money?

So we can protect ourselves

if he was up to no good?

Or should we respect his privacy,

protect his dignity and leave him alone?

Which one is it?

How should we figure it out?

I know: crowdsource. Let's crowdsource this.

So to get people warmed up,

let's start with an easy question --

something I'm sure everybody here has an opinion about:

iPhone versus Android.

Let's do a show of hands -- iPhone.

Uh huh.

Android.

You'd think with a bunch of smart people

we wouldn't be such suckers just for the pretty phones.

(Laughter)

Next question,

a little bit harder.

Should we be collecting all of that guy's data

to make his experiences better

and to protect ourselves in case he's up to no good?

Or should we leave him alone?

Collect his data.

Leave him alone.

You're safe. It's fine.

(Laughter)

Okay, last question --

harder question --

when trying to evaluate

what we should do in this case,

should we use a Kantian deontological moral framework,

or should we use a Millian consequentialist one?

Kant.

Mill.

Not as many votes.

(Laughter)

Yeah, that's a terrifying result.

Terrifying, because we have stronger opinions

about our hand-held devices

than about the moral framework

we should use to guide our decisions.

How do we know what to do with all the power we have

if we don't have a moral framework?

We know more about mobile operating systems,

but what we really need is a moral operating system.

What's a moral operating system?

We all know right and wrong, right?

You feel good when you do something right,

you feel bad when you do something wrong.

Our parents teach us that: praise with the good, scold with the bad.

But how do we figure out what's right and wrong?

And from day to day, we have the techniques that we use.

Maybe we just follow our gut.

Maybe we take a vote -- we crowdsource.

Or maybe we punt --

ask the legal department, see what they say.

In other words, it's kind of random,

kind of ad hoc,

how we figure out what we should do.

And maybe, if we want to be on surer footing,

what we really want is a moral framework that will help guide us there,

that will tell us what kinds of things are right and wrong in the first place,

and how would we know in a given situation what to do.

So let's get a moral framework.

We're numbers people, living by numbers.

How can we use numbers

as the basis for a moral framework?

I know a guy who did exactly that.

A brilliant guy --

he's been dead 2,500 years.

Plato, that's right.

Remember him -- old philosopher?

You were sleeping during that class.

And Plato, he had a lot of the same concerns that we did.

He was worried about right and wrong.

He wanted to know what is just.

But he was worried that all we seem to be doing

is trading opinions about this.

He says something's just. She says something else is just.

It's kind of convincing when he talks and when she talks too.

I'm just going back and forth; I'm not getting anywhere.

I don't want opinions; I want knowledge.

I want to know the truth about justice --

like we have truths in math.

In math, we know the objective facts.

Take a number, any number -- two.

Favorite number. I love that number.

There are truths about two.

If you've got two of something,

you add two more, you get four.

That's true no matter what thing you're talking about.

It's an objective truth about the form of two,

the abstract form.

When you have two of anything -- two eyes, two ears, two noses,

just two protrusions --

those all partake of the form of two.

They all participate in the truths that two has.

They all have two-ness in them.

And therefore, it's not a matter of opinion.

What if, Plato thought,

ethics was like math?

What if there were a pure form of justice?

What if there are truths about justice,

and you could just look around in this world

and see which things participated,

partook of that form of justice?

Then you would know what was really just and what wasn't.

It wouldn't be a matter

of just opinion or just appearances.

That's a stunning vision.

I mean, think about that. How grand. How ambitious.

That's as ambitious as we are.

He wants to solve ethics.

He wants objective truths.

If you think that way,

you have a Platonist moral framework.

If you don't think that way,

well, you have a lot of company in the history of Western philosophy,

because the tidy idea, you know, people criticized it.

Aristotle, in particular, he was not amused.

He thought it was impractical.

Aristotle said, "We should seek only so much precision in each subject

as that subject allows."

Aristotle thought ethics wasn't a lot like math.

He thought ethics was a matter of making decisions in the here-and-now

using our best judgment

to find the right path.

If you think that, Plato's not your guy.

But don't give up.

Maybe there's another way

that we can use numbers as the basis of our moral framework.

How about this:

What if in any situation you could just calculate,

look at the choices,

measure out which one's better and know what to do?

That sound familiar?

That's a utilitarian moral framework.

John Stuart Mill was a great advocate of this --

nice guy besides --

and only been dead 200 years.

So basis of utilitarianism --

I'm sure you're familiar at least.

The three people who voted for Mill before are familiar with this.

But here's the way it works.

What if morals, what if what makes something moral

is just a matter of if it maximizes pleasure

and minimizes pain?

It does something intrinsic to the act.

It's not like its relation to some abstract form.

It's just a matter of the consequences.

You just look at the consequences

and see if, overall, it's for the good or for the worse.

That would be simple. Then we know what to do.

Let's take an example.

Suppose I go up

and I say, "I'm going to take your phone."

Not just because it rang earlier,

but I'm going to take it because I made a little calculation.

I thought, that guy looks suspicious.

And what if he's been sending little messages to Bin Laden's hideout --

or whoever took over after Bin Laden --

and he's actually like a terrorist, a sleeper cell.

I'm going to find that out, and when I find that out,

I'm going to prevent a huge amount of damage that he could cause.

That has a very high utility to prevent that damage.

And compared to the little pain that it's going to cause --

because it's going to be embarrassing when I'm looking on his phone

and seeing that he has a Farmville problem and that whole bit --

that's overwhelmed

by the value of looking at the phone.

If you feel that way,

that's a utilitarian choice.

But maybe you don't feel that way either.

Maybe you think, it's his phone.

It's wrong to take his phone

because he's a person

and he has rights and he has dignity,

and we can't just interfere with that.

He has autonomy.

It doesn't matter what the calculations are.

There are things that are intrinsically wrong --

like lying is wrong,

like torturing innocent children is wrong.

Kant was very good on this point,

and he said it a little better than I'll say it.

He said we should use our reason

to figure out the rules by which we should guide our conduct,

and then it is our duty to follow those rules.

It's not a matter of calculation.

So let's stop.

We're right in the thick of it, this philosophical thicket.

And this goes on for thousands of years,

because these are hard questions,

and I've only got 15 minutes.

So let's cut to the chase.

How should we be making our decisions?

Is it Plato, is it Aristotle, is it Kant, is it Mill?

What should we be doing? What's the answer?

What's the formula that we can use in any situation

to determine what we should do,

whether we should use that guy's data or not?

What's the formula?

There's not a formula.

There's not a simple answer.

Ethics is hard.

Ethics requires thinking.

And that's uncomfortable.

I know; I spent a lot of my career

in artificial intelligence,

trying to build machines that could do some of this thinking for us,

that could give us answers.

But they can't.

You can't just take human thinking

and put it into a machine.

We're the ones who have to do it.

Happily, we're not machines, and we can do it.

Not only can we think,

we must.

Hannah Arendt said,

"The sad truth

is that most evil done in this world

is not done by people

who choose to be evil.

It arises from not thinking."

That's what she called the "banality of evil."

And the response to that

is that we demand the exercise of thinking

from every sane person.

So let's do that. Let's think.

In fact, let's start right now.

Every person in this room do this:

think of the last time you had a decision to make

where you were worried to do the right thing,

where you wondered, "What should I be doing?"

Bring that to mind,

and now reflect on that

and say, "How did I come up that decision?

What did I do? Did I follow my gut?

Did I have somebody vote on it? Or did I punt to legal?"

Or now we have a few more choices.

"Did I evaluate what would be the highest pleasure

like Mill would?

Or like Kant, did I use reason to figure out what was intrinsically right?"

Think about it. Really bring it to mind. This is important.

It is so important

we are going to spend 30 seconds of valuable TEDTalk time

doing nothing but thinking about this.

Are you ready? Go.

Stop. Good work.

What you just did,

that's the first step towards taking responsibility

for what we should do with all of our power.

Now the next step -- try this.

Go find a friend and explain to them

how you made that decision.

Not right now. Wait till I finish talking.

Do it over lunch.

And don't just find another technologist friend;

find somebody different than you.

Find an artist or a writer --

or, heaven forbid, find a philosopher and talk to them.

In fact, find somebody from the humanities.

Why? Because they think about problems

differently than we do as technologists.

Just a few days ago, right across the street from here,

there was hundreds of people gathered together.

It was technologists and humanists

at that big BiblioTech Conference.

And they gathered together

because the technologists wanted to learn

what it would be like to think from a humanities perspective.

You have someone from Google

talking to someone who does comparative literature.

You're thinking about the relevance of 17th century French theater --

how does that bear upon venture capital?

Well that's interesting. That's a different way of thinking.

And when you think in that way,

you become more sensitive to the human considerations,

which are crucial to making ethical decisions.

So imagine that right now

you went and you found your musician friend.

And you're telling him what we're talking about,

about our whole data revolution and all this --

maybe even hum a few bars of our theme music.

♫ Dum ta da da dum dum ta da da dum ♫

Well, your musician friend will stop you and say,

"You know, the theme music

for your data revolution,

that's an opera, that's Wagner.

It's based on Norse legend.

It's Gods and mythical creatures

fighting over magical jewelry."

That's interesting.

Now it's also a beautiful opera,

and we're moved by that opera.

We're moved because it's about the battle

between good and evil,

about right and wrong.

And we care about right and wrong.

We care what happens in that opera.

We care what happens in "Apocalypse Now."

And we certainly care

what happens with our technologies.

We have so much power today,

it is up to us to figure out what to do,

and that's the good news.

We're the ones writing this opera.

This is our movie.

We figure out what will happen with this technology.

We determine how this will all end.

Thank you.

(Applause)