The era of blind faith in big data must end | Cathy O'Neil

Prevodilac: Romeo Mlinar Lektor: Tijana Mihajlović

Algorithms are everywhere.

Algoritmi su svuda.

They sort and separate the winners from the losers.

Filtriraju i odvajaju pobednike od gubitnika.

The winners get the job

Pobednici dobijaju posao

ili dobru ponudu kreditne kartice.

or a good credit card offer.

The losers don't even get an interview

Za gubitnike nema ni intervjua

or they pay more for insurance.

ili plaćaju više za osiguranje.

Ocenjuju nas tajne formule koje ne razumemo,

We're being scored with secret formulas that we don't understand

that often don't have systems of appeal.

za koje često ne postoje sistemi za podnošenje žalbe.

To nas tera se zapitamo:

That begs the question:

What if the algorithms are wrong?

„Šta ako algoritmi greše?“

To build an algorithm you need two things:

Da napravite algoritam potrebne su vam dve stvari:

you need data, what happened in the past,

podaci, šta je bilo u prošlosti,

and a definition of success,

i definicija uspeha,

the thing you're looking for and often hoping for.

koji tražite i kojem se često nadate.

Osposobljavate algoritam posmatranjem i zaključivanjem.

You train an algorithm by looking, figuring out.

Algoritam pronalazi vezu sa uspehom.

The algorithm figures out what is associated with success.

What situation leads to success?

Koja situacija vodi ka uspehu?

Actually, everyone uses algorithms.

Zapravo, svi koriste algoritme,

They just don't formalize them in written code.

ali ih ne formalizuju u pisani kôd.

Evo vam primera.

Let me give you an example.

Svaki dan koristim algoritam da napravim porodici doručak.

I use an algorithm every day to make a meal for my family.

The data I use

Podaci koje koristim su:

is the ingredients in my kitchen,

namirnice u kuhinji,

the time I have,

vreme koje imam na raspolaganju,

the ambition I have,

ambicija koju imam,

and I curate that data.

i te podatke obrađujem.

I don't count those little packages of ramen noodles as food.

U hranu ne računam ona mala pakovanja instant špageta.

(Laughter)

(Smeh)

My definition of success is:

Moja definicija uspeha je -

a meal is successful if my kids eat vegetables.

obrok je uspeo ako moja deca jedu povrće.

To bi izgledalo mnogo drugačije da se moj sin pita.

It's very different from if my youngest son were in charge.

He'd say success is if he gets to eat lots of Nutella.

Njemu bi uspeh bio da se najede nutele.

But I get to choose success.

Ali, ja biram uspeh.

Ja sam zadužena za to. Moje mišljenje je važno.

I am in charge. My opinion matters.

That's the first rule of algorithms.

To je prvo pravilo algoritama.

Algorithms are opinions embedded in code.

Algoritmi su mišljenja ugrađena u kôd.

It's really different from what you think most people think of algorithms.

To se veoma razlikuje od onoga što mislite da većina ljudi misli o algoritmima.

They think algorithms are objective and true and scientific.

Ljudi misle da su algoritmi objektivni, istiniti i naučni.

To je marketinški trik.

That's a marketing trick.

It's also a marketing trick

Marketinški trik jeste

to intimidate you with algorithms,

i kada vas plaše algoritmima,

to make you trust and fear algorithms

kada vas teraju da verujete i da se plašite algoritama

jer verujete matematici i plašite je se.

because you trust and fear mathematics.

A lot can go wrong when we put blind faith in big data.

Mnogo grešaka može se desiti kada slepo verujemo u masovne podatke.

This is Kiri Soares. She's a high school principal in Brooklyn.

Ovo je Kiri Soares, direktor srednje škole u Bruklinu.

In 2011, she told me her teachers were being scored

Godine 2011. rekla mi je da njen kolektiv ocenjuju

with a complex, secret algorithm

složenim, tajnim algoritmom

koji se zove „model dodatne vrednosti“.

called the "value-added model."

I told her, "Well, figure out what the formula is, show it to me.

Rekla sam joj: „Saznaj koja je formula i pokaži mi je.

Objasniću ti je.“

I'm going to explain it to you."

Rekla je: „Pokušala sam dobiti formulu,

She said, "Well, I tried to get the formula,

ali osoba iz Ministarstva obrazovanja mi je rekla da je to matematika

but my Department of Education contact told me it was math

and I wouldn't understand it."

i da je neću razumeti.“

Postaje sve gore.

It gets worse.

The New York Post filed a Freedom of Information Act request,

Njujork Post je podneo zahtev na osnovu zakona o slobodi informacija,

dobio imena i ocene svih nastavnika

got all the teachers' names and all their scores

and they published them as an act of teacher-shaming.

i onda ih objavio kao čin sramoćenja nastavnika.

When I tried to get the formulas, the source code, through the same means,

Kada sam probala istim putem doći do formula, do izvornog kôda,

I was told I couldn't.

rečeno mi je da ne može.

I was denied.

Odbijena sam.

Posle sam saznala

I later found out

da niko u Njujorku nema podatke o toj formuli.

that nobody in New York City had access to that formula.

Niko je nije razumeo.

No one understood it.

Then someone really smart got involved, Gary Rubinstein.

Onda se uključio neko veoma bistar, Geri Rubenstajn.

He found 665 teachers from that New York Post data

Pronašao je 665 nastavnika iz onog članka u Njujork Postu

that actually had two scores.

koji zapravo imaju dva rezultata.

That could happen if they were teaching

Ovo se moglo desiti jer su predavali matematiku u sedmom i osmom razredu.

seventh grade math and eighth grade math.

He decided to plot them.

Odlučio je da ih ubaci u grafikon.

Each dot represents a teacher.

Svaka tačka je nastavnica ili nastavnik.

(Laughter)

(Smeh)

What is that?

Šta je to?

(Laughter)

(Smeh)

To se nikako nije trebalo koristiti za individualne procene.

That should never have been used for individual assessment.

It's almost a random number generator.

Ovo je kao generator nasumičnih brojeva.

(Applause)

(Aplauz)

But it was.

Ali, korišćeno je.

This is Sarah Wysocki.

Ovo je Sara Visoki.

She got fired, along with 205 other teachers,

Otpuštena je kad i 205 drugih nastavnika

iz škola vašingtonskog okruga

from the Washington, DC school district,

even though she had great recommendations from her principal

iako je imala odlučne preporuke direktora

and the parents of her kids.

i roditelja učenika.

Znam šta mnogi od vas ovde sada misle,

I know what a lot of you guys are thinking,

posebno naučnici za podatke, stručnjaci za veštačku inteligenciju.

especially the data scientists, the AI experts here.

You're thinking, "Well, I would never make an algorithm that inconsistent."

Mislite: „Pa, nikada ne bismo napravili tako nedosledan algoritam.“

But algorithms can go wrong,

Ali, algoritmi mogu pogrešiti,

even have deeply destructive effects with good intentions.

čak imati i duboko destruktivno dejstvo sa dobrom namerama.

And whereas an airplane that's designed badly

Dok loše napravljen avion

crashes to the earth and everyone sees it,

padne na tlo i svi to vide,

an algorithm designed badly

loše osmišljen algoritam

može trajati dugo i potajno i tiho praviti ogromnu štetu.

can go on for a long time, silently wreaking havoc.

This is Roger Ailes.

Ovo je Rodžer Ejls.

(Smeh)

(Laughter)

He founded Fox News in 1996.

Osnovao je Foks Njuz 1996. godine.

Preko 20 žena žalilo se na seksualno uznemiravanje.

More than 20 women complained about sexual harassment.

They said they weren't allowed to succeed at Fox News.

Rekle su da im u Foks Njuzu nije dozvoljen uspeh.

Izbačen je prošle godine,

He was ousted last year, but we've seen recently

ali nedavno smo videli da problemi još nisu rešeni.

that the problems have persisted.

That begs the question:

To zahteva da se postavi pitanje

What should Fox News do to turn over another leaf?

šta Foks Njuz treba da uradi da okrene novi list.

Šta bi se desilo da proces zapošljavanja

Well, what if they replaced their hiring process

zamene mašinskim algoritmom koji uči?

with a machine-learning algorithm?

That sounds good, right?

Zvuči dobro, zar ne?

Think about it.

Razmislite o tome.

The data, what would the data be?

Podaci. Šta bi bi bili podaci?

A reasonable choice would be the last 21 years of applications to Fox News.

Ima smisla izabrati prijave za Foks Njuz tokom poslednjih 21 godina.

Reasonable.

Ima smisla.

What about the definition of success?

A definicija uspeha?

Reasonable choice would be,

Razuman izbor bio bi, valjda,

neko ko je uspešan u Foks Njuzu?

well, who is successful at Fox News?

I guess someone who, say, stayed there for four years

Recimo, osoba koja je tamo bila četiri godine

and was promoted at least once.

i dobila unapređenje makar jednom.

Sounds reasonable.

Ima smisla.

And then the algorithm would be trained.

Onda bismo osposobljavali algoritam.

It would be trained to look for people to learn what led to success,

Osposobili bismo ga da traži ljude, da uči šta je vodilo ka uspehu,

kakve vrste prijava su vremenom vodile ka uspehu

what kind of applications historically led to success

by that definition.

u skladu sa tom definicijom.

Razmislite sada šta bi se desilo

Now think about what would happen

if we applied that to a current pool of applicants.

kada bismo to primenili na trenutne kandidate.

It would filter out women

Izbacilo bi žene

because they do not look like people who were successful in the past.

jer ne deluju kao osobe koje su bile uspešne u prošlosti.

Algorithms don't make things fair

Algoritmi ne popravljaju stvari

ako ih samo nonšalantno i slepo primenjujete.

if you just blithely, blindly apply algorithms.

They don't make things fair.

Ne popravljaju stvari.

They repeat our past practices,

Ponavljaju našu praksu iz prošlosti,

our patterns.

naše šablone.

They automate the status quo.

Automatizuju status kvo.

That would be great if we had a perfect world,

Da živimo u savršenom svetu to bi bilo sjajno,

but we don't.

ali ne živimo.

Dodaću da većina firmi nema sramne parnice,

And I'll add that most companies don't have embarrassing lawsuits,

but the data scientists in those companies

ali naučnicima za podatke u tim kompanijama

are told to follow the data,

rečeno je idu tragom podataka,

da paze na tačnost.

to focus on accuracy.

Razmislite šta to znači.

Think about what that means.

Because we all have bias, it means they could be codifying sexism

Pošto smo svi pristrasni, to znači da će možda kodifikovati seksizam

or any other kind of bigotry.

ili drugu netrpeljivost.

Thought experiment,

Misaoni eksperiment,

because I like them:

jer ih volim:

an entirely segregated society --

jedno društvo, skroz podeljeno -

na osnovu rase, svi gradovi, sve opštine -

racially segregated, all towns, all neighborhoods

and where we send the police only to the minority neighborhoods

a policiju šaljemo samo u delove gde živi manjina

to look for crime.

u potrazi za kriminalom.

The arrest data would be very biased.

Podaci o hapšenjima bili bi veoma pristrasni.

What if, on top of that, we found the data scientists

Šta ako bismo, povrh svega, pronašli naučnike za podatke

and paid the data scientists to predict where the next crime would occur?

i platili tim naučnicima da predvide mesto sledećeg zločina?

Opštine sa manjinama.

Minority neighborhood.

Ili da predvide ko će sledeći biti kriminalac?

Or to predict who the next criminal would be?

A minority.

Neko iz manjine.

The data scientists would brag about how great and how accurate

Naučnici bi se hvalisali svojim sjajnim i tačnim modelom,

their model would be,

and they'd be right.

i bili bi u pravu.

Now, reality isn't that drastic, but we do have severe segregations

Realnost nije tako drastična, ali postoje ozbiljne podele

in many cities and towns,

u mnogim malim i velikom gradovima,

and we have plenty of evidence

i imamo mnoštvo dokaza

of biased policing and justice system data.

o pristrasnim podacima u sistemu policije i pravosuđa.

And we actually do predict hotspots,

Mi zapravo predviđamo krizna mesta,

places where crimes will occur.

mesta gde će se desiti nasilje.

I predviđamo, zapravo, pojedinačni kriminalitet,

And we do predict, in fact, the individual criminality,

kriminalitet pojedinaca.

the criminality of individuals.

The news organization ProPublica recently looked into

Novinska organizacija Propablika nedavno je proverila

one of those "recidivism risk" algorithms,

jedan „algoritam ugrožen recidivizmom“

as they're called,

kako ih zovu,

koje sudije koriste u presudama na Floridi.

being used in Florida during sentencing by judges.

Bernard, on the left, the black man, was scored a 10 out of 10.

Bernard, levo, crnac, dobio je 10 od 10 poena.

Dylan, on the right, 3 out of 10.

Dilan, desno, 3 od 10.

Deset od deset, visok rizik. Tri od deset, nizak rizik.

10 out of 10, high risk. 3 out of 10, low risk.

They were both brought in for drug possession.

Obojica su privedeni zbog posedovanja droge.

They both had records,

Obojica su imali dosije,

ali Dilan je imao krivično delo,

but Dylan had a felony

but Bernard didn't.

a Bernard nije.

This matters, because the higher score you are,

Ovo je bitno jer što su ti veći poeni,

the more likely you're being given a longer sentence.

veće su šanse da dobiješ dužu kaznu.

O čemu se ovde radi?

What's going on?

Data laundering.

Pranje podataka.

It's a process by which technologists hide ugly truths

Proces kojim tehnolozi sakrivaju ružnu istinu

inside black box algorithms

u crne kutije algoritama

i nazivaju ih objektivnima;

and call them objective;

nazivaju ih meritokratskim.

call them meritocratic.

When they're secret, important and destructive,

Za tajne, važne i destruktivne algoritme

I've coined a term for these algorithms:

sam skovala frazu „oružje za matematičko uništenje“.

"weapons of math destruction."

(Laughter)

(Smeh)

(Aplauz)

(Applause)

They're everywhere, and it's not a mistake.

Oni su svuda i to nije greška.

These are private companies building private algorithms

To su privatne kompanije koje prave privatne algoritme

for private ends.

za privatne ciljeve.

Čak i one već spomenute, za nastavnike i policiju,

Even the ones I talked about for teachers and the public police,

those were built by private companies

napravile su privatne kompanije

i zatim ih prodale vladinim telima.

and sold to the government institutions.

They call it their "secret sauce" --

Zovu ih „tajnim umakom“;

zato nam ništa ne mogu reći o tome.

that's why they can't tell us about it.

It's also private power.

To je i privatna moć.

They are profiting for wielding the authority of the inscrutable.

Zarađuju na korišćenju autoriteta koji se ne može proveriti.

Now you might think, since all this stuff is private

Možda ste pomislili da, pošto je ovo privatno,

and there's competition,

postoji konkurencija;

možda će slobodno tržište rešiti problem.

maybe the free market will solve this problem.

It won't.

Neće.

There's a lot of money to be made in unfairness.

Mnogo se novca može napraviti nepravdom.

Also, we're not economic rational agents.

Uz to, mi nismo ekonomski racionalni činioci.

We all are biased.

Svi smo pristrasni.

We're all racist and bigoted in ways that we wish we weren't,

Svi smo rasisti i netrpeljivi onako kako ne želimo biti

in ways that we don't even know.

u oblicima koje i ne poznajemo.

Ipak, znamo da je to kolektivno

We know this, though, in aggregate,

because sociologists have consistently demonstrated this

jer to sociolozi dosledno dokazuju

with these experiments they build,

eksperimentima koje osmišljavaju,

kada pošalju gomilu prijava za posao,

where they send a bunch of applications to jobs out,

equally qualified but some have white-sounding names

podjednako dobrih, ali neke imaju imena koja zvuče belački

and some have black-sounding names,

a neke koje zvuče kao crnački,

and it's always disappointing, the results -- always.

i uvek su razočaravajući rezultati; uvek.

So we are the ones that are biased,

Tako, mi smo pristrasni

i mi u algoritme ubacujemo pristrasnost

and we are injecting those biases into the algorithms

by choosing what data to collect,

izborom podataka za prikupljanje,

like I chose not to think about ramen noodles --

kao kada sam odlučila da ne mislim o instant-špagetama;

odlučila sam da su nebitne.

I decided it was irrelevant.

But by trusting the data that's actually picking up on past practices

Ako verujemo podacima koji otkrivaju praksu iz prošlosti

and by choosing the definition of success,

i biramo definiciju uspeha,

how can we expect the algorithms to emerge unscathed?

kako onda očekujemo da algoritmi ostanu neoštećeni?

We can't. We have to check them.

Ne možemo. Moramo ih proveriti.

We have to check them for fairness.

Moramo proveriti da li su pravični.

The good news is, we can check them for fairness.

Dobra vest jeste da možemo proveriti jesu li pravični.

Algorithms can be interrogated,

Algoritme možemo ispitati

and they will tell us the truth every time.

i reći će nam istinu svaki put.

And we can fix them. We can make them better.

I možemo ih popraviti. Možemo ih poboljšati.

I call this an algorithmic audit,

To zovem revizijom algoritma

and I'll walk you through it.

i ukratko ću vam je objasniti.

First, data integrity check.

Prvo, provera integriteta podataka.

For the recidivism risk algorithm I talked about,

Zbog algoritma rizika od recidivizma o kojem sam govorila,

a data integrity check would mean we'd have to come to terms with the fact

provera integriteta podataka značila bi prihvatanje činjenice

that in the US, whites and blacks smoke pot at the same rate

da u SAD crnci i belci podjednako puše travu

but blacks are far more likely to be arrested --

ali crnci imaju mnogo više šanse da budu uhapšeni -

četiri ili pet puta, zavisi od kraja.

four or five times more likely, depending on the area.

Kako ta pristrasnost izgleda u drugim kriminalnim oblastima,

What is that bias looking like in other crime categories,

and how do we account for it?

i kako je uzimamo u obzir?

Second, we should think about the definition of success,

Drugo, treba da razmislimo o definiciji uspeha,

da je revidiramo.

audit that.

Remember -- with the hiring algorithm? We talked about it.

Setite se algoritma za zapošljavanje koji smo spomenuli.

Someone who stays for four years and is promoted once?

Osoba koja je tu četiri godine i unapređena je jednom?

Well, that is a successful employee,

Pa, to je uspešan zaposleni,

ali je takođe i zaposleni u skladu sa njihovom kulturom.

but it's also an employee that is supported by their culture.

That said, also it can be quite biased.

Tako i to može biti pristrasno.

We need to separate those two things.

Moramo razdvojiti te dve stvari.

We should look to the blind orchestra audition

Treba da uzmemo slepe audicije za orkestar kao primer.

as an example.

That's where the people auditioning are behind a sheet.

Tamo ljudi konkurišu su iza zastora.

What I want to think about there

Što je meni bitno jeste

is the people who are listening have decided what's important

da ljudi koji slušaju odlučuju šta je bitno

and they've decided what's not important,

i odlučuju su šta nije bitno,

tako da im to ne odvlači pažnju.

and they're not getting distracted by that.

When the blind orchestra auditions started,

Otkad su počele slepe audicije za orkestre,

the number of women in orchestras went up by a factor of five.

broj žena u orkestrima povećao se pet puta.

Zatim, moramo razmotriti tačnost.

Next, we have to consider accuracy.

Tada bi se model dodatne vrednosti za nastavnike odmah raspao.

This is where the value-added model for teachers would fail immediately.

No algorithm is perfect, of course,

Nema savršenog algoritma, naravno,

so we have to consider the errors of every algorithm.

pa moramo razmotriti greške svakog algoritma.

How often are there errors, and for whom does this model fail?

Koliko su greške česte i za koga ovaj model ne funkcioniše?

What is the cost of that failure?

Koja je cena te nefunkcionalnosti?

And finally, we have to consider

Na kraju, moramo razmotriti

the long-term effects of algorithms,

dugoročne efekte algoritama,

the feedback loops that are engendering.

njihove povratne kružne sprege koje se stvaraju.

That sounds abstract,

Ovo zvuči apstraktno,

but imagine if Facebook engineers had considered that

ali zamislite da su Fejsbukovi inženjeri to uzeli u obzir

pre odluke da nam prikažu samo postove naših prijatelja.

before they decided to show us only things that our friends had posted.

I have two more messages, one for the data scientists out there.

Imam još dve poruke, jednu za naučnike koji se bave podacima.

Data scientists: we should not be the arbiters of truth.

Naučnici za podatke - ne treba da budemo sudije istine.

We should be translators of ethical discussions that happen

Treba da budemo prevodioci etičkih rasprava

koje se odvijaju u širem društvu.

in larger society.

(Applause)

(Aplauz)

And the rest of you,

A za vas ostale,

the non-data scientists:

koji niste naučnici za podatke:

this is not a math test.

ovo nije test iz matematike.

This is a political fight.

Ovo je politička borba.

We need to demand accountability for our algorithmic overlords.

Od naših algoritamskih vladara moramo zahtevati odgovornost.

(Applause)

(Aplauz)

The era of blind faith in big data must end.

Doba slepe vere u masovne podatke mora se okončati.

Thank you very much.

Hvala vam mnogo.

(Applause)

(Aplauz)