Can a robot pass a university entrance exam? | Noriko Arai

Translator: Benjamin Hilligsøe Harvey Reviewer: Jette Thrane

Today, I'm going to talk about AI and us.

I dag, skal jeg tale om kunstig intelligens og os.

AI researchers have always said

Forskere har altid sagt

that we humans do not need to worry,

at vi mennesker ikke skal bekymrer os,

because only menial jobs will be taken over by machines.

fordi kun det kedeligste arbejde bliver overtaget af maskiner.

Is that really true?

Er det virkelig sandt?

They have also said that AI will create new jobs,

De siger også, at kunstig intelligens skaber nye arbejdspladser,

so those who lose their jobs will find a new one.

så de der mister jobbet kan finde et nyt.

Of course.

Selvfølgelig.

But the real question is:

Men, det rigtige spørgsmål er:

How many of those who may lose their jobs to AI

Hvor mange af dem som mister deres arbejde til kunstig intelligens

will be able to land a new one,

vil kunne finde nyt

especially when AI is smart enough to learn better than most of us?

når kunstig intelligens er bedre til at lære, end de fleste mennesker?

Let me ask you a question:

Lade mig spørge jer om noget:

How many of you think

Hvor mange af jer tror

that AI will pass the entrance examination of a top university by 2020?

at kunstig intelligens kan bestå en optagelsesprøve til universitetet inden 2020?

Oh, so many. OK.

Nå, så mange! Ok.

So some of you may say, "Of course, yes!"

Nogle af jer siger, "Ja, selvfølgelig!"

Now singularity is the issue.

og tænker singularitet.

And some others may say, "Maybe,

Og andre siger, "Måske,

because AI already won against a top Go player."

fordi kunstig intelligens har slået en top-Go spiller."

And others may say, "No, never. Uh-uh."

Stadig andre siger, "Nej, aldrig. Slet ikke."

That means we do not know the answer yet, right?

Det må jo betyde at vi endnu ikke kender svaret?

So that was the reason why I started Todai Robot Project,

Det var grunden til at jeg startede Todai Robot projektet

making an AI which passes the entrance examination

og skabte en kunstig intelligens som består optagelsesprøven

of the University of Tokyo,

på Tokyo Universitet,

the top university in Japan.

Japans top-universitet.

This is our Todai Robot.

Det er vores Todai Robot.

And, of course, the brain of the robot is working in the remote server.

Og selvfølgelig arbejder robottens hjerne på en selvstændig server.

It is now writing a 600-word essay

Lige nu skriver den en 600-ords stil,

on maritime trade in the 17th century.

om maritimhandel i det 17. århundrede.

How does that sound?

Vildt, ikke?

Why did I take the entrance exam as its benchmark?

Hvorfor valgte jeg prøven som grundlag?

Because I thought we had to study the performance of AI

Fordi vi bør studere hvad kunstig intelligens kan præstere

in comparison to humans,

sammenlignet med mennesker,

especially on the skills and expertise

specielt de evner og ekspertise

which are believed to be acquired only by humans

man tror at kun mennesker mestrer

and only through education.

og kun igennem undervisning.

To enter Todai, the University of Tokyo,

For at blive optaget på Todai, som er Tokyo Universitet

you have to pass two different types of exams.

skal man bestå to forskellige prøver.

The first one is a national standardized test

Først en statslig nationalprøve

in multiple-choice style.

i 'multiple-choice' stil.

You have to take seven subjects

Man skal op i syv fag

and achieve a high score --

og opnå en høj score -

I would say like an 85 percent or more accuracy rate --

nok mere end 84 procent -

to be allowed to take the second stage written test

for at kom videre til den anden prøve,

prepared by Todai.

som Todai står for.

So let me first explain how modern AI works,

Lade mig først forklare hvordan moderne kunstig intelligens virker,

taking the "Jeopardy!" challenge as an example.

ved at bruge "Jeopardy!" som et eksempel.

Here is a typical "Jeopardy!" question:

Her er et typisk "Jeopardy!" spørgsmål:

"Mozart's last symphony shares its name with this planet."

"Mozarts sidste symfoni deler navn med denne planet."

Interestingly, a "Jeopardy!" question always asks,

Interessant nok, så spørger en "Jeopardy!" spørgsmål altid,

always ends with "this" something:

ender altid med "denne" noget,

"this" planet, "this" country,

"denne" planet, "dette" land,

"this" rock musician, and so on.

"denne" rockmusiker, og så videre.

In other words, "Jeopardy!" doesn't ask many different types of questions,

Sagt med andre ord, har "Jeopardy!" ikke mange forskellige typer spørgsmål,

but a single type,

men kun en type,

which we call "factoid questions."

der kaldes "faktuelt spørgsmål".

By the way, do you know the answer?

Forresten, kender du svaret?

If you do not know the answer and if you want to know the answer,

Hvis man ikke kender svaret, og vil vide det,

what would you do?

hvad gøre man så?

You Google, right? Of course.

Man googler det, ikke? Selvfølgelig.

Why not?

Hvorfor ikke?

But you have to pick appropriate keywords

Men man skal vælge relaterede ord,

like "Mozart," "last" and "symphony" to search.

som "Mozart", "sidste", og "symfoni" til at søge med.

The machine basically does the same.

Maskinen gør nogenlunde det samme.

Then this Wikipedia page will be ranked top.

Denne Wikipedia rangerer højt.

Then the machine reads the page.

Så læser maskinen siden.

No, uh-uh.

Nej, desværre ikke.

Unfortunately, none of the modern AIs,

Ingen af de moderne kunstig intelligenser,

including Watson, Siri and Todai Robot,

herunder Watson, Siri og Todai Robot,

is able to read.

kan læse.

But they are very good at searching and optimizing.

Men, de er gode til at søge og optimere.

It will recognize

Den ville genkende

that the keywords "Mozart," "last" and "symphony"

at "Mozart", "sidste", og "symfoni"

are appearing heavily around here.

dukker op mange gange her.

So if it can find a word which is a planet

Så hvis den finder et ord som er en planet

and which is co-occurring with these keywords,

og som dukker op sammen med de andre ord,

that must be the answer.

så må det være svaret.

This is how Watson finds the answer "Jupiter," in this case.

Det er sådan at Watson finder svaret "Jupiter" i dette tilfælde.

Our Todai Robot works similarly, but a bit smarter

Vores Todai Robot fungerer på en lignende måde, men lidt bedre,

in answering history yes-no questions,

når den svarer historiske ja-nej spørgsmål

like, "'Charlemagne repelled the Magyars.' Is this sentence true or false?"

som "Charlemagne udstødte magyarerne".

Our robot starts producing a factoid question,

Robotten skaber selv et faktum spørgsmål:

like: "Charlemagne repelled [this person type]" by itself.

"Charlemagne udstødte [persontype]".

Then, "Avars" but not "Magyars" is ranked top.

Her er det "avarerne", ikke "magyarerne", som rangerer øverst.

This sentence is likely to be false.

Sætningen er sandsynligvis falsk.

Our robot does not read, does not understand,

Vores robot læser ikke, forstår ikke,

but it is statistically correct in many cases.

men er statistisk set korrekt i mange tilfælde.

For the second stage written test,

I anden skriftlige test,

it is required to write a 600-word essay like this one:

skal man skrive en 600-ords stil som denne:

[Discuss the rise and fall of the maritime trade

[Diskuter vækst og fald for maritimhandel

in East and Southeast Asia in the 17th century ...]

i Øst- og Sydøstasien i det 17. århundrede]

and as I have shown earlier,

og som jeg viste tidligere,

our robot took the sentences from the textbooks and Wikipedia,

tog vores robot sætninger fra fagbøger og Wikipedia,

combined them together,

kombinerede dem,

and optimized it to produce an essay

og formede en stil

without understanding a thing.

uden at forstå noget som helst.

(Laughter)

(latter)

But surprisingly, it wrote a better essay

Men overraskende nok skrev den en bedre stil

than most of the students.

en de fleste af de studerende.

(Laughter)

(latter)

How about mathematics?

Hvad med matematik?

A fully automatic math-solving machine

En fuldautomatisk matematikmaskine

has been a dream

har været en drøm

since the birth of the word "artificial intelligence,"

siden fødslen af ordet "kunstig intelligens",

but it has stayed at the level of arithmetic for a long, long time.

men har holdt sig på niveauet af regning i langt, langt tid.

Last year, we finally succeeded in developing a system

Sidste år, lykkes det os til sidst at lave et system

which solved pre-university-level problems from end to end,

som løste komplette opgaver på pre-universitets-niveau,

like this one.

som denne her.

This is the original problem written in Japanese,

Dette er den originale problemformulering på Japansk

and we had to teach it 2,000 mathematical axioms

og vi måtte lære den 2.000 matematiske aksiomer

and 8,000 Japanese words

og 8.000 Japanske ord

to make it accept the problems written in natural language.

før den forstod opgaven på på naturligt sprog.

And it is now translating the original problems

Nu oversætter den de oprindelige opgaver

into machine-readable formulas.

til maskinsprog.

Weird, but it is now ready to solve it, I think.

Underligt, men nu tror jeg at den er klar til at løse den.

Go and solve it.

Løse den så.

Yes! It is now executing symbolic computation.

Ja! Nu udfører den symbolske udregninger.

Even more weird,

Endnu mere underligt,

but probably this is the most fun part for the machine.

men det her er nok den sjoveste del for maskinen.

(Laughter)

(latter)

Now it outputs a perfect answer,

Nu udskriver den et perfekt svar,

though its proof is impossible to read, even for mathematicians.

men beviset er umuligt at læse, selv for matematikere.

Anyway, last year our robot was among the top one percent

Sidste år var vores robot blandt de 1% bedste

in the second stage written exam in mathematics.

i den skriftlige prøve på andet niveau matematik.

(Applause)

(publikum klapper)

Thank you.

Tak.

So, did it enter Todai?

Kom den ind på Todai?

No, not as I expected.

Nej, ikke som jeg forventede.

Why?

Hvorfor?

Because it doesn't understand any meaning.

Fordi den forstår slet ikke mening.

Let me show you a typical error it made in the English test.

Lade mig vise jer en typisk fejl, som den lavede i engelsk prøven.

[Nate: We're almost at the bookstore. Just a few more minutes.

[Nate: Vi er næsten ved boghandleren. Bare et par minutter mere.

Sunil: Wait. ______ . Nate: Thank you! That always happens ...]

Sunil: Vent. _____. Nate: Tak! Det sker altid...]

Two people are talking.

To mennesker taler.

For us, who can understand the situation --

For os, som kan forstå situationen -

[1. "We walked for a long time." 2. "We're almost there."

[1. "Vi gik i lang tid." 2. "Vi er der næsten."

3. "Your shoes look expensive." 4. "Your shoelace is untied."]

3. "Dine sko ser dyre ud." 4. "Dit snørebånd er gået op."]

it is obvious number four is the correct answer, right?

- er det klart at nummer fire er det rigtig svar, ikke?

But Todai Robot chose number two,

Men Todai Robot valgte nummer to,

even after learning 15 billion English sentences

selv efter at havde lært 15 milliarder engelske sætninger

using deep learning technologies.

ved brug af deep-learning teknologier.

OK, so now you might understand what I said:

Ok, så nu forstår i måske hvad jeg sagde,

modern AIs do not read,

moderne kunstig intelligenser læser ikke,

do not understand.

forstår ikke.

They only disguise as if they do.

De lader kun som om at de gør.

This is the distribution graph

Det er fordelingsgrafen

of half a million students who took the same exam as Todai Robot.

af en halv million studerende som tog den samme prøve som Todai Robot.

Now our Todai Robot is among the top 20 percent,

Vores Todai Robot er blandt de 20% bedste,

and it was capable to pass

og den kunne havde bestået

more than 60 percent of the universities in Japan --

på mere end 60% af universiteterne i Japan

but not Todai.

men ikke på Todai.

But see how it is beyond the volume zone

Bemærk at den overgår flokken

of to-be white-collar workers.

af dem som bliver kontorarbejdere.

You might think I was delighted.

I tror måske at jeg var rigtig glad.

After all, my robot was surpassing students everywhere.

Min robot vandt jo over studerende overalt.

Instead, I was alarmed.

Men i sted for, var jeg bekymret.

How on earth could this unintelligent machine outperform students --

Hvordan i alverden kunne den her dumme maskine præstere bedre end studerende -

our children?

vores børn?

Right?

Ikke?

I decided to investigate what was going on in the human world.

Jeg bestemte mig for at undersøge det der skete i den menneskelige verden.

I took hundreds of sentences from high school textbooks

Jeg tog hundredevis af sætninger fra gymnasiebøger

and made easy multiple-choice quizzes,

og skabte nemme 'multiple-choice' opgaver

and asked thousands of high school students to answer.

og fik tusindvis af gymnasiestuderende til at svare.

Here is an example:

Her er et eksempel:

[Buddhism spread to ... , Christianity to ... and Oceania,

[Buddhisme udbredt til ..., kristendom til ... og Oceanien,

and Islam to ...]

og islam til ...]

Of course, the original problems are written in Japanese,

De originale opgaver er på Japansk,

their mother tongue.

deres modersmål.

[ ______ has spread to Oceania.

[ _____ er udbredt til Oceanien.

1. Hinduism 2. Christianity 3. Islam 4. Buddhism ]

1. Hinduisme 2. Kristendom 3. Islam 4. Buddhisme ]

Obviously, Christianity is the answer, isn't it?

Det giver jo sig selv at svaret er kristendom?

It's written!

Det står der jo!

And Todai Robot chose the correct answer, too.

Og Todai Robot valgte også rigtig.

But one-third of junior high school students

Men en 1/3 af eleverne i 1.g

failed to answer this question.

kunne ikke svare på spørgsmålet.

Do you think it is only the case in Japan?

Tror i at det kun er tilfældet i Japan?

I do not think so,

Det tror jeg ikke,

because Japan is always ranked among the top in OECD PISA tests,

for Japan ligger altid i toppen i OECD PISA testen,

measuring 15-year-old students' performance in mathematics,

som vurderer 15-åriges evner i matematik,

science and reading

naturvidenskab og læsning

every three years.

hver tredje år.

We have been believing

Vi har haft opfattelsen

that everybody can learn

at alle kan lære

and learn well,

og lære godt,

as long as we provide good learning materials

så længe det rigtige materiale er til rådighed

free on the web

gratis online

so that they can access through the internet.

så de kan tilgå det via internet.

But such wonderful materials may benefit only those who can read well,

Men disse gode redskaber hjælper måske kun dem der læser godt,

and the percentage of those who can read well

og andelen af dygtige læsere

may be much less than we expected.

er måske meget lavere end vi tror.

How we humans will coexist with AI

Hvordan vi kan sameksistere med kunstig intelligens

is something we have to think about carefully,

er noget vi må tænke nøje over,

based on solid evidence.

baseret på solid information.

At the same time, we have to think in a hurry

Men samtidig må vi tænke hurtigt,

because time is running out.

for tiden er kort.

Thank you.

Tak.

(Applause)

(publikum klapper)

Chris Anderson: Noriko, thank you.

Chris Anderson: Noriko, mange tak.

Noriko Arai: Thank you.

Noriko Arai: Tak.

CA: In your talk, you so beautifully give us a sense of how AIs think,

CA: Din tale giver os et smukt indblik i hvordan kunstig intelligens tænker,

what they can do amazingly

hvad de kan gøre super godt

and what they can't do.

og hvad de ikke kan.

But -- do I read you right,

Men - mener du virkelig,

that you think we really need quite an urgent revolution in education

at vi har brug for en hurtig revolution inden for undervisning,

to help kids do the things that humans can do better than AIs?

for at hjælpe børn til at gøre det vi allerede gør bedre end kunstig intelligens?

NA: Yes, yes, yes.

NA: Ja, ja, ja.

Because we humans can understand the meaning.

Fordi vi mennesker forstår meningen.

That is something which is very, very lacking in AI.

Det er noget som virkelig mangler i kunstig intelligens.

But most of the students just pack the knowledge

Men de fleste elever suger bare viden,

without understanding the meaning of the knowledge,

uden at forstå betydningen af den viden,

so that is not knowledge, that is just memorizing,

så det er ikke at lære noget, det er bare at huske,

and AI can do the same thing.

og kunstig intelligens kan gøre det samme.

So we have to think about a new type of education.

Så vi må opfinde en ny type undervisning.

CA: A shift from knowledge, rote knowledge, to meaning.

CA: En ændring fra viden, tom viden, til mening.

NA: Mm-hmm.

NA: Ja.

CA: Well, there's a challenge for the educators. Thank you so much.

CA: Nå, der er da en udfordring for lærerne. Mange tak.

NA: Thank you very much. Thank you.

NA: Tusind tak. Tak.

(Applause)

(publikum klapper)