Can a robot pass a university entrance exam? | Noriko Arai

Today, I'm going to talk about AI and us.

AI researchers have always said

that we humans do not need to worry,

because only menial jobs will be taken over by machines.

Is that really true?

They have also said that AI will create new jobs,

so those who lose their jobs will find a new one.

Of course.

But the real question is:

How many of those who may lose their jobs to AI

will be able to land a new one,

especially when AI is smart enough to learn better than most of us?

Let me ask you a question:

How many of you think

that AI will pass the entrance examination of a top university by 2020?

Oh, so many. OK.

So some of you may say, "Of course, yes!"

Now singularity is the issue.

And some others may say, "Maybe,

because AI already won against a top Go player."

And others may say, "No, never. Uh-uh."

That means we do not know the answer yet, right?

So that was the reason why I started Todai Robot Project,

making an AI which passes the entrance examination

of the University of Tokyo,

the top university in Japan.

This is our Todai Robot.

And, of course, the brain of the robot is working in the remote server.

It is now writing a 600-word essay

on maritime trade in the 17th century.

How does that sound?

Why did I take the entrance exam as its benchmark?

Because I thought we had to study the performance of AI

in comparison to humans,

especially on the skills and expertise

which are believed to be acquired only by humans

and only through education.

To enter Todai, the University of Tokyo,

you have to pass two different types of exams.

The first one is a national standardized test

in multiple-choice style.

You have to take seven subjects

and achieve a high score --

I would say like an 85 percent or more accuracy rate --

to be allowed to take the second stage written test

prepared by Todai.

So let me first explain how modern AI works,

taking the "Jeopardy!" challenge as an example.

Here is a typical "Jeopardy!" question:

"Mozart's last symphony shares its name with this planet."

Interestingly, a "Jeopardy!" question always asks,

always ends with "this" something:

"this" planet, "this" country,

"this" rock musician, and so on.

In other words, "Jeopardy!" doesn't ask many different types of questions,

but a single type,

which we call "factoid questions."

By the way, do you know the answer?

If you do not know the answer and if you want to know the answer,

what would you do?

You Google, right? Of course.

Why not?

But you have to pick appropriate keywords

like "Mozart," "last" and "symphony" to search.

The machine basically does the same.

Then this Wikipedia page will be ranked top.

Then the machine reads the page.

No, uh-uh.

Unfortunately, none of the modern AIs,

including Watson, Siri and Todai Robot,

is able to read.

But they are very good at searching and optimizing.

It will recognize

that the keywords "Mozart," "last" and "symphony"

are appearing heavily around here.

So if it can find a word which is a planet

and which is co-occurring with these keywords,

that must be the answer.

This is how Watson finds the answer "Jupiter," in this case.

Our Todai Robot works similarly, but a bit smarter

in answering history yes-no questions,

like, "'Charlemagne repelled the Magyars.' Is this sentence true or false?"

Our robot starts producing a factoid question,

like: "Charlemagne repelled [this person type]" by itself.

Then, "Avars" but not "Magyars" is ranked top.

This sentence is likely to be false.

Our robot does not read, does not understand,

but it is statistically correct in many cases.

For the second stage written test,

it is required to write a 600-word essay like this one:

[Discuss the rise and fall of the maritime trade

in East and Southeast Asia in the 17th century ...]

and as I have shown earlier,

our robot took the sentences from the textbooks and Wikipedia,

combined them together,

and optimized it to produce an essay

without understanding a thing.

(Laughter)

But surprisingly, it wrote a better essay

than most of the students.

(Laughter)

How about mathematics?

A fully automatic math-solving machine

has been a dream

since the birth of the word "artificial intelligence,"

but it has stayed at the level of arithmetic for a long, long time.

Last year, we finally succeeded in developing a system

which solved pre-university-level problems from end to end,

like this one.

This is the original problem written in Japanese,

and we had to teach it 2,000 mathematical axioms

and 8,000 Japanese words

to make it accept the problems written in natural language.

And it is now translating the original problems

into machine-readable formulas.

Weird, but it is now ready to solve it, I think.

Go and solve it.

Yes! It is now executing symbolic computation.

Even more weird,

but probably this is the most fun part for the machine.

(Laughter)

Now it outputs a perfect answer,

though its proof is impossible to read, even for mathematicians.

Anyway, last year our robot was among the top one percent

in the second stage written exam in mathematics.

(Applause)

Thank you.

So, did it enter Todai?

No, not as I expected.

Why?

Because it doesn't understand any meaning.

Let me show you a typical error it made in the English test.

[Nate: We're almost at the bookstore. Just a few more minutes.

Sunil: Wait. ______ . Nate: Thank you! That always happens ...]

Two people are talking.

For us, who can understand the situation --

[1. "We walked for a long time." 2. "We're almost there."

3. "Your shoes look expensive." 4. "Your shoelace is untied."]

it is obvious number four is the correct answer, right?

But Todai Robot chose number two,

even after learning 15 billion English sentences

using deep learning technologies.

OK, so now you might understand what I said:

modern AIs do not read,

do not understand.

They only disguise as if they do.

This is the distribution graph

of half a million students who took the same exam as Todai Robot.

Now our Todai Robot is among the top 20 percent,

and it was capable to pass

more than 60 percent of the universities in Japan --

but not Todai.

But see how it is beyond the volume zone

of to-be white-collar workers.

You might think I was delighted.

After all, my robot was surpassing students everywhere.

Instead, I was alarmed.

How on earth could this unintelligent machine outperform students --

our children?

Right?

I decided to investigate what was going on in the human world.

I took hundreds of sentences from high school textbooks

and made easy multiple-choice quizzes,

and asked thousands of high school students to answer.

Here is an example:

[Buddhism spread to ... , Christianity to ... and Oceania,

and Islam to ...]

Of course, the original problems are written in Japanese,

their mother tongue.

[ ______ has spread to Oceania.

1. Hinduism 2. Christianity 3. Islam 4. Buddhism ]

Obviously, Christianity is the answer, isn't it?

It's written!

And Todai Robot chose the correct answer, too.

But one-third of junior high school students

failed to answer this question.

Do you think it is only the case in Japan?

I do not think so,

because Japan is always ranked among the top in OECD PISA tests,

measuring 15-year-old students' performance in mathematics,

science and reading

every three years.

We have been believing

that everybody can learn

and learn well,

as long as we provide good learning materials

free on the web

so that they can access through the internet.

But such wonderful materials may benefit only those who can read well,

and the percentage of those who can read well

may be much less than we expected.

How we humans will coexist with AI

is something we have to think about carefully,

based on solid evidence.

At the same time, we have to think in a hurry

because time is running out.

Thank you.

(Applause)

Chris Anderson: Noriko, thank you.

Noriko Arai: Thank you.

CA: In your talk, you so beautifully give us a sense of how AIs think,

what they can do amazingly

and what they can't do.

But -- do I read you right,

that you think we really need quite an urgent revolution in education

to help kids do the things that humans can do better than AIs?

NA: Yes, yes, yes.

Because we humans can understand the meaning.

That is something which is very, very lacking in AI.

But most of the students just pack the knowledge

without understanding the meaning of the knowledge,

so that is not knowledge, that is just memorizing,

and AI can do the same thing.

So we have to think about a new type of education.

CA: A shift from knowledge, rote knowledge, to meaning.

NA: Mm-hmm.

CA: Well, there's a challenge for the educators. Thank you so much.

NA: Thank you very much. Thank you.

(Applause)