Kalika Bali: The giant leaps in language technology -- and who's left behind | TED

Transcriber:

翻译人员: yue sun 校对人员: suya f.

我是Kalika Bali， 一位经受培训的语言学家，

也是一名职业技师。

二十年来，我在学术界工作过，

也曾创业、在小公司 和跨国企业工作过。

I'm Kalika Bali, I'm a linguist by training

and a technologist by profession,

研究并开发语言系统

I have worked in academia,

in startups, in small companies and multinationals for over two decades,

我的梦想是看到技术 能够跨越语言壁垒

doing research in and building language technology systems.

作为印度微软实验室的研究员

我的工作领域是语言及语音技术

My dream is to see technology work across the language barrier.

As a researcher at Microsoft Research Labs India

我关注如何能够使技术

I work in the field of language technology and speech technology.

普及全球

无论他们的母语是什么

And I worry about how can we make technology accessible

所以自然语言处理

人工智能，语音技术

to people across the board,

这些非常时髦的术语

you know, irrespective of the language that they speak.

每个人都在讨论什么是自然语言处理

So natural language processing,

artificial intelligence, speech technology,

these are very big words, they are buzzwords right now.

简而言之

这是计算机科学工程的一部分

Everybody is talking about what exactly is NLP or natural language processing.

让机器处理

理解和生成自然语言

So in very simple terms,

this is the part of computer science engineering

也就是人类说的语言

that makes machines process,

当你和机器沟通交流 想要预定你的火车票

understand and generate natural language,

或者机票

which is the language that humans speak.

当你和手机里的电子语言助手说话时

When you are interacting with a bot trying to book your train tickets

这就是自然语言处理

它支撑了整个技术

or flight tickets,

when you are speaking to a voice-based digital assistant in your phone,

但是它是如何运作的呢

自然语言处理是如何运作的

it's natural language processing

最基本的

that underpins the entire technology that makes that work.

就是数据

But how does this work?

大量的关于人类如何使用语言的数据

How does NLP work?

In a very, very basic way,

it's about data.

并特定的算法的技术所处理

So a huge amount of data of how actually humans use language

让机器学会

is then processed by certain algorithms and techniques

人类自然语言的模式

近年来，另外一个很流行的术语，

that make the machines learn the patterns

你可能会经常听到， 就是深度神经网络

of natural language of humans, right?

这是更高级的技术

这个技术是自然语言处理的基石

These days, another buzzword that you hear a lot about is deep neural networks.

我不会具体阐述这个是如何工作的

And these are the advanced techniques

但是你需要理解并记住的是

that underpin a lot of the NLP stuff that happens right now.

这些都需要海量数据

And I will not go into the details of how that works,

but the thing that you really have to understand and keep in mind

自然语言数据

如果你想要一个可以和 你用古吉拉特语交流的语音系统

is that all of this requires a humungous amount of data,

首先你需要

natural language data.

大量的古吉拉特人

If you want a speech system to converse with you in Gujarati,

用本地语言互相交流的数据

the first thing you require

is a lot of data of Gujarati people speaking to each other

2017年，微软开发了 一个语音识别系统

它能够将语音翻译成文字

in their own language.

So 2017, Microsoft came up with a speech recognition system

并且比人类做的都好

这个系统使用了

which was able to transcribe speech into text

2亿个单词进行训练

2018年，

better than a human did.

一个英文转汉语的翻译软件

And this system was trained

实现了将英语翻译成中文的功能

on 200 million transcribed words.

并且达到了和一个双语者 一样的精确程度

In 2018, an English-Chinese machine translation system

这个系统使用了 1800万条双语词对

was able to translate from English to Chinese

as well as any human bilingual could.

这是自然语言处理技术领域

And this was trained on 18 million bilingual sentence pairs.

非常激动人心的时刻

我们曾经在科幻类书籍 和电影中看到的

This is a very, very exciting time in natural language processing

在现实生活中即将实现

and in technology as such.

我们正在实现科技进步的大跨越

You know, we are seeing science fiction, which we had read about and watched,

但是这些巨大进步 仅限于少数几种语言

kind of come true in front of our own eyes.

We are making giant leaps in technical advancement.

所以Monojit Choudhury

我的好朋友

But these giant leaps are limited to very few languages.

兼同事

他正在研究这个领域

他研究了全球语言资源分布

So Monojit Choudhury,

who's like a very good friend of mine

and a colleague,

他发现这符合幂函数分布

he has studied this in some detail

and he has looked at resource distribution across languages in the world.

意味着，有四种语言

阿拉伯语，中文， 英语，西班牙语

And he says that these follow what is called a power-law distribution,

拥有最多的资源

which essentially means that there are four languages,

其他一些语言也可以

Arabic, Chinese, English and Spanish,

which have the maximum amount of resources available.

当前的资源和技术中获利

There are another handful of languages which can also benefit from, you know,

但是全世界90%的语言

the resources and the technology that's available right now.

基本没有

或者有非常少的资源

But there are 90 percent of the world's languages

我们所讨论的变革

避开了全世界5000多种语言

which have no resources

or very little resources available.

现在，这意味着 那些资源丰富的语言

This revolution that we are talking about

已拥有相关的技术

has essentially bypassed 5,000 languages of the world.

所以研究技术人员被它们所吸引

为它们创建更多的技术和资源

Now, what this means is that resource-rich languages

have technologies built for them,

这就陷入了一个富人更富的循环

so researchers and technologists get attracted towards them.

资源稀少的语言只能保持匮乏

They build more technologies for them. They create more resources.

它们没有技术没有研究员

So it's like a rich getting richer kind of a cycle.

这种语言间的鸿沟

And the resource-poor languages stay poor,

越来越大

也就意味着， 不同语言社群的鸿沟

there's no technology for them, nobody works for them.

And this divide, digital divide between languages

正在扩大

is ever-expanding

and by implication also the divide between the communities

所以微软的ellora计划 致力于消除这种隔阂

that speak these languages is expanding.

我们正在努力尝试用创新 的方法产生更多的数据

So in Microsoft, in Project Ellora, we aim to bridge this gap.

更多的技术， 在没有足够资源的情况下

We are trying to see how can we create more data by innovative methods,

以及有哪些应用 可以真正惠顾这些社群

have more techniques to build technology without having a lot of resources,

所以当前，尽管这看起来非常理论化

and what are the applications that can truly benefit these communities.

正如他所说，数据，技术，科技

让我举一个实实在在的例子

So at the moment, this might seem very theoretical,

我是个语言爱好者， 我喜欢语言，喜欢谈论语言

like what is he talking about, data and techniques and technology.

So let me give you a very concrete example here.

让我举一个很多人没有听过的语言

I'm a linguist at heart, I love languages, and that's what I love talking about.

冈德语

冈德语是一种中南地区的达罗毗荼语

So let me tell you about a language that many of you might not know about.

有三百万五个邦的印度人说它

Gondi.

做个比较

Gondi is a South-Central Dravidian language.

五百万人说挪威语

It is spoken by three million people in five states of India.

略少于一百万人说威尔士语

And to put this in some kind of perspective,

所以冈德是一个相对 来说活跃和大的

Norwegian is spoken by five million people

and Welsh by a little under a million.

印度社群

So Gondi is actually a pretty robust and pretty large community

但是根据unesco 的濒危语言图册

冈德语被列为危险的语言

of the Gond tribals in India.

But by UNESCO's Atlas of Languages in Danger,

CGNet Swara是一个 非政府组织，

Gondi is designated vulnerable status.

它为冈德市民提供了 手机端本地新闻

CGNet Swara is an NGO that provides a citizen journalism portal

冈德语没有技术支持

for the Gond community

没有数据，没有可获得的资源

by making local stories accessible through mobile phones.

There's absolutely no tech support for Gondi.

所以所有的内容都是 人工创造，审核和编辑的

There is no data available for Gondi, no resources available for Gondi.

如今，在ellora计划下

我们所做的就是把所有的参与方，

So all content that is created, moderated and edited is done manually.

像CGNet Swara这样 的非政府组织

以及学术机构 比如IIT Naya Raipur，

Now, under Project Ellora,

what we did was that we brought together all the stakeholders,

非盈利组织比如儿童书 出版商Pratham Books，

an NGOs like CGNet Swara,

以及最重要的， 社群中的母语者，集中起来

and academic institutions, like IIIT Naya Raipur,

冈德部落人自己参与到这样的活动中

a not-for-profit children's book publisher,

like Pratham Books,

并第一次参与编辑和翻译儿童书

and most importantly, the speakers of the community.

The Gond tribals themselves participated in this activity

我们才能第一次把200多本 儿童书翻译成冈德语

and for the first time edited and translated children’s books in Gondi.

这样儿童就能够阅读属于 自己语言的故事和书籍

We were able to put out 200 books for the very first time in Gondi,

Adivasi Radio是另外 一种形式的延伸

这是一个微软实验室开放的app

so that the children had access to stories and books in their own language.

把它和其他参与人资源放在一起

Another extension of this was Adivasi Radio,

which was like an app that we built and developed in Microsoft Research,

接入一个印度语文字转语音系统

and then put out there, along with our stakeholders,

它能够将Cgnet Swara 提供的新闻和文章

which takes a Hindi text-to-speech system

用冈德语读出来

and allows it to read out news and articles provided by CGNet Swara

客户可以用这个app来读 和看新闻和各种信息

in Gondi language.

通过自己的文字和语音

Users can now use this app to read,

有趣的是如今这个app还被

watch news and access any information

用来将印度语翻译成冈德语

through text and voice in their own language.

A very interesting thing is that this app is now being used to translate --

这产生了许多平行数据

我们可以使用平行数据

by the community to translate text from Hindi to Gondi.

来创造冈德语的机器翻译系统

这样为冈德人打开了通往世界的窗口

Now, what that will result in is a lot of parallel data,

that we call parallel data,

that will allow us to build machine translation systems for Gondi,

更重要的是我们知道了如何去操作

which will truly open up a window for the Gond community to the world.

我们拥有了一个综合 解决方案，可以将它复制

And what is even more important is now we know how to do this.

到任何

和冈德部落相同的语言社群中

We have the entire pipeline and we can replicate this for any language

还有教育和信息资源中

and any language community

which is in a similar situation as the Gond tribals.

那么谋生呢

Also education -- yes, you know, information access -- yes,

对，我们如何让这些人获得收入

通过这些我们习以为常的电子工具

but what about earning a living?

Right? What about -- how can we make these people earn a living

Vivek Seshadri， 微软实验室的另一个研究院

以及他的合作者 Manu Chopra

through the digital tools that all of us just take for granted these days?

他们设计了一个叫做Karya的

为弱势群体提供电子微任务的平台

Vivek Seshadri, who's another researcher at MSR,

and his collaborator, Manu Chopra,

他的目的是

they've designed a platform called Karya

for providing digital microtasks to the underserved communities.

为这个国家农村和城市贫穷人口

提供有尊严的服务

His aim was basically to find a way to provide a means of dignified labor

他们没有

to the populations, the rural populations

使用电子平台的知识

and the urban poor populations of this country.

不像我们每天使用早已习以为常

They don't have access to all the knowledge

但是

to use the digital platforms

这里有一大群

that all of us use every day without even thinking, right?

能够识字的人想要工作

But ...

我们如何做到呢

Here is a large

karya就是这样一个

literate population that wants to work, right,

能够为这些人群提供 电子世界的入口的平台

and how can we make this possible for them?

你知道

So Karya is one such way

通过找工作完成任务他们可以赚钱

through which this population can get on to the digital world

所以当我们看到这个时感觉不可思议

and, you know,

我们可以用这个做数据收集

through that find work and do tasks that can then earn them money.

所以我们来到Amale，

一个居住了200的

So we saw this and we thought, oh, this is wonderful.

坐落在马哈拉施特拉邦 的Wada地区的小村庄

We could probably use this for data collection as well.

并决定使用Karya收集马拉地语数据

So we went to Amale,

which is a small village of 200 people

我知道你在想什么

我知道肯定很多人马拉地人， 并且在座的听众

in the Wada district of Maharashtra

觉得马拉地语 不是一个资源缺乏的语言

and decided to use Karya to collect Marathi data.

马拉地语的确是 这个国家的主流语言之一

Now, I know what you are thinking --

I'm sure a lot of Marathi speakers also in the audience --

但是就目前的语言技术而言

that Marathi is not a low-resource language.

马拉地语属于少资源语言

Marathi is definitely a mainstream language of the country.

所以我们来到这个村庄

这是一个非常成功的数据收集之旅

But as far as language technology is concerned,

并且这个村庄非常偏远

Marathi is a low-resource language.

So we went to this village

他们没有电视和电力

and we had a very successful data-collection trip.

没有手机信号

And, you know, this village is very remote.

你必须爬上一个小山坡 并且挥舞你的手机

They have no TV, they have no electricity,

如果你想要给别人打电话

they have no mobile signal.

他们不仅给了我们所有的数据

You have to climb a hill and wave your phone around

还有人生中非常重要的一课

if you want to, you know, use your mobile to call anyone.

第一就是我们为自己的语言感到自豪

So they gave us all this data.

Amela人也非常兴奋

But more than that, they gave us very valuable lessons in life.

因为他们这么做能够改进自己的语言

One is this pride in one's own language.

The people of Amale were thrilled to be doing this

第二就是社群的价值

because they were advancing their own language by doing this.

很快这成了全村落共同的努力

人们团结起来完成任务

The second was the value of community.

Very quickly, this became a village community effort.

第三就是讲故事的重要性

People would gather together in tasks and do this together as a group.

Amela非常渴望故事内容

And the third is the importance of storytelling.

白天他们会用Karya来记录故事

People of Amale were so starved of content that in the morning, during the daytime,

然后在夜晚整个村庄会聚集在一起

把这些故事复述给村里的人听

they would do recordings of stories in Karya

and then in the evening they would gather the entire village

作为科学家，我们沉迷于

技术和科学世界

and retell and recount these stories to the village.

什么模型最好

So as scientists, we get so caught up

如何提高模型的准确性

in the science and technology part of what we are doing, you know --

如何建造最好的系统

which is the next best model to have,

但是我们经常忘记 我们为什么这么做：为了人民

how can we increase the accuracy of my system,

how can I build the next best system there is --

任何成功的技术必须 要能够让使用它的人

that we forget the reason why we are doing this: the people.

能够与时俱进

And any successful technology is the one that keeps the people and the users

当我们开始这么做时

我们也意识到科技知识 其中的一小部分

这其中还有一些其他的因素

up front and center.

可能有社会，文化和政策的参与

And when they start doing that,

we also realize that technology is probably a very small part of this

这些都是必须的，就像科技一样

and there are other things in the story.

我曾经参与过一个 叫做Videokheti的项目

Maybe there are social, cultural and policy interventions

它能让印度中部的说印度语的工人

that are required, as much as technology.

通过语音搜索到农业知识视频

So some time back, I worked on a project called VideoKheti

that allowed Hindi-speaking farmers in Central India

于是我们来到 Madhya Pradesh搜集数据

to search for agricultural videos by speaking into a phone-based app.

等我们回去训练我们的模型时

我们才意识到结果非常糟糕

So we went to Madhya Pradesh to collect data for this,

根本不管用

我们很困惑到底哪里出错了

and we came back and we were training our models

我们仔细研究了数据

发现我们的数据

and we discovered we're getting very bad results.

This is not working.

来源于一个我们认为 夜晚很安静的村庄

So we were very confused. Why is this happening?

So we looked deeper and deeper into the data

然而我们没有听到

and discovered that, yes, we had collected data

from what we thought was a very silent, quiet village in the evening.

夜晚的持续的虫鸣声

在录制过程中，这些 虫鸣声也被录进去了

But what we hadn't heard while we were doing this

从而干扰了我们的语音

was that there was this constant buzz of night insects, you know?

第二个原因是

So throughout the recordings, we had this "bzz" of the insects,

我们到达村庄开始测试app时

which was actually distorting our speech.

我和我的同事 Indrani Medhi

The second thing was that when we went there

他是个非常受人尊敬的设计研究者

to kind of test our app in the village,

我们发现这里的女人无法准确读出

I and my colleague Indrani Medhi,

who is a very well-regarded design researcher,

一些梵语化的单词

（梵语化指底层人民习 得上层精英的语言）

比如

we found that the women couldn't pronounce the sanskritized words

【印度语】

that we had for some of the search terms.

比如化学杀虫剂这个单词

So, like ...

因为我们从农业中心获得了这些单词

(speaks Hindi)

Which is like the term for chemical pesticides, right?

但这些女人，尽管她们会耕种

但她们不与这些中心有交流

Because we got these terms from the agricultural extension center

男人会，但是女人会 用一起更简单的单词

and the women, even though they are farming,

比如【印度语】

就是杀死害虫的药

do not interact with that center at all.

The men do, the women probably use something much simpler, like ...

所以在这趟旅程中我意识到

(speaks Hindi)

和我想要传导给你们的

Which basically means killing pests with medicine.

我希望你们能够理解

So what I have learned through my journey

世界上大部分的语言

and what I would like to put across to you --

需要有密集的资源投资

by now, I hope you've understood me,

才能从语言科技中获利

is that there is the majority of the world's languages

而这没有快速有效的方式

that require intensive investment for resource creation

if they are to benefit from language technology.

所以我们必须保证

And this is unlikely to happen in a very fast and efficient manner.

社群能够从

语言科技领域中获得最大收益

So it is extremely important for us to ensure

that the community derives maximum benefit

为了达到为这些社群

提供积极的社会影响的目的

from whatever that we are doing in the language tech area.

我们运用了改良版 的4-D设计思维方法

And to do this and deliver a positive social impact

4-D的意思是：发现， 设计，发展，运用

on these communities,

we follow what we call the modified 4-D design thinking methodology.

所以我们需要发掘语言科技

So the 4-D means: discover, design, develop and deploy.

可以为特定语言社群解决的问题

这种观察为主导的方式 能够帮助我们有效分配资源

So discover the problem that language technology can solve

到他们最需要的地方

为使用者和他们的语言设计

for a particular language community.

This observation-led approach can help allocate resources

了解语言属性的多样性

和世界语言

where they are most needed,

designed for the users and their language,

不要觉得语言技术 是专为英语而设计的

现在，我们如何将它改编 到马拉语或冈德语上

understand the diversity in the linguistic properties

and the languages of the world.

快速发展，高频使用

And don't think, oh, this is made for English.

这是一种快速失败反复改进的过程

Now, how can we just adapt it for Marathi or for Gondi, right?

早期的失败能够带来最终的成功

Develop rapidly and deploy frequently.

It's an iterative process that will help you fail fast

最重要的是坚持

不放弃

and early failures will eventually lead to success.

我记得有一个关于两个 澳大利亚土著女人的故事

The important thing is to persevere.

Patricia O’Connor and Ysola Best

Do not give up.

And I remember the story of these two Aborigine Australian women,

90年代中期， 她们来到昆士兰大学

Patricia O'Connor and Ysola Best.

他们想要学习他们自己 的语言Yugambeh

但是却被直接告知 你们的语言已经死了

In the mid-90s, they went to the University of Queensland

已经死亡三十年了

and they wanted to learn their own language, called Yugambeh,

你不能研究这个，换个课题吧

她们没有放弃

and they were told very bluntly, "Your language is dead.

她们来到社群

It's been dead for three decades.

发掘口头的记忆，传统和文献

You cannot work on this. Find something else to work on."

成立了Yugambeh博物馆

They did not give up.

They went to the community,

they dug up oral memories, oral traditions, oral literature,

这之后变成了Yugambeh语

and founded the Yugambeh Museum,

最重要的文化和语言中心

她们没有技术，只有毅力

which became the most important cultural and linguistic center for the language

现在拥有了科技的力量

我们能够用芬兰的Salmi语，

and its community.

They did not have technology. They only had their willpower.

Now, with the power of technology,

加拿大的Lillooet语和印度 的Mundari语谱写新的篇章

we can ensure that the next page is written in Salmi from Finland,

谢谢

Lillooet from Canada or Mundari from India.

Thank you.