Rupal Patel: Synthetic voices, as unique as fingerprints

I'd like to talk today

about a powerful and fundamental aspect

of who we are: our voice.

Each one of us has a unique voiceprint

that reflects our age, our size,

even our lifestyle and personality.

In the words of the poet Longfellow,

"the human voice is the organ of the soul."

As a speech scientist, I'm fascinated

by how the voice is produced,

and I have an idea for how it can be engineered.

That's what I'd like to share with you.

I'm going to start by playing you a sample

of a voice that you may recognize.

(Recording) Stephen Hawking: "I would have thought

it was fairly obvious what I meant."

Rupal Patel: That was the voice

of Professor Stephen Hawking.

What you may not know is that same voice

may also be used by this little girl

who is unable to speak

because of a neurological condition.

In fact, all of these individuals

may be using the same voice,

and that's because there's only a few options available.

In the U.S. alone, there are 2.5 million Americans

who are unable to speak,

and many of whom use computerized devices

to communicate.

Now that's millions of people worldwide

who are using generic voices,

including Professor Hawking,

who uses an American-accented voice.

This lack of individuation of the synthetic voice

really hit home

when I was at an assistive technology conference

a few years ago,

and I recall walking into an exhibit hall

and seeing a little girl and a grown man

having a conversation using their devices,

different devices, but the same voice.

And I looked around and I saw this happening

all around me, literally hundreds of individuals

using a handful of voices,

voices that didn't fit their bodies

or their personalities.

We wouldn't dream of fitting a little girl

with the prosthetic limb of a grown man.

So why then the same prosthetic voice?

It really struck me,

and I wanted to do something about this.

I'm going to play you now a sample

of someone who has, two people actually,

who have severe speech disorders.

I want you to take a listen to how they sound.

They're saying the same utterance.

(First voice)

(Second voice)

You probably didn't understand what they said,

but I hope that you heard

their unique vocal identities.

So what I wanted to do next is,

I wanted to find out how we could harness

these residual vocal abilities

and build a technology

that could be customized for them,

voices that could be customized for them.

So I reached out to my collaborator, Tim Bunnell.

Dr. Bunnell is an expert in speech synthesis,

and what he'd been doing is building

personalized voices for people

by putting together

pre-recorded samples of their voice

and reconstructing a voice for them.

These are people who had lost their voice

later in life.

We didn't have the luxury

of pre-recorded samples of speech

for those born with speech disorder.

But I thought, there had to be a way

to reverse engineer a voice

from whatever little is left over.

So we decided to do exactly that.

We set out with a little bit of funding from the National Science Foundation,

to create custom-crafted voices that captured

their unique vocal identities.

We call this project VocaliD, or vocal I.D.,

for vocal identity.

Now before I get into the details of how

the voice is made and let you listen to it,

I need to give you a real quick speech science lesson. Okay?

So first, we know that the voice is changing

dramatically over the course of development.

Children sound different from teens

who sound different from adults.

We've all experienced this.

Fact number two is that speech

is a combination of the source,

which is the vibrations generated by your voice box,

which are then pushed through

the rest of the vocal tract.

These are the chambers of your head and neck

that vibrate,

and they actually filter that source sound

to produce consonants and vowels.

So the combination of source and filter

is how we produce speech.

And that happens in one individual.

Now I told you earlier that I'd spent

a good part of my career

understanding and studying

the source characteristics of people

with severe speech disorder,

and what I've found

is that even though their filters were impaired,

they were able to modulate their source:

the pitch, the loudness, the tempo of their voice.

These are called prosody, and I've been documenting for years

that the prosodic abilities of these individuals

are preserved.

So when I realized that those same cues

are also important for speaker identity,

I had this idea.

Why don't we take the source

from the person we want the voice to sound like,

because it's preserved,

and borrow the filter

from someone about the same age and size,

because they can articulate speech,

and then mix them?

Because when we mix them,

we can get a voice that's as clear

as our surrogate talker --

that's the person we borrowed the filter from—

and is similar in identity to our target talker.

It's that simple.

That's the science behind what we're doing.

So once you have that in mind,

how do you go about building this voice?

Well, you have to find someone

who is willing to be a surrogate.

It's not such an ominous thing.

Being a surrogate donor

only requires you to say a few hundred

to a few thousand utterances.

The process goes something like this.

(Video) Voice: Things happen in pairs.

I love to sleep.

The sky is blue without clouds.

RP: Now she's going to go on like this

for about three to four hours,

and the idea is not for her to say everything

that the target is going to want to say,

but the idea is to cover all the different combinations

of the sounds that occur in the language.

The more speech you have,

the better sounding voice you're going to have.

Once you have those recordings,

what we need to do

is we have to parse these recordings

into little snippets of speech,

one- or two-sound combinations,

sometimes even whole words

that start populating a dataset or a database.

We're going to call this database a voice bank.

Now the power of the voice bank

is that from this voice bank,

we can now say any new utterance,

like, "I love chocolate" --

everyone needs to be able to say that—

fish through that database

and find all the segments necessary

to say that utterance.

(Video) Voice: I love chocolate.

RP: So that's speech synthesis.

It's called concatenative synthesis, and that's what we're using.

That's not the novel part.

What's novel is how we make it sound

like this young woman.

This is Samantha.

I met her when she was nine,

and since then, my team and I

have been trying to build her a personalized voice.

We first had to find a surrogate donor,

and then we had to have Samantha

produce some utterances.

What she can produce are mostly vowel-like sounds,

but that's enough for us to extract

her source characteristics.

What happens next is best described

by my daughter's analogy. She's six.

She calls it mixing colors to paint voices.

It's beautiful. It's exactly that.

Samantha's voice is like a concentrated sample

of red food dye which we can infuse

into the recordings of her surrogate

to get a pink voice just like this.

(Video) Samantha: Aaaaaah.

RP: So now, Samantha can say this.

(Video) Samantha: This voice is only for me.

I can't wait to use my new voice with my friends.

RP: Thank you. (Applause)

I'll never forget the gentle smile

that spread across her face

when she heard that voice for the first time.

Now there's millions of people

around the world like Samantha, millions,

and we've only begun to scratch the surface.

What we've done so far is we have

a few surrogate talkers from around the U.S.

who have donated their voices,

and we have been using those

to build our first few personalized voices.

But there's so much more work to be done.

For Samantha, her surrogate

came from somewhere in the Midwest, a stranger

who gave her the gift of voice.

And as a scientist, I'm so excited

to take this work out of the laboratory

and finally into the real world

so it can have real-world impact.

What I want to share with you next

is how I envision taking this work

to that next level.

I imagine a whole world of surrogate donors

from all walks of life, different sizes, different ages,

coming together in this voice drive

to give people voices

that are as colorful as their personalities.

To do that as a first step,

we've put together this website, VocaliD.org,

as a way to bring together those

who want to join us as voice donors,

as expertise donors,

in whatever way to make this vision a reality.

They say that giving blood can save lives.

Well, giving your voice can change lives.

All we need is a few hours of speech

from our surrogate talker,

and as little as a vowel from our target talker,

to create a unique vocal identity.

So that's the science behind what we're doing.

I want to end by circling back to the human side

that is really the inspiration for this work.

About five years ago, we built our very first voice

for a little boy named William.

When his mom first heard this voice,

she said, "This is what William

would have sounded like

had he been able to speak."

And then I saw William typing a message

on his device.

I wondered, what was he thinking?

Imagine carrying around someone else's voice

for nine years

and finally finding your own voice.

Imagine that.

This is what William said:

"Never heard me before."

Thank you.

(Applause)