Abe Davis: New video technology that reveals an object's hidden properties

譯者: Tian Meng 審譯者: Coco Shen

Most of us think of motion as a very visual thing.

絕大部分的人認為 「動作」是非常視覺的。

If I walk across this stage or gesture with my hands while I speak,

如果我走過這個舞台， 或在說話的時候使用手勢，

that motion is something that you can see.

這些動作都是你可以看見的。

But there's a world of important motion that's too subtle for the human eye,

但有一些細微的重要動作， 是人類的眼睛無法看見的。

and over the past few years,

而在過去的幾年中，

we've started to find that cameras

我們發現鏡頭，

can often see this motion even when humans can't.

能夠看到這些人類所看不見的動作。

So let me show you what I mean.

所以讓我來帶大家了解一下，

On the left here, you see video of a person's wrist,

在左邊，你可以看到一個人手腕的影片；

and on the right, you see video of a sleeping infant,

在右邊，你可以看到一個睡著嬰兒的影片。

but if I didn't tell you that these were videos,

但是如果我沒有告訴你這是影片的話，

you might assume that you were looking at two regular images,

你有可能會認為 你只是在看兩張非常普通的圖片。

因為在這兩個情況之下，

because in both cases,

這些影片看起來只是完全靜止的。

these videos appear to be almost completely still.

But there's actually a lot of subtle motion going on here,

事實上這裏發生著許多細微的動作。

and if you were to touch the wrist on the left,

如果你觸碰左邊手腕的話，

you would feel a pulse,

你會感覺到脈搏。

and if you were to hold the infant on the right,

如果你抱起這個右邊嬰兒的話，

you would feel the rise and fall of her chest

你會感覺到她的胸部上下起伏，

as she took each breath.

當她正在呼吸的時候。

And these motions carry a lot of significance,

這些動作十分重要，

but they're usually too subtle for us to see,

但對我們來說太細微 以至於我們看不到。

so instead, we have to observe them

所以我們要用

through direct contact, through touch.

直接的觸碰去感知它們。

But a few years ago,

但是，幾年前，

my colleagues at MIT developed what they call a motion microscope,

我麻省理工的同事 研究出了動作顯微鏡。

which is software that finds these subtle motions in video

這軟體可用來尋找影片中的細微動作，

and amplifies them so that they become large enough for us to see.

然後將它們放大使得我們可以看到。

And so, if we use their software on the left video,

所以說，如果我們在左邊的影片上 使用那個軟體，

it lets us see the pulse in this wrist,

它能讓我們看到腕部的脈搏。

and if we were to count that pulse,

而且如果我們數一數脈搏的話，

we could even figure out this person's heart rate.

我們甚至可以得出這個人的心率。

And if we used the same software on the right video,

如果我們在右邊的影片上用同一軟體的話，

it lets us see each breath that this infant takes,

它可以讓我們看到嬰兒的每一個呼吸。

and we can use this as a contact-free way to monitor her breathing.

我們可以將這個軟體視為 不用接觸就能觀察嬰兒呼吸的探測器。

And so this technology is really powerful because it takes these phenomena

所以這種科技非常強大， 它記錄的現象

是原本我們得觸摸才能感受到的現象，

that we normally have to experience through touch

and it lets us capture them visually and non-invasively.

而且它可以讓我們可視地、無創地觀察他們。

So a couple years ago, I started working with the folks that created that software,

所以數年以前，我開始 和這些開發軟體的人一起工作，

and we decided to pursue a crazy idea.

然後我們決定去追尋一個瘋狂的主意。

We thought, it's cool that we can use software

我們認為用這個軟體發現細小的動作

to visualize tiny motions like this,

是很酷炫的，

and you can almost think of it as a way to extend our sense of touch.

而且可以將它視為我們觸覺的延伸。

But what if we could do the same thing with our ability to hear?

但是如果我們可以 做出擴展我們聽覺的軟體呢？

What if we could use video to capture the vibrations of sound,

如果我們可以通過影片 從而獲得聲音的振動，

which are just another kind of motion,

振動是另外一種動作，

and turn everything that we see into a microphone?

然後將我們看到的所有東西 轉化為聲音進入麥克風呢？

Now, this is a bit of a strange idea,

這是一個有點奇怪的主意，

so let me try to put it in perspective for you.

所以讓我將它變得更加易懂一些。

Traditional microphones work by converting the motion

傳統的麥克風通過將內部隔膜的振動

of an internal diaphragm into an electrical signal,

轉換為電信號，

and that diaphragm is designed to move readily with sound

設計讓隔膜隨著聲音方便移動。

so that its motion can be recorded and interpreted as audio.

它的振動可以被記錄和轉換成聲音。

But sound causes all objects to vibrate.

但是聲音可以使任何物體產生振動。

Those vibrations are just usually too subtle and too fast for us to see.

那些振動與我們來說太小太快， 以至於我們不能看不見。

So what if we record them with a high-speed camera

要是我們用高速錄影機記錄下振動，

and then use software to extract tiny motions

然後用軟體從高速錄影機的影片中

分離出細微的動作，

from our high-speed video,

and analyze those motions to figure out what sounds created them?

然後分析那些動作 並且搞清楚是什麼聲音創造了振動呢？

This would let us turn visible objects into visual microphones from a distance.

這樣我們可以在一定距離內 將可視物體轉換到可視話筒中。

And so we tried this out,

然後我們進行了實驗。

and here's one of our experiments,

這裏可以看到我們的實驗。

where we took this potted plant that you see on the right

在右邊我們放置了一盆盆栽，

and we filmed it with a high-speed camera

然後我們用高速錄影機記錄了下來。

while a nearby loudspeaker played this sound.

同事在邊上用擴音器發出這個聲音。

(Music: "Mary Had a Little Lamb")

（音樂：“瑪麗有隻小綿羊”）

And so here's the video that we recorded,

然後下面是我們記錄下的聲音。

and we recorded it at thousands of frames per second,

我們每秒鐘記錄下上千次畫面，

but even if you look very closely,

但是就算你再仔細地看，

all you'll see are some leaves

你只會看到一些好看的樹葉，

that are pretty much just sitting there doing nothing,

就只是靜止在那什麼也不做。

because our sound only moved those leaves by about a micrometer.

那是因為我們的聲音 只移動了樹葉大約一微米的距離，

That's one ten-thousandth of a centimeter,

那個距離大概是萬分之一釐米，

which spans somewhere between a hundredth and a thousandth

在千分之一和百分之一之間，

of a pixel in this image.

只是這一個圖像的像素點。

So you can squint all you want,

所以你可以盡量瞇著眼睛看，

but motion that small is pretty much perceptually invisible.

但是細小的動作是不容易被感知到的。

But it turns out that something can be perceptually invisible

但結果卻是一些不容易被看到的物體，

and still be numerically significant,

在數字上仍然非常重要。

because with the right algorithms,

因為當使用了正確的演算法之後，

we can take this silent, seemingly still video

我們可以獲取這段看起來靜止無聲的影片，

and we can recover this sound.

然後還原出聲音。

(Music: "Mary Had a Little Lamb")

（音樂：“瑪麗有隻小綿羊”）

(Applause)

（掌聲）

So how is this possible?

這是為什麼呢？

How can we get so much information out of so little motion?

我們是如何在這細小的動作中 得到如此多訊息的呢？

Well, let's say that those leaves move by just a single micrometer,

那麼讓我們假設這些樹葉 只是移動了一點點距離，

and let's say that that shifts our image by just a thousandth of a pixel.

再者樹葉只是移動了 千分之一個像素的距離。

That may not seem like much,

那看起來並不多，

but a single frame of video

但是一個單一幀率的影片，

may have hundreds of thousands of pixels in it,

可能有不計其數的像素。

and so if we combine all of the tiny motions that we see

所以如果我們將這些細小的動作

from across that entire image,

從整個畫面中截取出來的話，可以看到

then suddenly a thousandth of a pixel

一個像素的千分之一

can start to add up to something pretty significant.

可以累計變得十分重要。

On a personal note, we were pretty psyched when we figured this out.

就我個人來說， 我們研究出來時高興得都要瘋了。

(Laughter)

（笑聲）

But even with the right algorithm,

但是就算擁有正確的算法，

we were still missing a pretty important piece of the puzzle.

我們仍然會丟失整個拼圖中最重要的部分。

You see, there are a lot of factors that affect when and how well

你們知道有許多的因素

會對這個技術正常工作造成影響。

this technique will work.

There's the object and how far away it is;

這些因素包括，物體離得有多少遠、

there's the camera and the lens that you use;

拍攝的時候使用的鏡頭、

how much light is shining on the object and how loud your sound is.

有多少光照在物體上，還有放出的聲音多響。

And even with the right algorithm,

而且就算擁有正確的算法，

we had to be very careful with our early experiments,

我們在早期的試驗中必須十分小心。

because if we got any of these factors wrong,

如果說我們弄錯了其中任何一個細節，

there was no way to tell what the problem was.

我們沒有辦法找出問題所在，

We would just get noise back.

只會得到一段噪音。

And so a lot of our early experiments looked like this.

所以我們早期的研究是像這樣的。

And so here I am,

這就是我。

and on the bottom left, you can kind of see our high-speed camera,

在畫面的左下角可以看到我們的高速錄影機，

which is pointed at a bag of chips,

它正對著一包洋芋片，

and the whole thing is lit by these bright lamps.

所有的事物被一盞燈所照亮。

And like I said, we had to be very careful in these early experiments,

就像我說的，我們需要格外小心。

so this is how it went down.

這就是這個試驗如何進行的。

(Video) Abe Davis: Three, two, one, go.

（影片）亞伯戴維斯：三二一開始。

Mary had a little lamb! Little lamb! Little lamb!

瑪麗有隻小綿羊 小綿羊！小綿羊！

(Laughter)

（笑聲）

AD: So this experiment looks completely ridiculous.

亞伯戴維斯：所以這個實驗看起來十分可笑。

(Laughter)

（笑聲）

I mean, I'm screaming at a bag of chips --

我對著一袋洋芋片尖叫。

(Laughter) --

（笑聲）

and we're blasting it with so much light,

我們在充足的光照下對著它大叫，

we literally melted the first bag we tried this on. (Laughter)

我們確實將第一個實驗的洋芋融化了。（笑聲）

But ridiculous as this experiment looks,

儘管看上去很可笑，

it was actually really important,

這確實是十分重要，

because we were able to recover this sound.

因為我們可以復原這個聲音。

(Audio) Mary had a little lamb! Little lamb! Little lamb!

（聲音）瑪麗有隻小綿羊，小綿羊！小綿羊！

(Applause)

（掌聲）

AD: And this was really significant,

這是十分重要的。

because it was the first time we recovered intelligible human speech

因為這是第一次 我們從一個物體靜止的影像中，

from silent video of an object.

復原出了清楚的人聲。

And so it gave us this point of reference,

所以這給了我們一個參考，

and gradually we could start to modify the experiment,

並且可以逐漸去修改這個實驗。

using different objects or moving the object further away,

用不同的物體或者把物體移到更遠的地方，

using less light or quieter sounds.

或者使用少量的光和更加輕的聲音。

And we analyzed all of these experiments

我們分析了實驗，

until we really understood the limits of our technique,

直到我們弄清楚我們技術的侷限性在哪，

because once we understood those limits,

因為只要我們明白它們的極限，

we could figure out how to push them.

就可以知道如何去推動它們。

And that led to experiments like this one,

我們的實驗就可能變成這一個，

where again, I'm going to speak to a bag of chips,

我在同一個地方 再一次向一包洋芋片說話，

but this time we've moved our camera about 15 feet away,

但是這一次我們把攝影機往後移動了15英尺，

outside, behind a soundproof window,

放置在隔音玻璃後面，

and the whole thing is lit by only natural sunlight.

所有的東西僅僅是被太陽光所照亮。

And so here's the video that we captured.

這是我們拍攝到的影片。

And this is what things sounded like from inside, next to the bag of chips.

聽起來這個聲音是從洋芋片內部發出來的。

(Audio) Mary had a little lamb whose fleece was white as snow,

（聲音）瑪麗有隻小綿羊， 牠的毛白得像雪一樣。

and everywhere that Mary went, that lamb was sure to go.

並且瑪麗走到哪裏， 小綿羊就跟到哪裏。

AD: And here's what we were able to recover from our silent video

這是我們能夠通過在玻璃外面捕捉的

captured outside behind that window.

靜止影像中還原出來的。

(Audio) Mary had a little lamb whose fleece was white as snow,

（聲音）瑪麗有隻小綿羊， 牠的毛白得像雪一樣。

and everywhere that Mary went, that lamb was sure to go.

並且瑪麗走到哪裏， 小綿羊就跟到哪裏。

(Applause)

（掌聲）

AD: And there are other ways that we can push these limits as well.

還有其他辦法去推動這些限制，

So here's a quieter experiment

所以下面是一個更安靜的實驗。

where we filmed some earphones plugged into a laptop computer,

我們拍攝了一些插在電腦上的耳機。

and in this case, our goal was to recover the music that was playing on that laptop

我們的目標是還原出 在手提電腦上所放出的聲音，

from just silent video

從這兩個小耳機中的

of these two little plastic earphones,

從靜止影片中得到。

and we were able to do this so well

並且我們可以做得很好，

that I could even Shazam our results.

甚至能夠用聽歌識曲軟體鑑別我們的結果。

(Laughter)

（笑聲）

(Music: "Under Pressure" by Queen)

（音樂：皇后樂隊 “壓力之下”）

(Applause)

（掌聲）

And we can also push things by changing the hardware that we use.

我們也可以通過改變硬體來推動事物。

Because the experiments I've shown you so far

我給你們看的這些實驗

were done with a camera, a high-speed camera,

都使用了攝影機，高速攝影機，

that can record video about a 100 times faster

我們可以比大多數手機

than most cell phones,

快一百倍地記錄影片。

but we've also found a way to use this technique

但是我們也找到了用普通攝影機

with more regular cameras,

使用這一項技術的方法。

and we do that by taking advantage of what's called a rolling shutter.

我們採用普通照相機 像百葉窗一樣記錄東西優點來記錄。

You see, most cameras record images one row at a time,

你們知道，大多數照相機 一段時間記錄一排的圖像，

and so if an object moves during the recording of a single image,

所以如果一個物體 只在被記錄的圖像中移動，

there's a slight time delay between each row,

在記錄的每一排之間幾乎沒有延遲。

and this causes slight artifacts

這樣就可以使物體

that get coded into each frame of a video.

被記錄到影片的每一個部分之中。

And so what we found is that by analyzing these artifacts,

我們發現透過分析這些東西，

we can actually recover sound using a modified version of our algorithm.

實際上只是用了更改過的算法來還原出聲音。

So here's an experiment we did

下面就是我們所做的實驗。

where we filmed a bag of candy

我們拍攝了一袋糖，

while a nearby loudspeaker played

同時邊上有一個擴音器正在播放著

the same "Mary Had a Little Lamb" music from before,

與之前相同的“瑪麗有隻小綿羊”。

but this time, we used just a regular store-bought camera,

但是這一次我們僅使用從商店買來的攝影機。

and so in a second, I'll play for you the sound that we recovered,

馬上我就向你們播放我們還原出的聲音。

and it's going to sound distorted this time,

這一次聲音聽起來有一些扭曲，

but listen and see if you can still recognize the music.

但是請聽聽看能否分辨出這音樂。

(Audio: "Mary Had a Little Lamb")

（聲音：“瑪麗有隻小綿羊”）

And so, again, that sounds distorted,

聲音確實被扭曲了，

but what's really amazing here is that we were able to do this

但是神奇的是，我們能夠做這個事情，

with something that you could literally run out

運用一些用完以後

and pick up at a Best Buy.

就可以在 Best Buy 買到的東西。

So at this point,

所以在這時

a lot of people see this work,

很多人可以看到我們的研究結果，

and they immediately think about surveillance.

然後他們立刻會想到監視。

And to be fair,

公平的說，

it's not hard to imagine how you might use this technology to spy on someone.

不難想到你們可以用這項技術去監視其他人。

But keep in mind that there's already a lot of very mature technology

但是要記住早就有許多成熟的技術

out there for surveillance.

為監視所準備。

In fact, people have been using lasers

事實上，人們數十年來使用雷射

to eavesdrop on objects from a distance for decades.

去竊聽別的事物。

But what's really new here,

但是，這個技術新穎的地方、

what's really different,

完全不同的地方，

is that now we have a way to picture the vibrations of an object,

是我們現在有辦法拍攝出物體的振動。

which gives us a new lens through which to look at the world,

讓我們獲得了觀察這個世界的新鏡頭，

and we can use that lens

並且可以使用這鏡頭，

to learn not just about forces like sound that cause an object to vibrate,

不僅僅是為了去瞭解導致物體振動的聲音，

but also about the object itself.

還瞭解了物體本身。

And so I want to take a step back

所以我想往回退一步

and think about how that might change the ways that we use video,

去思考這個技術會如何改變 我們應用影片的方法。

because we usually use video to look at things,

因為我們用影片通常來看一些東西，

and I've just shown you how we can use it

並且我剛才已經展示如何使用它

to listen to things.

去聽一些東西。

But there's another important way that we learn about the world:

但是另外有一個我們瞭解世界的重要方法，

that's by interacting with it.

那就是和它互動。

We push and pull and poke and prod things.

我們推、拉、戳、刺一些事物，

We shake things and see what happens.

我們搖動物體來明白發生了什麼事。

And that's something that video still won't let us do,

那是影片無法做到的。

at least not traditionally.

至少過去不行。

So I want to show you some new work,

所以我想向你們展示一些新的成品，

and this is based on an idea I had just a few months ago,

這源自我幾個月之前的想法，

so this is actually the first time I've shown it to a public audience.

所以這是我第一次公眾展示。

And the basic idea is that we're going to use the vibrations in a video

而且基本的想法就是 我們會用影片之中的振動，

to capture objects in a way that will let us interact with them

來捕捉物體 在某種程度上這樣可以使我們與物體互動，

and see how they react to us.

並且可以知道它們如何對我們進行反應。

So here's an object,

這是一個物體。

and in this case, it's a wire figure in the shape of a human,

這是一個用線做成的人。

and we're going to film that object with just a regular camera.

我們要用普通的相機去拍攝它，

So there's nothing special about this camera.

所以這個相機沒有什麼特別之處。

In fact, I've actually done this with my cell phone before.

事實上我曾經用我的手機完成過這件事

But we do want to see the object vibrate,

但是我們確實希望這個物體振動。

so to make that happen,

所以為了做到這點，

we're just going to bang a little bit on the surface where it's resting

我們在拍攝的時候

while we record this video.

在它放置的地方敲擊。

So that's it: just five seconds of regular video,

這就是全部了：一個僅僅五秒鐘的普通影片，

while we bang on this surface,

拍攝我們敲擊表面的時候。

and we're going to use the vibrations in that video

我們將要用這個影片的震動

to learn about the structural and material properties of our object,

去瞭解這個物體的結構和組織組成，

and we're going to use that information to create something new and interactive.

然後使用這個訊息 去創造新穎和互動性的東西。

And so here's what we've created.

這就是我們所創造的。

And it looks like a regular image,

這看起來像一個普通的圖片，

but this isn't an image, and it's not a video,

但是這不是圖片也不是影片。

because now I can take my mouse

因為我可以用我的游標，

and I can start interacting with the object.

也可以和我這個物體互動。

And so what you see here

所以你們看到的

is a simulation of how this object

是一個我們從來沒看到過的，

would respond to new forces that we've never seen before,

關於物體如何對新的力量進行反應。

and we created it from just five seconds of regular video.

我們只是使用了五秒鐘的影片。

(Applause)

（掌聲）

And so this is a really powerful way to look at the world,

所以這是個十分有力的看世界的方法，

because it lets us predict how objects will respond

讓我們能推測物體是如何在新環境

to new situations,

做出反應的。

and you could imagine, for instance, looking at an old bridge

並且可以想像，例如看一個古老的橋樑，

and wondering what would happen, how would that bridge hold up

並思考開車經過那座橋時

if I were to drive my car across it.

它會如何支撐住。

And that's a question that you probably want to answer

那是一個在你開車穿過之前，

before you start driving across that bridge.

你會想先知道答案的問題。

And of course, there are going to be limitations to this technique,

的確，這項技術還是有侷限性的，

just like there were with the visual microphone,

就像視覺麥克風也有缺陷一樣，

but we found that it works in a lot of situations

但是我們發現它適用於許多情況，

that you might not expect,

你可能沒有想到的，

especially if you give it longer videos.

尤其是拍攝更長影片的時候。

So for example, here's a video that I captured

例如，這是我拍的一段影片，

of a bush outside of my apartment,

是我公寓外的灌木叢。

and I didn't do anything to this bush,

我沒有對這灌木叢做什麼事，

but by capturing a minute-long video,

但是透過拍攝一段一分鐘的影片，

a gentle breeze caused enough vibrations

一陣輕風可以產生足夠的振動，

that we could learn enough about this bush to create this simulation.

我們可以足夠地瞭解這個灌木叢 從而創造出這樣的模擬情況。

(Applause)

（掌聲）

And so you could imagine giving this to a film director,

所以你可以想像將這個技術給一個電影導演，

and letting him control, say,

讓他來控制

the strength and direction of wind in a shot after it's been recorded.

影片拍攝完後的風力強度和方向。

Or, in this case, we pointed our camera at a hanging curtain,

我們也將相機指向了一個掛著的窗簾，

and you can't even see any motion in this video,

你幾乎看不到影片中有任何動作，

but by recording a two-minute-long video,

但是拍攝兩分鐘的影片後，

natural air currents in this room

在這個房間中的天然氣流

created enough subtle, imperceptible motions and vibrations

創造了足夠細微、不可被察覺的動作和振動，

that we could learn enough to create this simulation.

這樣我們也可以透過振動製造出模擬。

And ironically,

可笑的是，

we're kind of used to having this kind of interactivity

我們只是在

when it comes to virtual objects,

虛擬的物體上，

when it comes to video games and 3D models,

電視遊戲和3D模型中使用這種互動。

but to be able to capture this information from real objects in the real world

但是僅僅使用簡單普通的影片

using just simple, regular video,

去捕捉現實世界中的真實物體，

is something new that has a lot of potential.

仍然有很大的潛力。

So here are the amazing people who worked with me on these projects.

這裏有許多傑出的人 與我共同研究這些計劃。

(Applause)

（掌聲）

And what I've shown you today is only the beginning.

我今天展示給你們看的只是個開始。

We've just started to scratch the surface

我們僅僅開始挖出表面的一部分，

of what you can do with this kind of imaging,

看看這樣的成像技術能做到什麼事。

because it gives us a new way

因為它給了我們一個新的方法

to capture our surroundings with common, accessible technology.

透過平常可得到的技術去捕捉周圍的東西。

And so looking to the future,

所以展望未來，

it's going to be really exciting to explore

探索這個技術 可以告訴我們關於這個世界

what this can tell us about the world.

會變得格外激動人心。

Thank you.

謝謝。

(Applause)

（掌聲）