Abe Davis: New video technology that reveals an object's hidden properties

翻译人员: Hong Li 校对人员: Li Li

Most of us think of motion as a very visual thing.

大部分人认为 动作是明显可见的。

If I walk across this stage or gesture with my hands while I speak,

比如我走过这个舞台， 或者边做手势边说话，

that motion is something that you can see.

这些动作都能被大家看到。

But there's a world of important motion that's too subtle for the human eye,

但还有很多重要的动作 肉眼很难察觉到，

and over the past few years,

在过去几年中，

we've started to find that cameras

我们致力于寻找某种摄像机

can often see this motion even when humans can't.

可以捕捉到人眼看不到的运动。

So let me show you what I mean.

请看大屏幕。

On the left here, you see video of a person's wrist,

左边是一个人的手腕，

and on the right, you see video of a sleeping infant,

右边是一个熟睡的婴儿，

but if I didn't tell you that these were videos,

但是如果我不告诉你们这是一段视频，

you might assume that you were looking at two regular images,

你们可能会认为 这只是两张普通的图片，

because in both cases,

因为乍一看，

这两段视频几乎是完全静止的。

these videos appear to be almost completely still.

But there's actually a lot of subtle motion going on here,

但实际上，画面中 有许多细微的运动变化，

and if you were to touch the wrist on the left,

如果你能碰到左边的那个手腕，

you would feel a pulse,

你会感受到脉搏的跳动，

and if you were to hold the infant on the right,

如果你抱起右边的婴儿，

you would feel the rise and fall of her chest

你能感受到她胸腔的起伏，

as she took each breath.

感受到她的每一次呼吸。

And these motions carry a lot of significance,

这些动作都很重要，

but they're usually too subtle for us to see,

但由于过于细微， 很难被我们察觉，

so instead, we have to observe them

要想感受到这些动作的存在

through direct contact, through touch.

只能通过直接接触。

But a few years ago,

然而几年前，

my colleagues at MIT developed what they call a motion microscope,

我在麻省理工学院的同事们 开发出了一种被称为“动作显微镜”的软件，

which is software that finds these subtle motions in video

能够发现视频中细微的运动，

and amplifies them so that they become large enough for us to see.

并将其放大到肉眼可见的级别。

And so, if we use their software on the left video,

如果我们运用这一软件分析左边的视频，

it lets us see the pulse in this wrist,

我们就能看到手腕上的脉搏跳动，

and if we were to count that pulse,

通过计算脉搏数量，

we could even figure out this person's heart rate.

就能得知这个人的心率。

And if we used the same software on the right video,

而用这一软件分析右边的视频，

it lets us see each breath that this infant takes,

我们就能看清婴儿的每一次呼吸，

and we can use this as a contact-free way to monitor her breathing.

不需要触碰就能监控她的呼吸。

And so this technology is really powerful because it takes these phenomena

这项技术非常强大， 因为它能帮助我们看到

that we normally have to experience through touch

原本要靠触觉才能感受到的东西，

and it lets us capture them visually and non-invasively.

并且这一过程是可见和无创的。

So a couple years ago, I started working with the folks that created that software,

因此在几年前，我开始 与这个软件的编写者们一起工作，

and we decided to pursue a crazy idea.

我们产生了一个疯狂的想法。

We thought, it's cool that we can use software

我们觉得，运用软件将细微的动作

to visualize tiny motions like this,

可视化的这个点子非常酷，

and you can almost think of it as a way to extend our sense of touch.

你甚至可以把它当做拓展 人类触觉感官的好方法。

But what if we could do the same thing with our ability to hear?

那如果我们能用相同的方法 来增强我们的听觉呢？

What if we could use video to capture the vibrations of sound,

如果我们能通过视频捕捉到声音的振动，

which are just another kind of motion,

声音的振动实际上也是一种运动，

and turn everything that we see into a microphone?

将“看到”的东西录入麦克风呢？

Now, this is a bit of a strange idea,

也许听起来有点不太好理解，

so let me try to put it in perspective for you.

我试着为大家解释一下。

Traditional microphones work by converting the motion

传统麦克风的工作原理

of an internal diaphragm into an electrical signal,

是将其内部薄膜的振动转换成电信号，

and that diaphragm is designed to move readily with sound

这个薄膜极易随声音振动，

so that its motion can be recorded and interpreted as audio.

这个振动可以被记录下来 并还原成声音。

But sound causes all objects to vibrate.

而声音事实上可以 引起任何物体的振动。

Those vibrations are just usually too subtle and too fast for us to see.

只不过这种振动对我们而言 通常很细微而且转瞬即逝。

So what if we record them with a high-speed camera

但如果我们用高速摄影机 将这种振动录下来，

and then use software to extract tiny motions

并通过软件从这些高速视频中

from our high-speed video,

提取出这些细小的振动，

and analyze those motions to figure out what sounds created them?

然后分析这些振动来 弄清声音的来源，会怎么样呢？

This would let us turn visible objects into visual microphones from a distance.

这样一来我们可以将远处的 可见物体转化为可视化麦克风。

And so we tried this out,

我们进行了各种尝试，

and here's one of our experiments,

以下是我们的试验之一，

where we took this potted plant that you see on the right

右边是一株盆栽植物，

and we filmed it with a high-speed camera

我们用高速摄影机拍下它，

while a nearby loudspeaker played this sound.

同时旁边的音箱在播放这个声音。

(Music: "Mary Had a Little Lamb")

（音乐：玛丽有一只小羊羔）

And so here's the video that we recorded,

这是我们录下的视频，

and we recorded it at thousands of frames per second,

用的是每秒数千帧的速度，

but even if you look very closely,

但即使你凑得非常近，

all you'll see are some leaves

也只能看到一些叶子

that are pretty much just sitting there doing nothing,

静静地呆在那儿，一动不动，

because our sound only moved those leaves by about a micrometer.

因为刚才的音乐 只能让叶子移动一微米，

That's one ten-thousandth of a centimeter,

也就是一厘米的万分之一，

which spans somewhere between a hundredth and a thousandth

只占这幅图像中一个像素的

of a pixel in this image.

百分之一到千分之一。

So you can squint all you want,

你大可以眯着眼使劲儿看，

but motion that small is pretty much perceptually invisible.

但如此细微的运动 从感官上来说是不可见的。

But it turns out that something can be perceptually invisible

但事实证明感官上不可见的东西

and still be numerically significant,

在数值上可能很惊人,

because with the right algorithms,

因为通过正确的算法，

we can take this silent, seemingly still video

我们就可以从这段无声的 看似静止的视频中

and we can recover this sound.

还原出这段声音。

(Music: "Mary Had a Little Lamb")

（音乐：玛丽有一只小羊羔）

(Applause)

（掌声）

So how is this possible?

这怎么可能呢？

How can we get so much information out of so little motion?

我们怎么能从如此细小的运动中 得到如此丰富的信息？

Well, let's say that those leaves move by just a single micrometer,

我们必须承认这些叶子 只移动了一微米，

and let's say that that shifts our image by just a thousandth of a pixel.

只改变了图像中一个像素的千分之一。

That may not seem like much,

看起来很微不足道，

but a single frame of video

但是视频中的每一帧

may have hundreds of thousands of pixels in it,

都包含数以万计的像素，

and so if we combine all of the tiny motions that we see

当我们将整幅画面中 所有细微的运动

from across that entire image,

组合在一起来看的时候，

then suddenly a thousandth of a pixel

无数个千分之一像素聚在一起

can start to add up to something pretty significant.

就能组合出有十分意义的信息。

On a personal note, we were pretty psyched when we figured this out.

老实说，当我们想通 这一点的时候真是乐疯了。

(Laughter)

（笑声）

But even with the right algorithm,

但是，即便运用正确的算法

we were still missing a pretty important piece of the puzzle.

我们还是会丢失掉很多重要的信息。

You see, there are a lot of factors that affect when and how well

这项技术能否成功

this technique will work.

取决于很多因素。

There's the object and how far away it is;

比如目标物体的距离；

there's the camera and the lens that you use;

摄影机和镜头的选用；

how much light is shining on the object and how loud your sound is.

光线是否充足， 声音是否够大等等。

And even with the right algorithm,

因此，即便我们的算法正确，

we had to be very careful with our early experiments,

在早期试验中 我们还是得万分谨慎，

because if we got any of these factors wrong,

因为一着不慎，满盘皆输，

there was no way to tell what the problem was.

得不到有用的信息， 也查不出原因。

We would just get noise back.

还原出来的只有噪音。

And so a lot of our early experiments looked like this.

初期的试验场景是这样的。

And so here I am,

左边的是我，

and on the bottom left, you can kind of see our high-speed camera,

左下角是我们的高速摄影机，

which is pointed at a bag of chips,

正对着一袋薯片，

and the whole thing is lit by these bright lamps.

薯片被一盏明亮的灯照着。

And like I said, we had to be very careful in these early experiments,

就像刚才我说的， 在初期试验中我们需要十分小心，

so this is how it went down.

得有多小心呢？请看。

(Video) Abe Davis: Three, two, one, go.

（视频：三、二、一，开始）

Mary had a little lamb! Little lamb! Little lamb!

（视频：玛丽有一只小羊羔！ 小羊羔！小羊羔！）

(Laughter)

（笑声）

AD: So this experiment looks completely ridiculous.

这试验看起来真是弱爆了。

(Laughter)

（笑声）

I mean, I'm screaming at a bag of chips --

我可是对着一袋薯片在咆哮——

(Laughter) --

（笑声）

and we're blasting it with so much light,

而且我们用的灯功率太大，

we literally melted the first bag we tried this on. (Laughter)

差点把第一袋薯片点着了。 （笑声）

But ridiculous as this experiment looks,

虽然看起来很不靠谱，

it was actually really important,

但结果还是不错的，

because we were able to recover this sound.

因为我们最终还原出了这段声音。

(Audio) Mary had a little lamb! Little lamb! Little lamb!

（音频：玛丽有一只小羊羔！ 小羊羔！小羊羔！）

(Applause)

（掌声）

AD: And this was really significant,

这绝对是一个里程碑，

because it was the first time we recovered intelligible human speech

因为这是我们第一次 从一段无声录像中

from silent video of an object.

还原出具有意义的人声。

And so it gave us this point of reference,

因此我们以此为出发点

and gradually we could start to modify the experiment,

不断修正我们的试验，

using different objects or moving the object further away,

更换试验对象，调整距离，

using less light or quieter sounds.

减小光线强度，降低声音等等。

And we analyzed all of these experiments

我们不断分析试验结果，

until we really understood the limits of our technique,

直到发现这一技术的局限性，

because once we understood those limits,

因为只有搞清楚局限在哪儿

we could figure out how to push them.

我们才能不断取得突破。

And that led to experiments like this one,

于是，就有了下面这个试验，

where again, I'm going to speak to a bag of chips,

这一次，我还是对着一袋薯片说话，

but this time we've moved our camera about 15 feet away,

但将摄影机后退到了15英尺 （4.572米）远的室外，

outside, behind a soundproof window,

隔着一层隔音玻璃，

and the whole thing is lit by only natural sunlight.

只借助自然光线。

And so here's the video that we captured.

这是我们拍下的视频。

And this is what things sounded like from inside, next to the bag of chips.

这是在室内， 在薯片旁说话的原声。

(Audio) Mary had a little lamb whose fleece was white as snow,

（音频：玛丽有一只小羊羔， 身上羊毛白又好，

and everywhere that Mary went, that lamb was sure to go.

无论玛丽走到哪， 小羊都会跟着跑。）

AD: And here's what we were able to recover from our silent video

这是通过我们从室外 隔音玻璃后采集的无声影像

captured outside behind that window.

还原出来的声音。

(Audio) Mary had a little lamb whose fleece was white as snow,

（音频：玛丽有一只小羊羔， 身上羊毛白又好，

and everywhere that Mary went, that lamb was sure to go.

无论玛丽走到哪， 小羊都会跟着跑。）

(Applause)

（掌声）

AD: And there are other ways that we can push these limits as well.

我们还调整了其它参数。

So here's a quieter experiment

比如说降低音量，

where we filmed some earphones plugged into a laptop computer,

这有一副耳机，插在笔记本电脑上，

and in this case, our goal was to recover the music that was playing on that laptop

在这个实验中，我们想仅通过拍摄下 这对塑料耳机的

from just silent video

无声视频来还原

of these two little plastic earphones,

笔记本里播放的音乐，

and we were able to do this so well

结果很理想，

that I could even Shazam our results.

我甚至能用Shazam 来识别出这段音乐。

(Laughter)

（笑声）

(Music: "Under Pressure" by Queen)

（音乐：“皇后乐队”的《重压之下》）

(Applause)

（掌声）

And we can also push things by changing the hardware that we use.

我们还尝试了更换试验设备 来完善我们的成果。

Because the experiments I've shown you so far

因为前面我给大家展示的试验

were done with a camera, a high-speed camera,

都是通过高速摄影机完成的，

that can record video about a 100 times faster

它的拍摄速度比大多数手机摄像头

than most cell phones,

快100倍，

but we've also found a way to use this technique

但是我们也找到了用普通摄影机

with more regular cameras,

来完成试验的方法，

and we do that by taking advantage of what's called a rolling shutter.

我们利用了叫做“滚动快门”的技术。

You see, most cameras record images one row at a time,

大部分摄像头是逐行拍摄影像的，

and so if an object moves during the recording of a single image,

因此如果在拍摄单张照片时 物体发生了移动，

there's a slight time delay between each row,

每一行影像间就会出现少许延迟，

and this causes slight artifacts

这种延迟使得视频的每一帧

that get coded into each frame of a video.

都会产生轻微的变形。

And so what we found is that by analyzing these artifacts,

通过分析这种变形，

we can actually recover sound using a modified version of our algorithm.

运用调整过的算法 我们还是可以还原声音。

So here's an experiment we did

在接下来这个试验里，

where we filmed a bag of candy

我们拍摄的是一袋糖果，

while a nearby loudspeaker played

旁边的喇叭里播放的

the same "Mary Had a Little Lamb" music from before,

还是之前那首“玛丽有一只小羊羔”，

but this time, we used just a regular store-bought camera,

但这一次我们使用的是 能在店里买到的普通摄影机，

and so in a second, I'll play for you the sound that we recovered,

下面请听我们还原出来的声音，

and it's going to sound distorted this time,

这次的声音有些失真，

but listen and see if you can still recognize the music.

但仔细听一下， 看你能否分辨出来这段音乐。

(Audio: "Mary Had a Little Lamb")

（音频：玛丽有一只小羊羔）

And so, again, that sounds distorted,

就是这样，听起来有点失真，

but what's really amazing here is that we were able to do this

但别忘了 我们这次用的是普通摄影机，

with something that you could literally run out

你随便到一家百思买 这样的电器商店

and pick up at a Best Buy.

就可以买到。

So at this point,

那么目前为止，

a lot of people see this work,

相信许多人看到这儿

and they immediately think about surveillance.

立刻想到了监听。

And to be fair,

说实话，

it's not hard to imagine how you might use this technology to spy on someone.

用这个技术去监听 还真不是什么难事。

But keep in mind that there's already a lot of very mature technology

但请大家注意， 早就有很多成熟的技术

out there for surveillance.

被用于监听了。

In fact, people have been using lasers

实际上，将激光投射在物体上

to eavesdrop on objects from a distance for decades.

进行远距离监听的技术 已经出现几十年了。

But what's really new here,

但我们这项技术的创新之处，

what's really different,

与众不同之处

is that now we have a way to picture the vibrations of an object,

在于我们掌握了一种 描绘物体振动的方法，

which gives us a new lens through which to look at the world,

使我们能通过一种全新的镜头 去看这个世界。

and we can use that lens

通过这个镜头，

to learn not just about forces like sound that cause an object to vibrate,

不仅能看清使物体产生振动的外力， 比如声音，

but also about the object itself.

还能了解物体本身的性质。

And so I want to take a step back

因此我想换个角度

and think about how that might change the ways that we use video,

思考这将如何改变 我们使用视频的方式，

because we usually use video to look at things,

我们通常用视频来“看”东西，

and I've just shown you how we can use it

而我刚刚给大家展示的是如何用视频

to listen to things.

来“听”东西。

But there's another important way that we learn about the world:

但是还有一种认识世界的重要方式，

that's by interacting with it.

就是与世界互动。

We push and pull and poke and prod things.

我们可以移动或触碰某个物体。

We shake things and see what happens.

或者摇晃它，看它会发生什么变化。

And that's something that video still won't let us do,

但这一变化（可能太过微小） 视频没法捕捉，

at least not traditionally.

至少用传统的方式实现不了。

So I want to show you some new work,

因此我想向大家展示一项新的成果，

and this is based on an idea I had just a few months ago,

这项成果基于我几个月前的一个想法，

so this is actually the first time I've shown it to a public audience.

今天其实是我第一次将它公之于众。

And the basic idea is that we're going to use the vibrations in a video

简而言之就是， 我们会利用视频里的振动

to capture objects in a way that will let us interact with them

来与物体进行互动，

and see how they react to us.

然后看物体如何反应。

So here's an object,

这是我们的试验对象，

and in this case, it's a wire figure in the shape of a human,

一个用铁丝做成的小人，

and we're going to film that object with just a regular camera.

我们使用的是一台普通的摄影机。

So there's nothing special about this camera.

没有任何特别之处。

In fact, I've actually done this with my cell phone before.

实际上，我用手机也能做到。

But we do want to see the object vibrate,

但如果我们想让这个小人振动，

so to make that happen,

要怎么做呢，

we're just going to bang a little bit on the surface where it's resting

我们仅仅在放置小人的 台子上敲了几下，

while we record this video.

并把过程拍了下来。

So that's it: just five seconds of regular video,

就这样，我们得到了一段 五秒钟的普通视频，

while we bang on this surface,

敲了几下台子，

and we're going to use the vibrations in that video

我们将利用视频里的振动

to learn about the structural and material properties of our object,

来研究这个小人的 结构特征和材料特征，

and we're going to use that information to create something new and interactive.

并利用这些信息 创造出一种新的具有互动性的东西。

And so here's what we've created.

这就是我们的成果

And it looks like a regular image,

看起来像一张普通的图片，

but this isn't an image, and it's not a video,

但这不是图片， 也不是视频，

because now I can take my mouse

因为我可以移动鼠标

and I can start interacting with the object.

与这个小人进行互动。

And so what you see here

现在大家看到的

is a simulation of how this object

是模拟小人在受到外力时

would respond to new forces that we've never seen before,

会如何反应， 即使这种外力是初次施加的，

and we created it from just five seconds of regular video.

而这都来源于那 短短五秒钟的普通视频。

(Applause)

（掌声）

And so this is a really powerful way to look at the world,

这的确是一种审视世界的有效方法，

because it lets us predict how objects will respond

让我们可以预测物体在新的条件下

to new situations,

会作何反应，

and you could imagine, for instance, looking at an old bridge

想象一下，前面有一座很旧的桥，

and wondering what would happen, how would that bridge hold up

我们不知道它是否足够结实，

if I were to drive my car across it.

我们能不能把车开过去。

And that's a question that you probably want to answer

而这种问题 最好在你开车上桥之前

before you start driving across that bridge.

就搞清楚答案。

And of course, there are going to be limitations to this technique,

当然，这项技术有它的局限，

just like there were with the visual microphone,

就像之前的视觉麦克风试验一样，

but we found that it works in a lot of situations

但我们也发现 它能在许多场景下发挥作用，

that you might not expect,

有时甚至出乎你的意料，

especially if you give it longer videos.

特别是当视频时间足够长的时候。

So for example, here's a video that I captured

举个例子，这段视频

of a bush outside of my apartment,

拍的是我公寓外的灌木丛，

and I didn't do anything to this bush,

我没有动过它，

but by capturing a minute-long video,

只是拍了一段1分钟长的视频，

a gentle breeze caused enough vibrations

微风不断吹动灌木，

that we could learn enough about this bush to create this simulation.

让我能够收集到足够的信息 来完成这段模拟。

(Applause)

（掌声）

And so you could imagine giving this to a film director,

想象一下， 如果电影导演掌握了这项技术，

and letting him control, say,

他就可以在后期制作时

the strength and direction of wind in a shot after it's been recorded.

随心所欲地控制风的大小和方向。

Or, in this case, we pointed our camera at a hanging curtain,

来看另一个例子， 我们拍摄了一副挂起来的窗帘，

and you can't even see any motion in this video,

在这段视频里 你甚至看不出来窗帘在动，

but by recording a two-minute-long video,

但是利用2分钟长的一段视频，

natural air currents in this room

仅仅靠房间里的自然空气流动

created enough subtle, imperceptible motions and vibrations

引发的无法察觉的动作和振动，

that we could learn enough to create this simulation.

就能使我们提取出足够多的 信息来完成这段模拟。

And ironically,

神奇的是，

we're kind of used to having this kind of interactivity

以往我们都是针对虚拟物体，

when it comes to virtual objects,

针对游戏和3D模型

when it comes to video games and 3D models,

来实现这种互动，

but to be able to capture this information from real objects in the real world

而这项技术仅仅是利用 普通的视频

using just simple, regular video,

对现实世界中的 真实物体进行采样，

is something new that has a lot of potential.

它极富新意， 具有广阔的应用前景。

So here are the amazing people who worked with me on these projects.

这些是跟我共同研究 这项技术的优秀的同事。

(Applause)

（掌声）

And what I've shown you today is only the beginning.

今天向大家展示的 只是一个技术雏形。

We've just started to scratch the surface

关于如何使用这种新型图像，

of what you can do with this kind of imaging,

我们才刚刚入门，

because it gives us a new way

它为我们提供了一种

to capture our surroundings with common, accessible technology.

运用已有的普通技术 来记录周围事物的新方法。

And so looking to the future,

展望一下未来，

it's going to be really exciting to explore

我们迫不及待地想要看到如何

what this can tell us about the world.

利用这项技术去更好地了解世界。

Thank you.

谢谢大家。

(Applause)

（掌声）