Jared Ficklin: New ways to see music (with color! and fire!)

41,711 views ・ 2012-07-13

TED


Please double-click on the English subtitles below to play the video.

00:00
Translator: Joseph Geni Reviewer: Morton Bast
0
0
7000
00:15
My passions
1
15709
1033
00:16
are music, technology and making things.
2
16742
3689
00:20
And it's the combination of these things
3
20431
3183
00:23
that has led me to the hobby of sound visualization,
4
23614
2696
00:26
and, on occasion, has led me to play with fire.
5
26310
4078
00:30
This is a Rubens' tube. It's one of many I've made over the years,
6
30388
2377
00:32
and I have one here tonight.
7
32765
1746
00:34
It's about an 8-foot-long tube of metal,
8
34511
1525
00:36
it's got a hundred or so holes on top,
9
36036
1335
00:37
on that side is the speaker, and here
10
37371
1753
00:39
is some lab tubing, and it's connected to this tank
11
39124
2176
00:41
of propane.
12
41300
1601
00:44
So, let's fire it up and see what it does.
13
44347
3369
00:53
So let's play a 550-herz frequency
14
53147
1891
00:55
and watch what happens.
15
55038
1515
00:56
(Frequency)
16
56602
8483
01:05
Thank you. (Applause)
17
65085
2713
01:07
It's okay to applaud the laws of physics,
18
67798
1988
01:09
but essentially what's happening here
19
69786
1456
01:11
-- (Laughter) --
20
71242
1760
01:13
is the energy from the sound via the air and gas molecules
21
73002
4008
01:17
is influencing the combustion properties of propane,
22
77010
2504
01:19
creating a visible waveform,
23
79514
1904
01:21
and we can see the alternating regions of compression
24
81418
2320
01:23
and rarefaction that we call frequency,
25
83738
2040
01:25
and the height is showing us amplitude.
26
85778
1744
01:27
So let's change the frequency of the sound,
27
87522
2328
01:29
and watch what happens to the fire.
28
89850
1425
01:31
(Higher frequency)
29
91275
10130
01:41
So every time we hit a resonant frequency we get a standing wave
30
101405
3350
01:44
and that emergent sine curve of fire.
31
104755
1700
01:46
So let's turn that off. We're indoors.
32
106455
1578
01:48
Thank you. (Applause)
33
108033
5591
01:53
I also have with me a flame table.
34
113624
2347
01:55
It's very similar to a Rubens' tube, and it's also used
35
115971
1565
01:57
for visualizing the physical properties of sound,
36
117536
2121
01:59
such as eigenmodes, so let's fire it up
37
119657
1968
02:01
and see what it does.
38
121625
2250
02:07
Ooh. (Laughter)
39
127552
4468
02:12
Okay. Now, while the table comes up to pressure,
40
132020
2974
02:14
let me note here that the sound is not traveling
41
134994
1672
02:16
in perfect lines. It's actually traveling in all directions,
42
136666
2746
02:19
and the Rubens' tube's a little like bisecting those waves
43
139412
3109
02:22
with a line, and the flame table's a little like
44
142521
1996
02:24
bisecting those waves with a plane,
45
144517
1854
02:26
and it can show a little more subtle complexity, which is why
46
146371
4000
02:30
I like to use it to watch Geoff Farina play guitar.
47
150371
2353
02:32
(Music)
48
152724
42281
03:15
All right, so it's a delicate dance.
49
195005
1765
03:16
If you watch closely — (Applause)
50
196770
2545
03:19
If you watch closely, you may have seen
51
199315
2767
03:22
some of the eigenmodes, but also you may have seen
52
202082
2472
03:24
that jazz music is better with fire.
53
204554
4617
03:29
Actually, a lot of things are better with fire in my world,
54
209171
2010
03:31
but the fire's just a foundation.
55
211181
2432
03:33
It shows very well that eyes can hear,
56
213613
1148
03:34
and this is interesting to me because
57
214761
1395
03:36
technology allows us to present sound to the eyes
58
216156
2854
03:39
in ways that accentuate the strength of the eyes
59
219010
2863
03:41
for seeing sound, such as the removal of time.
60
221873
2697
03:44
So here, I'm using a rendering algorithm to paint
61
224570
3384
03:47
the frequencies of the song "Smells Like Teen Spirit"
62
227954
2463
03:50
in a way that the eyes can take them in
63
230417
2040
03:52
as a single visual impression, and the technique
64
232457
2244
03:54
will also show the strengths of the visual cortex
65
234701
1973
03:56
for pattern recognition.
66
236674
1616
03:58
So if I show you another song off this album,
67
238290
1879
04:00
and another, your eyes will easily pick out
68
240169
3481
04:03
the use of repetition by the band Nirvana,
69
243650
2928
04:06
and in the frequency distribution, the colors,
70
246578
1857
04:08
you can see the clean-dirty-clean sound
71
248435
3048
04:11
that they are famous for,
72
251483
1207
04:12
and here is the entire album as a single visual impression,
73
252690
4000
04:16
and I think this impression is pretty powerful.
74
256690
1880
04:18
At least, it's powerful enough that
75
258570
1714
04:20
if I show you these four songs,
76
260284
1342
04:21
and I remind you that this is "Smells Like Teen Spirit,"
77
261626
2441
04:24
you can probably correctly guess, without listening
78
264067
2319
04:26
to any music at all, that the song
79
266386
1434
04:27
a die hard Nirvana fan would enjoy is this song,
80
267820
2294
04:30
"I'll Stick Around" by the Foo Fighters,
81
270114
2256
04:32
whose lead singer is Dave Grohl,
82
272370
2000
04:34
who was the drummer in Nirvana.
83
274370
3778
04:38
The songs are a little similar, but mostly
84
278148
1300
04:39
I'm just interested in the idea that someday maybe
85
279448
1626
04:41
we'll buy a song because we like the way it looks.
86
281074
4412
04:45
All right, now for some more sound data.
87
285486
1100
04:46
This is data from a skate park,
88
286586
2652
04:49
and this is Mabel Davis skate park
89
289238
2032
04:51
in Austin, Texas. (Skateboard sounds)
90
291270
2142
04:53
And the sounds you're hearing came from eight
91
293412
1374
04:54
microphones attached to obstacles around the park,
92
294786
2216
04:57
and it sounds like chaos, but actually
93
297002
2184
04:59
all the tricks start with a very distinct slap,
94
299186
3347
05:02
but successful tricks end with a pop,
95
302533
1604
05:04
whereas unsuccessful tricks
96
304137
1793
05:05
more of a scratch and a tumble,
97
305930
1856
05:07
and tricks on the rail will ring out like a gong, and
98
307786
4010
05:11
voices occupy very unique frequencies in the skate park.
99
311796
2790
05:14
So if we were to render these sounds visually,
100
314586
1938
05:16
we might end up with something like this.
101
316524
1407
05:17
This is all 40 minutes of the recording,
102
317931
2456
05:20
and right away the algorithm tells us
103
320387
2160
05:22
a lot more tricks are missed than are made,
104
322547
2073
05:24
and also a trick on the rails is a lot more likely
105
324620
2335
05:26
to produce a cheer, and if you look really closely,
106
326955
2872
05:29
we can tease out traffic patterns.
107
329827
1733
05:31
You see the skaters often trick in this direction. The obstacles are easier.
108
331560
6087
05:37
And in the middle of the recording, the mics pick this up,
109
337647
1705
05:39
but later in the recording, this kid shows up,
110
339352
2804
05:42
and he starts using a line at the top of the park
111
342156
2928
05:45
to do some very advanced tricks on something
112
345084
1802
05:46
called the tall rail.
113
346886
1103
05:47
And it's fascinating. At this moment in time,
114
347989
1872
05:49
all the rest of the skaters turn their lines 90 degrees
115
349861
3503
05:53
to stay out of his way.
116
353364
1778
05:55
You see, there's a subtle etiquette in the skate park,
117
355142
2543
05:57
and it's led by key influencers,
118
357685
1600
05:59
and they tend to be the kids who can do the best tricks,
119
359285
3248
06:02
or wear red pants, and on this day the mics picked that up.
120
362533
2430
06:04
All right, from skate physics to theoretical physics.
121
364963
3901
06:08
I'm a big fan of Stephen Hawking,
122
368864
1616
06:10
and I wanted to use all eight hours
123
370480
1336
06:11
of his Cambridge lecture series to create an homage.
124
371816
2587
06:14
Now, in this series he's speaking with the aid of a computer,
125
374403
3064
06:17
which actually makes identifying the ends of sentences
126
377467
3104
06:20
fairly easy. So I wrote a steering algorithm.
127
380571
3426
06:23
It listens to the lecture, and then it uses
128
383997
1960
06:25
the amplitude of each word to move a point on the x-axis,
129
385957
2687
06:28
and it uses the inflection of sentences
130
388644
2633
06:31
to move a same point up and down on the y-axis.
131
391277
2154
06:33
And these trend lines, you can see, there's more questions
132
393431
2774
06:36
than answers in the laws of physics,
133
396205
1870
06:38
and when we reach the end of a sentence,
134
398075
2033
06:40
we place a star at that location.
135
400108
2335
06:42
So there's a lot of sentences, so a lot of stars,
136
402443
2800
06:45
and after rendering all of the audio, this is what we get.
137
405243
2356
06:47
This is Stephen Hawking's universe.
138
407599
2929
06:50
(Applause)
139
410528
6937
06:57
It's all eight hours of the Cambridge lecture series
140
417465
2487
06:59
taken in as a single visual impression,
141
419952
1848
07:01
and I really like this image,
142
421800
1757
07:03
but a lot of people think it's fake.
143
423557
1808
07:05
So I made a more interactive version,
144
425365
2344
07:07
and the way I did that is I used their position in time
145
427709
5456
07:13
in the lecture to place these stars into 3D space,
146
433165
2367
07:15
and with some custom software and a Kinect,
147
435532
2401
07:17
I can walk right into the lecture.
148
437933
2663
07:20
I'm going to wave through the Kinect here
149
440596
1800
07:22
and take control, and now I'm going to reach out
150
442396
1832
07:24
and I'm going to touch a star, and when I do,
151
444228
3239
07:27
it will play the sentence
152
447467
1944
07:29
that generated that star.
153
449411
1465
07:30
Stephen Hawking: There is one, and only one, arrangement
154
450876
3816
07:34
in which the pieces make a complete picture.
155
454692
2799
07:37
Jared Ficklin: Thank you. (Applause)
156
457491
4148
07:41
There are 1,400 stars.
157
461639
3173
07:44
It's a really fun way to explore the lecture,
158
464812
1664
07:46
and, I hope, a fitting homage.
159
466476
1467
07:47
All right. Let me close with a work in progress.
160
467943
5472
07:53
I think, after 30 years, the opportunity exists
161
473415
2983
07:56
to create an enhanced version of closed captioning.
162
476398
2104
07:58
Now, we've all seen a lot of TEDTalks online,
163
478502
2119
08:00
so let's watch one now with the sound turned off
164
480621
2927
08:03
and the closed captioning turned on.
165
483548
3922
08:07
There's no closed captioning for the TED theme song,
166
487470
2146
08:09
and we're missing it, but if you've watched enough of these,
167
489616
2113
08:11
you hear it in your mind's ear,
168
491729
1369
08:13
and then applause starts.
169
493098
2983
08:16
It usually begins here, and it grows and then it falls.
170
496081
2190
08:18
Sometimes you get a little star applause,
171
498271
1977
08:20
and then I think even Bill Gates takes a nervous breath,
172
500248
2486
08:22
and the talk begins.
173
502734
1690
08:24
All right, so let's watch this clip again.
174
504424
5698
08:30
This time, I'm not going to talk at all.
175
510122
1256
08:31
There's still going to be no audio,
176
511378
1367
08:32
but what I am going to do is I'm going to render the sound
177
512745
1904
08:34
visually in real time at the bottom of the screen.
178
514649
4316
08:38
So watch closely and see what your eyes can hear.
179
518965
2791
09:03
This is fairly amazing to me.
180
543140
1880
09:05
Even on the first view, your eyes will successfully
181
545020
3333
09:08
pick out patterns, but on repeated views,
182
548353
3088
09:11
your brain actually gets better
183
551441
1689
09:13
at turning these patterns into information.
184
553130
1656
09:14
You can get the tone and the timbre
185
554786
1591
09:16
and the pace of the speech,
186
556377
1223
09:17
things that you can't get out of closed captioning.
187
557600
2064
09:19
That famous scene in horror movies
188
559664
2184
09:21
where someone is walking up from behind
189
561848
2648
09:24
is something you can see,
190
564496
1960
09:26
and I believe this information would be something
191
566456
2704
09:29
that is useful at times when the audio is turned off
192
569160
2803
09:31
or not heard at all, and I speculate that deaf audiences
193
571963
2941
09:34
might actually even be better
194
574904
1177
09:36
at seeing sound than hearing audiences.
195
576081
1781
09:37
I don't know. It's a theory right now.
196
577862
1498
09:39
Actually, it's all just an idea.
197
579360
1521
09:40
And let me end by saying that sound moves in all directions,
198
580881
4191
09:45
and so do ideas.
199
585072
1977
09:47
Thank you. (Applause)
200
587049
3112
About this website

This site will introduce you to YouTube videos that are useful for learning English. You will see English lessons taught by top-notch teachers from around the world. Double-click on the English subtitles displayed on each video page to play the video from there. The subtitles scroll in sync with the video playback. If you have any comments or requests, please contact us using this contact form.

https://forms.gle/WvT1wiN1qDtmnspy7