3 ways to spot a bad statistic | Mona Chalabi

250,041 views ใƒป 2017-04-17

TED


ืื ื ืœื—ืฅ ืคืขืžื™ื™ื ืขืœ ื”ื›ืชื•ื‘ื™ื•ืช ื‘ืื ื’ืœื™ืช ืœืžื˜ื” ื›ื“ื™ ืœื”ืคืขื™ืœ ืืช ื”ืกืจื˜ื•ืŸ.

ืžืชืจื’ื: Ido Dekkers ืžื‘ืงืจ: hila scherba
00:12
I'm going to be talking about statistics today.
0
12704
2763
ืื ื™ ืขื•ืžื“ืช ืœื“ื‘ืจ ืขืœ ืกื˜ื˜ื™ืกื˜ื™ืงื” ื”ื™ื•ื.
00:15
If that makes you immediately feel a little bit wary, that's OK,
1
15491
3138
ืื ื–ื” ื’ื•ืจื ืœื›ื ืœื”ืจื’ื™ืฉ ืžืขื˜ ืžื•ื“ืื’ื™ื ืžื™ื™ื“ื™ืช, ื–ื” ื‘ืกื“ืจ,
00:18
that doesn't make you some kind of crazy conspiracy theorist,
2
18653
2859
ื–ื” ืœื ื”ื•ืคืš ืืชื›ื ืœืžืขื™ืŸ ืžืฉื•ื’ืขื™ ืชื™ืื•ืจื™ื•ืช ืงื•ืกืคื™ืจืฆื™ื”,
00:21
it makes you skeptical.
3
21536
1296
ื–ื” ืขื•ืฉื” ืืชื›ื ืกืงืคื˜ื™ื™ื.
00:22
And when it comes to numbers, especially now, you should be skeptical.
4
22856
3886
ื•ื›ืฉื–ื” ืžื’ื™ืข ืœืžืกืคืจื™ื, ื‘ืขื™ืงืจ ืขื›ืฉื™ื•, ืืชื ืฆืจื™ื›ื™ื ืœื”ื™ื•ืช ืกืงืคื˜ื™ื™ื.
00:26
But you should also be able to tell which numbers are reliable
5
26766
3011
ืื‘ืœ ืืชื ื’ื ืฆืจื™ื›ื™ื ืœื”ื™ื•ืช ืžืกื•ื’ืœื™ื ืœื“ืขืช ืื™ื–ื” ืžืกืคืจื™ื ืืžื™ื ื™ื
00:29
and which ones aren't.
6
29801
1160
ื•ืืœื• ืœื.
00:30
So today I want to try to give you some tools to be able to do that.
7
30985
3206
ืื– ื”ื™ื•ื ืื ื™ ืจื•ืฆื” ืœื ืกื•ืช ืœืชืช ืœื›ื ื›ืžื” ื›ืœื™ื ื›ื“ื™ ืฉืชื”ื™ื• ืžืกื•ื’ืœื™ื ืœืขืฉื•ืช ืืช ื–ื”.
00:34
But before I do,
8
34215
1169
ืื‘ืœ ืœืคื ื™ ืฉืืขืฉื” ื–ืืช,
00:35
I just want to clarify which numbers I'm talking about here.
9
35408
2839
ืื ื™ ืจืง ืจื•ืฆื” ืœื”ื‘ื”ื™ืจ ืขืœ ืื™ื–ื” ืžืกืคืจื™ื ืื ื™ ืžื“ื‘ืจืช.
ืื ื™ ืœื ืžื“ื‘ืจืช ืขืœ ื˜ืขื ื•ืช ื›ืžื•,
00:38
I'm not talking about claims like,
10
38271
1635
00:39
"9 out of 10 women recommend this anti-aging cream."
11
39930
2449
"9 ืžืชื•ืš 10 ื ืฉื™ื ืžืžืœื™ืฆื•ืช ืขืœ ืงืจื ื ื’ื“ ืงืžื˜ื™ื ื”ื–ื”."
00:42
I think a lot of us always roll our eyes at numbers like that.
12
42403
2972
ืื ื™ ื—ื•ืฉื‘ืช ืฉื”ืจื‘ื” ืžืื™ืชื ื• ืชืžื™ื“ ืžื’ืœื’ืœื™ื ืืช ืขื™ื ื™ื ื• ืขืœ ืžืกืคืจื™ื ื›ืืœื”.
00:45
What's different now is people are questioning statistics like,
13
45399
2984
ืžื” ืฉืฉื•ื ื” ืขื›ืฉื™ื• ื–ื” ืฉืื ืฉื™ื ืœื ืžืืžื™ื ื™ื ืœืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ื›ืžื•,
"ืจืžืช ื”ืื‘ื˜ืœื” ื‘ืืจืฆื•ืช ื”ื‘ืจื™ืช ื”ื™ื ื—ืžื™ืฉื” ืื—ื•ื–ื™ื."
00:48
"The US unemployment rate is five percent."
14
48407
2014
00:50
What makes this claim different is it doesn't come from a private company,
15
50445
3516
ืžื” ืฉืขื•ืฉื” ืืช ื”ื˜ืขื ื” ื”ื–ื• ืฉื•ื ื” ื–ื” ืฉื”ื™ื ืœื ืžื’ื™ืขื” ืžื—ื‘ืจื” ืคืจื˜ื™ืช,
00:53
it comes from the government.
16
53985
1388
ื”ื™ื ืžื’ื™ืขื” ืžื”ืžืžืฉืœื”.
ื‘ืขืจืš 4 ืžืชื•ืš 10 ืืžืจื™ืงืื™ื ืœื ืžืืžื™ื ื™ื ืœืžื™ื“ืข ื›ืœื›ืœื™
00:55
About 4 out of 10 Americans distrust the economic data
17
55397
3336
00:58
that gets reported by government.
18
58757
1573
ืฉืžื“ื•ื•ื— ืขืœ ื™ื“ื™ ื”ืžืžืฉืœ.
01:00
Among supporters of President Trump it's even higher;
19
60354
2491
ืืฆืœ ืชื•ืžื›ื™ื ืฉืœ ื”ื ืฉื™ื ื˜ืจืืžืค ื–ื” ืืคื™ืœื• ื’ื‘ื•ื” ื™ื•ืชืจ;
01:02
it's about 7 out of 10.
20
62869
1633
ื–ื” ื‘ืขืจืš 7 ืžืชื•ืš 10.
01:04
I don't need to tell anyone here
21
64526
1804
ืื ื™ ืœื ืฆืจื™ื›ื” ืœืกืคืจ ืœื›ื ืคื”
01:06
that there are a lot of dividing lines in our society right now,
22
66354
3011
ืฉื™ืฉ ื”ืจื‘ื” ื™ื•ืชืจ ืงื•ื•ื™ื ืžืคืจื™ื“ื™ื ื‘ื—ื‘ืจื” ืฉืœื ื• ืขื›ืฉื™ื•,
01:09
and a lot of them start to make sense,
23
69389
1825
ื•ื”ืจื‘ื” ืžื”ื ืžืชื—ื™ืœื™ื ืœื”ื™ื•ืช ื”ื’ื™ื•ื ื™ื™ื,
ื‘ืจื’ืข ืฉืืชื ืžื‘ื™ื ื™ื ืืช ื”ื™ื—ืกื™ื ืฉืœ ื”ืื ืฉื™ื ืขื ืžืกืคืจื™ื ืžืžืฉืœืชื™ื™ื.
01:11
once you understand people's relationships with these government numbers.
24
71238
3687
01:14
On the one hand, there are those who say these statistics are crucial,
25
74949
3336
ืžืฆื“ ืื—ื“, ื™ืฉ ืืช ืืœื” ืฉืื•ืžืจื™ื ืฉืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ื”ืŸ ื—ื™ื•ื ื™ื•ืช,
01:18
that we need them to make sense of society as a whole
26
78309
2630
ืฉืื ื—ื ื• ืฆืจื™ื›ื™ื ืื•ืชืŸ ื›ื“ื™ ืœื”ื‘ื™ืŸ ืืช ื”ื—ื‘ืจื” ืฉืœื ื• ื›ื›ืœืœ
01:20
in order to move beyond emotional anecdotes
27
80963
2164
ื›ื“ื™ ืœืขื‘ื•ืจ ืžืขื‘ืจ ืœืื ืงื“ื•ื˜ื•ืช ืจื’ืฉื™ื•ืช
01:23
and measure progress in an [objective] way.
28
83151
2410
ื•ืœืžื“ื•ื“ ื”ืชืงื“ืžื•ืช ื‘ื“ืจืš ืกื•ื‘ื™ื™ืงื˜ื™ื‘ื™ืช.
01:25
And then there are the others,
29
85585
1467
ื•ืื– ื™ืฉ ืืช ื”ืื—ืจื™ื,
01:27
who say that these statistics are elitist,
30
87076
2156
ืฉืื•ืžืจื™ื ืฉื”ืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ื”ืŸ ืืœื™ื˜ื™ืกื˜ื™ื•ืช,
01:29
maybe even rigged;
31
89256
1208
ืื•ืœื™ ืืคื™ืœื• ืžื•ื˜ื•ืช;
01:30
they don't make sense and they don't really reflect
32
90488
2394
ื”ืŸ ืœื ื”ื’ื™ื•ื ื™ื•ืช ื•ื”ืŸ ืœื ื‘ืืžืช ืžืฉืงืคื•ืช ืžื” ืฉืงื•ืจื” ื‘ื—ื™ื™ ื”ื™ื•ืžื™ื•ื ืฉืœ ืื ืฉื™ื.
01:32
what's happening in people's everyday lives.
33
92906
2296
01:35
It kind of feels like that second group is winning the argument right now.
34
95226
3487
ื–ื” ืงืฆืช ืžืจื’ื™ืฉ ื›ืื™ืœื• ื”ืงื‘ื•ืฆื” ื”ืฉื ื™ื” ื–ื•ื›ื” ื‘ื˜ื™ืขื•ื ื™ื ืขื›ืฉื™ื•.
01:38
We're living in a world of alternative facts,
35
98737
2108
ืื ื—ื ื• ื—ื™ื™ื ื‘ืขื•ืœื ืฉืœ ืขื•ื‘ื“ื•ืช ืืœื˜ืจื ื˜ื™ื‘ื™ื•ืช,
01:40
where people don't find statistics this kind of common ground,
36
100869
2935
ืฉื‘ื• ืื ืฉื™ื ืœื ืจื•ืื™ื ื‘ืกื˜ื˜ื™ืกื˜ื™ืงื” ื›ืกื•ื’ ืฉืœ ืžื›ื ื” ืžืฉื•ืชืฃ,
01:43
this starting point for debate.
37
103828
1636
ื›ื ืงื•ื“ืช ื”ื”ืชื—ืœื” ื”ื–ื• ืœื•ื™ื›ื•ื—.
01:45
This is a problem.
38
105488
1286
ื–ื• ื‘ืขื™ื”.
01:46
There are actually moves in the US right now
39
106798
2067
ื™ืฉ ืœืžืขืฉื” ืชื ื•ืขื•ืช ื‘ืืจืฆื•ืช ื”ื‘ืจื™ืช ืขื›ืฉื™ื•
01:48
to get rid of some government statistics altogether.
40
108889
2861
ื›ื“ื™ ืœื”ื™ืคื˜ืจ ืœื’ืžืจื™ ืžื›ืžื” ืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ืžืžืฉืœืชื™ื•ืช.
01:51
Right now there's a bill in congress about measuring racial inequality.
41
111774
3387
ืžืžืฉ ืขื›ืฉื™ื• ื™ืฉ ื”ืฆืขืช ื—ื•ืง ื‘ืงื•ื ื’ืจืก ื‘ื ื•ื’ืข ืœืžื“ื™ื“ืช ื—ื•ืกืจ ืฉื•ื•ื™ื•ืŸ ื’ื–ืขื ื™.
01:55
The draft law says that government money should not be used
42
115185
2801
ื”ืฆืขืช ื”ื—ื•ืง ืื•ืžืจืช ืฉื›ืกืฃ ืžืžืฉืœืชื™ ืœื ืฆืจื™ืš ืœื”ื™ื•ืช ื‘ืฉื™ืžื•ืฉ
01:58
to collect data on racial segregation.
43
118010
1902
ื‘ืฉื‘ื™ืœ ืœืืกื•ืฃ ืžื™ื“ืข ืขืœ ื”ืคืจื“ื” ื’ื–ืขื™ืช.
01:59
This is a total disaster.
44
119936
1885
ื–ื” ืืกื•ืŸ ืžื•ื—ืœื˜.
02:01
If we don't have this data,
45
121845
1748
ืื ืื™ืŸ ืœื ื• ืžื™ื“ืข,
02:03
how can we observe discrimination,
46
123617
1778
ืื™ืš ืื ื—ื ื• ื™ื›ื•ืœื™ื ืœืจืื•ืช ืืคืœื™ื”,
02:05
let alone fix it?
47
125419
1278
ืฉืœื ืœื“ื‘ืจ ืขืœ ืœืชืงืŸ ืื•ืชื”?
02:06
In other words:
48
126721
1188
ื‘ืžื™ืœื™ื ืื—ืจื•ืช:
02:07
How can a government create fair policies
49
127933
2059
ืื™ืš ืžืžืฉืœ ื™ื›ื•ืœ ืœื™ื™ืฆืจ ืžื“ื™ื ื™ื•ืช ื”ื•ื’ื ืช
02:10
if they can't measure current levels of unfairness?
50
130016
2771
ืื ื”ื ืœื ื™ื›ื•ืœื™ื ืœืžื“ื•ื“ ืจืžื•ืช ื ื•ื›ื—ื™ื•ืช ืฉืœ ื—ื•ืกืจ ื”ื•ื’ื ื•ืช?
02:12
This isn't just about discrimination,
51
132811
1794
ื–ื” ืœื ืจืง ื ื•ื’ืข ืœืืคืœื™ื”,
02:14
it's everything -- think about it.
52
134629
1670
ื–ื” ื”ื›ืœ -- ื—ืฉื‘ื• ืขืœ ื–ื”.
ืื™ืš ืื ื—ื ื• ื™ื›ื•ืœื™ื ืœื—ื•ืงืง ื‘ื ื•ื’ืข ืœื‘ืจื™ืื•ืช
02:16
How can we legislate on health care
53
136323
1690
ืื ืื™ืŸ ืœื ื• ืžื™ื“ืข ื˜ื•ื‘ ืขืœ ื‘ืจื™ืื•ืช ื•ืขื•ื ื™?
02:18
if we don't have good data on health or poverty?
54
138037
2271
ืื™ืš ื™ื›ื•ืœ ืœื”ื™ื•ืช ืœื ื• ื“ื™ื•ืŸ ืฆื™ื‘ื•ืจื™ ืขืœ ื”ื’ื™ืจื”
02:20
How can we have public debate about immigration
55
140332
2198
ืื ืื ื—ื ื• ืœื ื™ื›ื•ืœื™ื ืœืคื—ื•ืช ืœื”ืกื›ื™ื
02:22
if we can't at least agree
56
142554
1250
02:23
on how many people are entering and leaving the country?
57
143828
2643
ืขืœ ื›ืžื” ืื ืฉื™ื ื ื›ื ืกื™ื ื•ื™ื•ืฆืื™ื ืžื”ืžื“ื™ื ื”?
02:26
Statistics come from the state; that's where they got their name.
58
146495
3058
ืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ืžื’ื™ืขื•ืช ืžื”ืžื“ื™ื ื”; ืžืฉื ื”ืŸ ืงื™ื‘ืœื• ืืช ืฉืžืŸ.
ื”ื ืงื•ื“ื” ื”ื™ืชื” ืœืžื“ื•ื“ ื˜ื•ื‘ ื™ื•ืชืจ ืืช ื”ืื•ื›ืœื•ืกื™ื”
02:29
The point was to better measure the population
59
149577
2157
ื›ื“ื™ ืœืฉืจืช ืื•ืชื” ื˜ื•ื‘ ื™ื•ืชืจ.
02:31
in order to better serve it.
60
151758
1357
ืื– ืื ื—ื ื• ืฆืจื™ื›ื™ื ืืช ื”ืžืกืคืจื™ื ื”ืžืžืฉืœืชื™ื™ื ื”ืืœื”,
02:33
So we need these government numbers,
61
153139
1725
02:34
but we also have to move beyond either blindly accepting
62
154888
2647
ืื‘ืœ ืื ื—ื ื• ื’ื ืฆืจื™ื›ื™ื ืœื ื•ืข ืžืขื‘ืจ ืœืงื‘ืœื” ืขื™ื•ื•ืจืช ืื• ื“ื—ื™ื™ื” ืขื™ื•ื•ืจืช ืฉืœื”ื.
02:37
or blindly rejecting them.
63
157559
1268
02:38
We need to learn the skills to be able to spot bad statistics.
64
158851
2997
ืื ื—ื ื• ืฆืจื™ื›ื™ื ืœืœืžื•ื“ ืืช ื”ื›ื™ืฉื•ืจื™ื ืœื”ื™ื•ืช ืžืกื•ื’ืœื™ื ืœื–ื”ื•ืช ืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ื’ืจื•ืขื•ืช.
02:41
I started to learn some of these
65
161872
1528
ื”ืชื—ืœืชื™ ืœืœืžื•ื“ ื›ืžื” ืžืืœื”
02:43
when I was working in a statistical department
66
163424
2166
ื›ืฉืขื‘ื“ืชื™ ื‘ืžื—ืœืงืช ื”ืกื˜ื˜ื™ืกื˜ื™ืงื”
ืฉื”ื™ื ื—ืœืง ืžื”ืื•ืžื•ืช ื”ืžืื•ื—ื“ื•ืช.
02:45
that's part of the United Nations.
67
165614
1643
02:47
Our job was to find out how many Iraqis had been forced from their homes
68
167281
3406
ื”ืขื‘ื•ื“ื” ืฉืœื ื• ื”ื™ืชื” ืœื’ืœื•ืช ื›ืžื” ืขื™ืจืืงื™ื ื”ื•ืฆืื• ื‘ื›ื•ื— ืžื‘ืชื™ื”ื
02:50
as a result of the war,
69
170711
1158
ื›ืชื•ืฆืื” ืžื”ืžืœื—ืžื”,
02:51
and what they needed.
70
171893
1158
ื•ืžื” ื”ื ืฆืจื™ื›ื™ื.
ื–ื• ื”ื™ืชื” ืขื‘ื•ื“ื” ื‘ืืžืช ื—ืฉื•ื‘ื”, ืื‘ืœ ื”ื™ื ื’ื ื”ื™ืชื” ืžืžืฉ ืงืฉื”.
02:53
It was really important work, but it was also incredibly difficult.
71
173075
3178
02:56
Every single day, we were making decisions
72
176277
2018
ื›ืœ ื™ื•ื, ืขืฉื™ื ื• ื”ื—ืœื˜ื•ืช
02:58
that affected the accuracy of our numbers --
73
178319
2157
ืฉื”ืฉืคื™ืขื• ืขืœ ื”ื“ื™ื•ืง ืฉืœ ื”ืžืกืคืจื™ื ืฉืœื ื• --
03:00
decisions like which parts of the country we should go to,
74
180500
2744
ื”ื—ืœื˜ื•ืช ื›ืžื• ืœืื™ื–ื” ื—ืœืงื™ื ืฉืœ ื”ืžื“ื™ื ื” ืื ื—ื ื• ืฆืจื™ื›ื™ื ืœืœื›ืช,
03:03
who we should speak to,
75
183268
1156
ืขื ืžื™ ืฆืจื™ืš ืœื“ื‘ืจ,
03:04
which questions we should ask.
76
184448
1568
ืื™ื–ื” ืฉืืœื•ืช ืื ื—ื ื• ืฆืจื™ื›ื™ื ืœืฉืื•ืœ.
03:06
And I started to feel really disillusioned with our work,
77
186040
2680
ื•ื”ืชื—ืœืชื™ ืœื”ืจื’ื™ืฉ ื‘ืืžืช ื‘ืืฉืœื™ื” ืžื”ืขื‘ื•ื“ื” ืฉืœื ื•,
03:08
because we thought we were doing a really good job,
78
188744
2518
ื‘ื’ืœืœ ืฉื—ืฉื‘ื ื• ืฉืขืฉื™ื ื• ืขื‘ื•ื“ื” ืžืžืฉ ื˜ื•ื‘ื”,
03:11
but the one group of people who could really tell us were the Iraqis,
79
191286
3278
ืื‘ืœ ื”ืงื‘ื•ืฆื” ื”ื™ื—ื™ื“ื” ืฉืœ ื”ืื ืฉื™ื ืฉื‘ืืžืช ื™ื›ืœื” ืœืกืคืจ ืœื ื• ื”ื™ืชื” ื”ืขื™ืจืืงื™ื,
03:14
and they rarely got the chance to find our analysis, let alone question it.
80
194588
3540
ื•ืจืง ืœืขื™ืชื™ื ืจื—ื•ืงื•ืช ื”ื ืงื™ื‘ืœื• ืืช ื”ื”ื–ื“ืžื ื•ืช ืœื’ืœื•ืช ืืช ื”ืื ืœื™ื–ื” ืฉืœื ื•, ืฉืœื ืœื“ื‘ืจ ืขืœ ืœืคืงืคืง.
03:18
So I started to feel really determined
81
198152
1831
ืื– ื”ืชื—ืœืชื™ ืœื”ืจื’ื™ืฉ ืžืžืฉ ื ื—ื•ืฉื”
03:20
that the one way to make numbers more accurate
82
200007
2311
ืฉื”ื“ืจืš ื”ื™ื—ื™ื“ื” ืœืขืฉื•ืช ืืช ื”ืžืกืคืจื™ื ื™ื•ืชืจ ืžื“ื•ื™ื™ืงื™ื
03:22
is to have as many people as possible be able to question them.
83
202342
3053
ื”ื™ื ืฉื›ืžื” ืฉื™ื•ืชืจ ืื ืฉื™ื ื™ื”ื™ื• ืžืกื•ื’ืœื™ื ืœืคืงืคืง ื‘ื”ื.
03:25
So I became a data journalist.
84
205419
1434
ืื– ื”ืคื›ืชื™ ืœืขื™ืชื•ื ืื™ืช ืžื™ื“ืข.
03:26
My job is finding these data sets and sharing them with the public.
85
206877
3904
ื”ืขื‘ื•ื“ื” ืฉืœื™ ื”ื™ื ืœื’ืœื•ืช ืืช ืžืขืจื›ื™ ื”ืžื™ื“ืข ื”ืืœื” ื•ืœื—ืœื•ืง ืื•ืชื ืขื ื”ืฆื™ื‘ื•ืจ.
03:30
Anyone can do this, you don't have to be a geek or a nerd.
86
210805
3173
ื›ื•ืœื ื™ื›ื•ืœื™ื ืœืขืฉื•ืช ืืช ื–ื”, ืืชื ืœื ื—ื™ื™ื‘ื™ื ืœื”ื™ื•ืช ื’ื™ืงื™ื ืื• ื—ื ื•ื ื™ื.
ืืชื ื™ื›ื•ืœื™ื ืœื”ืชืขืœื ืžื”ืžื™ืœื™ื ื”ืืœื•; ื”ื ื‘ืฉื™ืžื•ืฉ ืขืœ ื™ื“ื™ ืื ืฉื™ื
03:34
You can ignore those words; they're used by people
87
214002
2355
03:36
trying to say they're smart while pretending they're humble.
88
216381
2822
ืฉืžื ืกื™ื ืœื”ื’ื™ื“ ืฉื”ื ื—ื›ืžื™ื ื‘ืขื•ื“ื ืžืขืžื™ื“ื™ื ืคื ื™ื ืฉื”ื ืฆื ื•ืขื™ื.
ืžืžืฉ ื›ืœ ืื—ื“ ื™ื›ื•ืœ ืœืขืฉื•ืช ืืช ื–ื”.
03:39
Absolutely anyone can do this.
89
219227
1589
03:40
I want to give you guys three questions
90
220840
2067
ืื ื™ ืจื•ืฆื” ืœืฉืื•ืœ ืืชื›ื ืฉืœื•ืฉ ืฉืืœื•ืช
03:42
that will help you be able to spot some bad statistics.
91
222931
3005
ืฉื™ืขื–ืจื• ืœื›ื ืœื”ื™ื•ืช ืžืกื•ื’ืœื™ื ืœื–ื”ื•ืช ื›ืžื” ืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ื’ืจื•ืขื•ืช.
03:45
So, question number one is: Can you see uncertainty?
92
225960
3507
ืื–, ืฉืืœื” ืžืกืคืจ ืื—ืช ื”ื™ื ื”ืื ืืชื ื™ื›ื•ืœื™ื ืœืจืื•ืช ื—ื•ืกืจ ื•ื•ื“ืื•ืช?
03:49
One of things that's really changed people's relationship with numbers,
93
229491
3364
ืื—ื“ ื”ื“ื‘ืจื™ื ืฉื‘ืืžืช ืฉื™ื ื• ืืช ื”ื™ื—ืกื™ื ืฉืœ ืื ืฉื™ื ืขื ืžืกืคืจื™ื,
03:52
and even their trust in the media,
94
232879
1641
ื•ืืคื™ืœื• ืืช ื”ืืžื•ืŸ ืฉืœื”ื ื‘ืชืงืฉื•ืจืช,
03:54
has been the use of political polls.
95
234544
2258
ื”ื™ื• ื”ืฉื™ืžื•ืฉ ื‘ืกืงืจื™ื ืคื•ืœื™ื˜ื™ื™ื.
03:56
I personally have a lot of issues with political polls
96
236826
2538
ืœื™ ืื™ืฉื™ืช ื™ืฉ ื”ืจื‘ื” ื‘ืขื™ื•ืช ืขื ืกืงืจื™ื ืคื•ืœื™ื˜ื™ื™ื
03:59
because I think the role of journalists is actually to report the facts
97
239388
3376
ื‘ื’ืœืœ ืฉืื ื™ ื—ื•ืฉื‘ืช ืฉื”ืชืคืงื™ื“ ืฉืœ ืขื™ืชื•ื ืื™ื ื”ื•ื ืœืžืขืฉื” ืœื“ื•ื•ื— ืขืœ ื”ืขื•ื‘ื“ื•ืช
04:02
and not attempt to predict them,
98
242788
1553
ื•ืœื ืœื ืกื•ืช ืœื—ื–ื•ืช ืื•ืชืŸ,
04:04
especially when those predictions can actually damage democracy
99
244365
2996
ื‘ืขื™ืงืจ ื›ืฉื”ืชื—ื–ื™ื•ืช ื”ืืœื• ื™ื›ื•ืœื•ืช ืœืžืขืฉื” ืœืคื’ื•ืข ื‘ื“ืžื•ืงืจื˜ื™ื” ืขืœ ื™ื“ื™ ืกื™ืžื•ืŸ ืื ืฉื™ื:
04:07
by signaling to people: don't bother to vote for that guy,
100
247385
2732
ืืœ ืชื˜ืจื—ื• ืœื”ืฆื‘ื™ืข ืœืื™ืฉ ื”ื–ื”, ืื™ืŸ ืœื• ืกื™ื›ื•ื™.
04:10
he doesn't have a chance.
101
250141
1205
ื”ื‘ื” ื ื ื™ื— ืืช ื–ื” ื‘ืฆื“ ื‘ื™ื ืชื™ื™ื ื•ื ื“ื‘ืจ ืขืœ ื”ื“ื™ื•ืง ืฉืœ ื”ืžืืžืฅ ื”ื–ื”.
04:11
Let's set that aside for now and talk about the accuracy of this endeavor.
102
251370
3654
ื‘ื”ืชื‘ืกืก ืขืœ ื‘ื—ื™ืจื•ืช ืืจืฆื™ื•ืช ื‘ืื ื’ืœื™ื”, ืื™ื˜ืœื™ื”, ื™ืฉืจืืœ
04:15
Based on national elections in the UK, Italy, Israel
103
255048
4608
04:19
and of course, the most recent US presidential election,
104
259680
2764
ื•ื›ืžื•ื‘ืŸ, ื”ื‘ื—ื™ืจื•ืช ื”ืื—ืจื•ื ื•ืช ื‘ืืจืฆื•ืช ื”ื‘ืจื™ืช,
04:22
using polls to predict electoral outcomes
105
262468
2137
ืฉื™ืžื•ืฉ ื‘ืกืงืจื™ื ื›ื“ื™ ืœื—ื–ื•ืช ืืช ืชื•ืฆืื•ืช ื”ื‘ื—ื™ืจื•ืช
04:24
is about as accurate as using the moon to predict hospital admissions.
106
264629
3812
ืžื“ื•ื™ืง ื‘ืขืจืš ื›ืžื• ืฉื™ืžื•ืฉ ื‘ื™ืจื— ื›ื“ื™ ืœื—ื–ื•ืช ืืฉืคื•ื– ื‘ื‘ืชื™ ื—ื•ืœื™ื.
04:28
No, seriously, I used actual data from an academic study to draw this.
107
268465
4200
ืœื, ื‘ืืžืช, ื”ืฉืชืžืฉืชื™ ื‘ืžื™ื“ืข ืืžื™ืชื™ ืžืžื—ืงืจ ืืงื“ืžื™ ื›ื“ื™ ืœืฆื™ื™ืจ ืืช ื–ื”.
04:32
There are a lot of reasons why polling has become so inaccurate.
108
272689
3727
ื™ืฉ ื”ืจื‘ื” ืกื™ื‘ื•ืช ืœืžื” ืกืงืจื™ื ื”ืคื›ื• ืœืœื ืžื“ื•ื™ื™ืงื™ื.
04:36
Our societies have become really diverse,
109
276440
1970
ื”ื—ื‘ืจื•ืช ืฉืœื ื• ื”ืคื›ื• ืœื‘ืืžืช ืžื’ื•ื•ื ื•ืช,
04:38
which makes it difficult for pollsters to get a really nice representative sample
110
278434
3821
ืžื” ืฉื’ื•ืจื ืœืงื•ืฉื™ ืœืกื•ืงืจื™ื ืœืงื‘ืœ ื“ื•ื’ืžื” ื‘ืืžืช ืžื™ื™ืฆื’ืช
04:42
of the population for their polls.
111
282279
1627
ืฉืœ ื”ืื•ื›ืœื•ืกื™ื” ืขื‘ื•ืจ ื”ืกืงืจื™ื ืฉืœื”ื.
04:43
People are really reluctant to answer their phones to pollsters,
112
283930
3006
ืื ืฉื™ื ื‘ืืžืช ื ืžื ืขื™ื ืžืœืขื ื•ืช ื‘ื˜ืœืคื•ืŸ ืœืกื•ืงืจื™ื,
04:46
and also, shockingly enough, people might lie.
113
286960
2276
ื•ื’ื, ืœืžืจื‘ื” ื”ื”ืคืชืขื”, ืื ืฉื™ื ื™ื›ื•ืœื™ื ืœืฉืงืจ.
04:49
But you wouldn't necessarily know that to look at the media.
114
289260
2811
ืื‘ืœ ืœื ื‘ื”ื›ืจื— ืชื“ืขื• ื–ืืช ืื ืชื‘ื™ื˜ื• ื‘ืชืงืฉื•ืจืช.
04:52
For one thing, the probability of a Hillary Clinton win
115
292095
2761
ืจืืฉื™ืช, ื”ื”ืกืชื‘ืจื•ืช ืฉื”ื™ืœืจื™ ืงืœื™ื ื˜ื•ืŸ ืชื ืฆื—
04:54
was communicated with decimal places.
116
294880
2791
ืชื•ืงืฉืจื” ืขื ื ืงื•ื“ื•ืช ืขืฉืจื•ื ื™ื•ืช.
04:57
We don't use decimal places to describe the temperature.
117
297695
2621
ืื ื—ื ื• ืœื ืžืฉืชืžืฉื™ื ื‘ื ืงื•ื“ื•ืช ืขืฉืจื•ื ื™ื•ืช ื›ื“ื™ ืœืชืืจ ืืช ื”ื˜ืžืคืจื˜ื•ืจื”.
05:00
How on earth can predicting the behavior of 230 million voters in this country
118
300340
4228
ืื™ืš ืœืขื–ืื–ืœ ืชื—ื–ื™ืช ื”ื”ืชื ื”ื’ื•ืช ืฉืœ 230 ืžืœื™ื•ืŸ ื‘ื•ื—ืจื™ื ื‘ืžื“ื™ื ื”
05:04
be that precise?
119
304592
1829
ื™ื›ื•ืœื” ืœื”ื™ื•ืช ื›ืœ ื›ืš ืžื“ื•ื™ื™ืงืช?
05:06
And then there were those sleek charts.
120
306445
2002
ื•ืื– ื™ืฉ ืืช ื”ื’ืจืคื™ื ื”ื™ืคื™ื ื”ืืœื”.
05:08
See, a lot of data visualizations will overstate certainty, and it works --
121
308471
3973
ื”ื‘ื™ื ื•, ื”ืจื‘ื” ืžืฆื•ืจื•ืช ื”ืฆื’ืช ืžื™ื“ืข ืžืคืจื™ื–ื•ืช ื‘-ื•ื•ื“ืื•ืช, ื•ื–ื” ืขื•ื‘ื“ --
05:12
these charts can numb our brains to criticism.
122
312468
2620
ื”ื˜ื‘ืœืื•ืช ื”ืืœื• ื™ื›ื•ืœื•ืช ืœื”ืงื”ื•ืช ืืช ื”ืžื•ื—ื•ืช ืฉืœื ื• ืœื‘ื™ืงื•ืจืช.
05:15
When you hear a statistic, you might feel skeptical.
123
315112
2558
ื›ืฉืืชื ืฉื•ืžืขื™ื ืกื˜ื˜ื™ืกื˜ื™ืงื”, ืืชื ืื•ืœื™ ืžืจื’ื™ืฉื™ื ืกืงืคื˜ื™ื™ื.
05:17
As soon as it's buried in a chart,
124
317694
1635
ื‘ืจื’ืข ืฉื–ื” ืงื‘ื•ืจ ื‘ื’ืจืฃ,
ื–ื” ืžืจื’ื™ืฉ ื›ืžื• ืกื•ื’ ืฉืœ ืžื“ืข ืื•ื‘ื™ื™ืงื˜ื™ื‘ื™,
05:19
it feels like some kind of objective science,
125
319353
2129
05:21
and it's not.
126
321506
1249
ื•ื–ื” ืœื.
05:22
So I was trying to find ways to better communicate this to people,
127
322779
3103
ืื– ื ื™ืกื™ืชื™ ืœื’ืœื•ืช ื“ืจื›ื™ื ืœืชืงืฉืจ ืืช ื–ื” ืœืื ืฉื™ื ื˜ื•ื‘ ื™ื•ืชืจ,
05:25
to show people the uncertainty in our numbers.
128
325906
2504
ื›ื“ื™ ืœื”ืจืื•ืช ืœืื ืฉื™ื ืืช ื—ื•ืกืจ ื”ื•ื•ื“ืื•ืช ื‘ืžืกืคืจื™ื ืฉืœื ื•.
05:28
What I did was I started taking real data sets,
129
328434
2246
ืžื” ืฉืขืฉื™ืชื™ ื”ื™ื” ืœื”ืชื—ื™ืœ ืœืงื—ืช ืžืื’ืจื™ ื ืชื•ื ื™ื ืืžื™ืชื™ื™ื,
05:30
and turning them into hand-drawn visualizations,
130
330704
2652
ื•ืœื”ืคื•ืš ืื•ืชื ืœืชืฆื•ื’ื•ืช ืžื™ื“ืข ื”ืžืฆื•ื™ื™ืจื•ืช ื™ื“ื ื™ืช,
05:33
so that people can see how imprecise the data is;
131
333380
2672
ื›ืš ืฉืื ืฉื™ื ื™ื•ื›ืœื• ืœืจืื•ืช ื›ืžื” ืœื ืžื“ื•ื™ื™ืง ื”ืžื™ื“ืข;
ื›ืš ืฉืื ืฉื™ื ื™ื›ื•ืœื™ื ืœืจืื•ืช ืฉืื“ื ืขืฉื” ืืช ื–ื”,
05:36
so people can see that a human did this,
132
336076
1996
ืื“ื ืžืฆื ืืช ื”ืžื™ื“ืข ื•ื”ืจืื” ืื•ืชื•.
05:38
a human found the data and visualized it.
133
338096
1972
05:40
For example, instead of finding out the probability
134
340092
2672
ืœื“ื•ื’ืžื”, ื‘ืžืงื•ื ืœื’ืœื•ืช ืืช ื”ื”ืกืชื‘ืจื•ืช
05:42
of getting the flu in any given month,
135
342788
2126
ืฉืœ ืœืงื‘ืœ ืฉืคืขืช ื‘ื›ืœ ื—ื•ื“ืฉ ืžืกื•ื™ื™ื,
05:44
you can see the rough distribution of flu season.
136
344938
2792
ืืชื ื™ื›ื•ืœื™ื ืœืจืื•ืช ืืช ื”ื”ืชืคืœื’ื•ืช ื”ื’ืกื” ืฉืœ ืขื•ื ื•ืช ื”ืฉืคืขืช.
05:47
This is --
137
347754
1167
ื–ื” --
05:48
(Laughter)
138
348945
1018
(ืฆื—ื•ืง)
05:49
a bad shot to show in February.
139
349987
1486
ืชืžื•ื ื” ื’ืจื•ืขื” ืœื”ืจืื•ืช ื‘ืคื‘ืจื•ืืจ.
05:51
But it's also more responsible data visualization,
140
351497
2455
ืื‘ืœ ื–ื” ื’ื ื”ืฆื’ืช ืžื™ื“ืข ื™ื•ืชืจ ืื—ืจืื™ืช,
05:53
because if you were to show the exact probabilities,
141
353976
2455
ื‘ื’ืœืœ ืฉืื ื”ื™ื™ืชื ืžืจืื™ื ืืช ื”ื”ืกืชื‘ืจื•ื™ื•ืช ื”ืžื“ื•ื™ื™ืงื•ืช,
05:56
maybe that would encourage people to get their flu jabs
142
356455
2592
ืื•ืœื™ ื–ื” ื™ืขื•ื“ื“ ืื ืฉื™ื ืœืงื‘ืœ ื—ื™ืกื•ื ื™ื ืœืฉืคืขืช ื‘ื–ืžืŸ ื”ืœื ื ื›ื•ืŸ.
05:59
at the wrong time.
143
359071
1456
06:00
The point of these shaky lines
144
360983
1693
ื”ื ืงื•ื“ื” ืฉืœ ื”ืงื•ื•ื™ื ื”ืจืขื•ืขื™ื ื”ืืœื”
06:02
is so that people remember these imprecisions,
145
362700
2911
ื”ื™ื ืฉืื ืฉื™ื ื™ื–ื›ืจื• ืืช ื—ื•ืกืจ ื”ื“ื™ื•ืงื™ื ื”ืืœื”,
06:05
but also so they don't necessarily walk away with a specific number,
146
365635
3227
ืื‘ืœ ื’ื ืฉื”ื ืœื ื‘ื”ื›ืจื— ื™ืœื›ื• ืขื ืžืกืคืจ ืžืกื•ื™ื™ื,
06:08
but they can remember important facts.
147
368886
1866
ืื‘ืœ ื”ื ื™ื›ื•ืœื™ื ืœื–ื›ื•ืจ ืขื•ื‘ื“ื•ืช ื—ืฉื•ื‘ื•ืช.
06:10
Facts like injustice and inequality leave a huge mark on our lives.
148
370776
4024
ืขื•ื‘ื“ื•ืช ื›ืžื• ื—ื•ืกืจ ืฆื“ืง ื•ื—ื•ืกืจ ืฉื•ื•ื™ื•ืŸ ืžืฉืื™ืจื™ื ืกื™ืžืŸ ื’ื“ื•ืœ ืขืœ ื”ื—ื™ื™ื ืฉืœื ื•.
06:14
Facts like Black Americans and Native Americans have shorter life expectancies
149
374824
4189
ืขื•ื‘ื“ื•ืช ื›ืžื• ืฉืœืืžืจื™ืงืื™ื ืฉื—ื•ืจื™ื ื•ืœืืžืจื™ืงืื™ื ื™ืœื™ื“ื™ื ื™ืฉ ืชื•ื—ืœืช ื—ื™ื™ื ืงืฆืจื” ื™ื•ืชืจ
06:19
than those of other races,
150
379037
1400
ืžืืœื• ืžื’ื–ืขื™ื ืื—ืจื™ื,
06:20
and that isn't changing anytime soon.
151
380461
2138
ื•ื–ื” ืœื ื™ืฉืชื ื” ื‘ืงืจื•ื‘.
06:22
Facts like prisoners in the US can be kept in solitary confinement cells
152
382623
3901
ืขื•ื‘ื“ื•ืช ื›ืžื• ืฉืืกื™ืจื™ื ื‘ืืจืฆื•ืช ื”ื‘ืจื™ืช ื™ื›ื•ืœื™ื ืœื”ื™ื•ืช ืžื•ื—ื–ืงื™ื ื‘ืชืื™ ื‘ื™ื“ื•ื“
06:26
that are smaller than the size of an average parking space.
153
386548
3342
ืฉืงื˜ื ื™ื ื™ื•ืชืจ ืžื’ื•ื“ืœ ืฉืœ ืžืงื•ื ื—ื ื™ื” ืžืžื•ืฆืข.
06:30
The point of these visualizations is also to remind people
154
390355
3335
ื”ื ืงื•ื“ื” ืฉืœ ื”ื”ื“ื’ืžื•ืช ื”ืืœื” ื”ื™ื ื’ื ืœื”ื–ื›ื™ืจ ืœืื ืฉื™ื
06:33
of some really important statistical concepts,
155
393714
2350
ื›ืžื” ื—ืฉื•ื‘ื™ื ืจืขื™ื•ื ื•ืช ืกื˜ื˜ื™ืกื˜ื™ื,
06:36
concepts like averages.
156
396088
1636
ืจืขื™ื•ื ื•ืช ื›ืžื• ืžืžื•ืฆืขื™ื.
06:37
So let's say you hear a claim like,
157
397748
1668
ืื– ื‘ื•ืื• ื ื’ื™ื“ ืฉืืชื ืฉื•ืžืขื™ื ื˜ืขื ื” ื›ืžื•,
06:39
"The average swimming pool in the US contains 6.23 fecal accidents."
158
399440
4434
"ื‘ืจื™ื›ืช ื”ืฉื—ื™ื” ื”ืžืžื•ืฆืขืช ื‘ืืจืฆื•ืช ื”ื‘ืจื™ืช ืžื›ื™ืœื” 6.23 ืชืื•ื ื•ืช ืฆื•ืืชื™ื•ืช."
06:43
That doesn't mean every single swimming pool in the country
159
403898
2797
ื–ื” ืœื ืื•ืžืจ ืฉื›ืœ ื‘ืจื™ื›ื” ื‘ืžื“ื™ื ื”
06:46
contains exactly 6.23 turds.
160
406719
2194
ืžื›ื™ืœื” 6.23 ืฆื•ืื•ืช.
06:48
So in order to show that,
161
408937
1417
ืื– ื›ื“ื™ ืœื”ืจืื•ืช ืืช ื–ื”,
06:50
I went back to the original data, which comes from the CDC,
162
410378
2841
ื—ื–ืจืชื™ ืœืžื™ื“ืข ื”ืžืงื•ืจื™, ืฉืžื’ื™ืข ืžื”ืžืจื›ื– ืœืฉืœื™ื˜ื” ื‘ืžื—ืœื•ืช,
06:53
who surveyed 47 swimming facilities.
163
413243
2065
ืฉืกืงืจ 47 ืžืชืงื ื™ื ืฉืœ ื‘ืจื™ื›ื•ืช ืฉื—ื™ื”.
06:55
And I just spent one evening redistributing poop.
164
415332
2391
ื•ื‘ื™ืœื™ืชื™ ืจืง ืขืจื‘ ืื—ื“ ื‘ืœืคื–ืจ ืžื—ื“ืฉ ืงืงื™.
06:57
So you can kind of see how misleading averages can be.
165
417747
2682
ืื– ืืชื ื™ื›ื•ืœื™ื ืœืจืื•ืช ื›ืžื” ืžื˜ืขื” ื”ืžืžื•ืฆืข ื”ื–ื” ื™ื›ื•ืœ ืœื”ื™ื•ืช.
07:00
(Laughter)
166
420453
1282
(ืฆื—ื•ืง)
07:01
OK, so the second question that you guys should be asking yourselves
167
421759
3901
ืื•ืงื™ื™, ืื– ื”ืฉืืœื” ื”ืฉื ื™ื” ืฉืืชื ืฆืจื™ื›ื™ื ืœืฉืื•ืœ ืืช ืขืฆืžื›ื
07:05
to spot bad numbers is:
168
425684
1501
ืœื–ื”ื•ืช ืžืกืคืจื™ื ื’ืจื•ืขื™ื ื”ื™ื:
07:07
Can I see myself in the data?
169
427209
1967
ื”ืื ืื ื™ ื™ื›ื•ืœ ืœืžืฆื•ื ืืช ืขืฆืžื™ ื‘ืžื™ื“ืข?
07:09
This question is also about averages in a way,
170
429200
2913
ื”ืฉืืœื” ื”ื–ื• ื ื•ื’ืขืช ื’ื ืœืžืžื•ืฆืขื™ื ื‘ื“ืจืš ืžืกื•ื™ื™ืžืช,
07:12
because part of the reason why people are so frustrated
171
432137
2605
ื‘ื’ืœืœ ืฉื—ืœืง ืžื”ืกื™ื‘ื” ืฉืื ืฉื™ื ื›ืœ ื›ืš ืžืชื•ืกื›ืœื™ื
07:14
with these national statistics,
172
434766
1495
ืžื”ืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ื”ืืจืฆื™ื•ืช ื”ืืœื•,
ื”ื™ื ืฉื”ืŸ ืœื ื‘ืืžืช ืžืกืคืจื•ืช ืืช ื”ืกื™ืคื•ืจ ืฉืœ ืžื™ ืžื ืฆื— ื•ืžื™ ืžืคืกื™ื“
07:16
is they don't really tell the story of who's winning and who's losing
173
436285
3273
07:19
from national policy.
174
439582
1156
ืžืžื“ื™ื ื™ื•ืช ืœืื•ืžื™ืช.
07:20
It's easy to understand why people are frustrated with global averages
175
440762
3318
ื–ื” ืงืœ ืœื”ื‘ื™ืŸ ืœืžื” ืื ืฉื™ื ืžืชื•ืกื›ืœื™ื ืžืžืžื•ืฆืขื™ื ืขื•ืœืžื™ื™ื
ื›ืฉื”ื ืœื ืžืชืื™ืžื™ื ืœื—ื•ื•ื™ื•ืช ื”ืื™ืฉื™ื•ืช ืฉืœื”ื.
07:24
when they don't match up with their personal experiences.
176
444104
2679
07:26
I wanted to show people the way data relates to their everyday lives.
177
446807
3263
ืจืฆื™ืชื™ ืœื”ืจืื•ืช ืœืื ืฉื™ื ืืช ื”ื“ืจืš ืฉืžื™ื“ืข ืžืชื™ื™ื—ืก ืœื—ื™ื™ื ื”ื™ื•ื ื™ื•ืžื™ื™ื ืฉืœื”ื.
ื”ืชื—ืœืชื™ ืืช ื˜ื•ืจ ื”ื™ื™ืขื•ืฅ ื”ื–ื” ืฉื ืงืจื "ืžื•ื ื” ื”ื™ืงืจื”,"
07:30
I started this advice column called "Dear Mona,"
178
450094
2246
ืฉื ืื ืฉื™ื ื™ื›ืชื‘ื• ืœื™ ืฉืืœื•ืช ื•ื“ืื’ื•ืช
07:32
where people would write to me with questions and concerns
179
452364
2726
ื•ื”ื™ื™ืชื™ ืžื ืกื” ืœืขื ื•ืช ืœื”ื ืขื ืžื™ื“ืข.
07:35
and I'd try to answer them with data.
180
455114
1784
ืื ืฉื™ื ืฉืืœื• ืื•ืชื™ ื›ืœ ื“ื‘ืจ.
07:36
People asked me anything.
181
456922
1200
ืฉืืœื•ืช ื›ืžื•, "ื”ืื ื–ื” ื ื•ืจืžืœื™ ืœื™ืฉื•ืŸ ื‘ืžื™ื˜ื” ื ืคืจื“ืช ืžืืฉืชื™?"
07:38
questions like, "Is it normal to sleep in a separate bed to my wife?"
182
458146
3261
"ื”ืื ืื ืฉื™ื ืžืชื—ืจื˜ื™ื ืขืœ ื›ืชื•ื‘ื•ืช ื”ืงืขืงืข ืฉืœื”ื?"
07:41
"Do people regret their tattoos?"
183
461431
1591
"ืžื” ื–ื” ืื•ืžืจ ืœืžื•ืช ืžืกื™ื‘ื•ืช ื˜ื‘ืขื™ื•ืช?"
07:43
"What does it mean to die of natural causes?"
184
463046
2164
07:45
All of these questions are great, because they make you think
185
465234
2966
ื›ืœ ื”ืฉืืœื•ืช ื”ืืœื• ืžืขื•ืœื•ืช, ื‘ื’ืœืœ ืฉื”ืŸ ื’ื•ืจืžื•ืช ืœื›ื ืœื—ืฉื•ื‘
ืขืœ ื“ืจื›ื™ื ืœืžืฆื•ื ื•ืœืชืงืฉืจ ืืช ื”ืžืกืคืจื™ื ื”ืืœื”.
07:48
about ways to find and communicate these numbers.
186
468224
2336
07:50
If someone asks you, "How much pee is a lot of pee?"
187
470584
2503
ืื ืžื™ืฉื”ื• ืฉื•ืืœ ืืชื›ื, "ื›ืžื” ืคื™ืคื™ ื–ื” ื”ืจื‘ื” ืคื™ืคื™?"
ืฉื–ื• ืฉืืœื” ืฉื ืฉืืœืชื™,
07:53
which is a question that I got asked,
188
473111
2458
07:55
you really want to make sure that the visualization makes sense
189
475593
2980
ืืชื ื‘ืืžืช ืจื•ืฆื™ื ืœื“ืื•ื’ ืฉื”ื•ื™ื–ื•ืืœื™ื–ืฆื™ื” ืชื”ื™ื” ื”ื’ื™ื•ื ื™ืช
ืœื›ืžื” ืฉื™ื•ืชืจ ืื ืฉื™ื ืฉืืคืฉืจ.
07:58
to as many people as possible.
190
478597
1747
08:00
These numbers aren't unavailable.
191
480368
1575
ื”ืžืกืคืจื™ื ื”ืืœื” ื›ืŸ ื–ืžื™ื ื™ื.
08:01
Sometimes they're just buried in the appendix of an academic study.
192
481967
3507
ืœืคืขืžื™ื ื”ื ืกืชื ืงื‘ื•ืจื™ื ื‘ื ืกืคื— ืฉืœ ืžื—ืงืจ ืืงื“ืžื™.
08:05
And they're certainly not inscrutable;
193
485498
1839
ื•ื”ื ื‘ื”ื—ืœื˜ ืœื ื‘ืœืชื™ ืžื•ื‘ื ื™ื;
08:07
if you really wanted to test these numbers on urination volume,
194
487361
2975
ืื ืืชื ื‘ืืžืช ืจื•ืฆื™ื ืœื‘ื—ื•ืŸ ืืช ื”ืžืกืคืจื™ื ื”ืืœื” ืขืœ ื›ืžื•ืช ื”ืฉืชืŸ,
ืืชื ื™ื›ื•ืœื™ื ืœืงื—ืช ื‘ืงื‘ื•ืง ื•ืœื ืกื•ืช ื‘ืขืฆืžื›ื.
08:10
you could grab a bottle and try it for yourself.
195
490360
2257
08:12
(Laughter)
196
492641
1008
(ืฆื—ื•ืง)
08:13
The point of this isn't necessarily
197
493673
1694
ื”ื ืงื•ื“ื” ืฉืœ ื–ื” ื”ื™ื ืœื ื‘ื”ื›ืจื— ืฉื›ืœ ืžืื’ืจ ื ืชื•ื ื™ื ื—ื™ื™ื‘ ืœื”ื™ื•ืช ืงืฉื•ืจ ืกืคืฆื™ืคื™ืช ืืœื™ื›ื.
08:15
that every single data set has to relate specifically to you.
198
495391
2877
ืื ื™ ืžืชืขื ื™ื™ื ืช ืœื›ืžื” ื ืฉื™ื ื‘ืฆืจืคืช ื ืชื ื• ืงื ืกื•ืช
08:18
I'm interested in how many women were issued fines in France
199
498292
2880
ืขืœ ืœื‘ื™ืฉืช ืจืขืœื”, ืื• ื ื™ืงืื‘,
08:21
for wearing the face veil, or the niqab,
200
501196
1959
ืืคื™ืœื• ืื ืื ื™ ืœื ื—ื™ื™ื” ื‘ืฆืจืคืช ืื• ืœื•ื‘ืฉืช ื›ื™ืกื•ื™ ืคื ื™ื.
08:23
even if I don't live in France or wear the face veil.
201
503179
2618
08:25
The point of asking where you fit in is to get as much context as possible.
202
505821
3835
ื”ื ืงื•ื“ื” ืฉืœ ืœืฉืื•ืœ ืื™ืš ื–ื” ืžืชืื™ื ืืœื™ืš ื”ื™ื ืœืงื‘ืœ ื›ืžื” ืฉื™ื•ืชืจ ื”ืงืฉืจ.
08:29
So it's about zooming out from one data point,
203
509680
2191
ื–ื” ืžื“ื‘ืจ ืขืœ ืœื”ืกืชื›ืœ ืžืจื—ื•ืง ืžื ืงื•ื“ืช ืžื‘ื˜ ืื—ืช,
08:31
like the unemployment rate is five percent,
204
511895
2104
ื›ืžื• ืฉืจืžืช ื”ืื‘ื˜ืœื” ื”ื™ื ื—ืžื™ืฉื” ืื—ื•ื–ื™ื,
ื•ืœืจืื•ืช ืื™ืš ื”ื™ื ืžืฉืชื ื” ืขื ื”ื–ืžืŸ,
08:34
and seeing how it changes over time,
205
514023
1757
08:35
or seeing how it changes by educational status --
206
515804
2650
ืื• ืœืจืื•ืช ืื™ืš ื”ื™ื ืžืฉืชื ื” ืขืœ ื™ื“ื™ ืกื˜ื˜ื•ืก ื—ื™ื ื•ื›ื™ --
08:38
this is why your parents always wanted you to go to college --
207
518478
3104
ื–ื• ื”ืกื™ื‘ื” ืฉื”ื”ื•ืจื™ื ืฉืœื›ื ืชืžื™ื“ ืจืฆื• ืฉืชืœื›ื• ืœืžื›ืœืœื” --
08:41
or seeing how it varies by gender.
208
521606
2032
ืื• ืœืจืื•ืช ื›ืžื” ื–ื” ืžืฉืชื ื” ืœืคื™ ืžื’ื“ืจ.
08:43
Nowadays, male unemployment rate is higher
209
523662
2127
ื”ื™ื•ื, ืื‘ื˜ืœื” ื’ื‘ืจื™ืช ื’ื‘ื•ื”ื” ื™ื•ืชืจ
08:45
than the female unemployment rate.
210
525813
1700
ืžืจืžืช ื”ืื‘ื˜ืœื” ื”ื ืฉื™ืช.
08:47
Up until the early '80s, it was the other way around.
211
527537
2695
ืขื“ ืœืชื—ื™ืœืช ืฉื ื•ืช ื” 80, ื–ื” ื”ื™ื” ื”ืคื•ืš.
ื–ื” ืกื™ืคื•ืจ ืขืœ ืื—ื“ ื”ืฉื™ื ื•ื™ื™ื ื”ื’ื“ื•ืœื™ื
08:50
This is a story of one of the biggest changes
212
530256
2117
08:52
that's happened in American society,
213
532397
1720
ืฉื”ืชืจื—ืฉื• ื‘ื—ื‘ืจื” ื”ืืžืจื™ืงืื™ืช,
ื•ื–ื” ื”ื›ืœ ืฉื ื‘ืชืจืฉื™ื, ื‘ืจื’ืข ืฉืืชื ืžื‘ื™ื˜ื™ื ืžืขื‘ืจ ืœืžืžื•ืฆืขื™ื.
08:54
and it's all there in that chart, once you look beyond the averages.
214
534141
3276
ื”ืฆื™ืจื™ื ื”ื ื”ื›ืœ;
08:57
The axes are everything;
215
537441
1165
08:58
once you change the scale, you can change the story.
216
538630
2669
ื‘ืจื’ืข ืฉืืชื ืžืฉื ื™ื ืืช ืงื ื” ื”ืžื™ื“ื”, ืืชื ื™ื›ื•ืœื™ื ืœืฉื ื•ืช ืืช ื”ืกื™ืคื•ืจ.
09:01
OK, so the third and final question that I want you guys to think about
217
541323
3380
ืื•ืงื™ื™, ืื– ื”ืฉืืœื” ื”ืฉืœื™ืฉื™ืช ื•ื”ืื—ืจื•ื ื” ืฉืื ื™ ืจื•ืฆื” ืฉืชื—ืฉื‘ื• ืขืœื™ื”
09:04
when you're looking at statistics is:
218
544727
1819
ื›ืฉืืชื ืžื‘ื™ื˜ื™ื ื‘ืกื˜ื˜ื™ืกื˜ื™ืงื” ื”ื™ื:
09:06
How was the data collected?
219
546570
1873
ืื™ืš ื”ืžื™ื“ืข ื ืืกืฃ?
09:09
So far, I've only talked about the way data is communicated,
220
549487
2939
ืขื“ ืขื›ืฉื™ื•, ื“ื™ื‘ืจืชื™ ืจืง ืขืœ ื”ื“ืจืš ืฉืžื™ื“ืข ืžืชื•ืงืฉืจ,
ืื‘ืœ ื”ื“ืจืš ื‘ื” ื”ื•ื ื ืืกืฃ ืžืฉื ื” ื‘ืื•ืชื” ืžื™ื“ื”.
09:12
but the way it's collected matters just as much.
221
552450
2276
ืื ื™ ื™ื•ื“ืขืช ืฉื–ื” ืงืฉื”,
09:14
I know this is tough,
222
554750
1167
ื‘ื’ืœืœ ืฉืžืชื•ื“ื•ืœื•ื’ื™ื•ืช ื™ื›ื•ืœื•ืช ืœื”ื™ื•ืช ืื˜ื•ืžื•ืช ื•ืœืžืขืฉื” ืกื•ื’ ืฉืœ ืžืฉืขืžืžื•ืช,
09:15
because methodologies can be opaque and actually kind of boring,
223
555941
3081
ืื‘ืœ ื™ืฉ ื›ืžื” ืฉืœื‘ื™ื ืคืฉื•ื˜ื™ื ืฉืืชื ื™ื›ื•ืœื™ื ืœืขืฉื•ืช ื›ื“ื™ ืœื‘ื“ื•ืง ืืช ื–ื”.
09:19
but there are some simple steps you can take to check this.
224
559046
2873
09:21
I'll use one last example here.
225
561943
1839
ืื ื™ ืืฉืชืžืฉ ื‘ื“ื•ื’ืžื” ืื—ืช ืื—ืจื•ื ื” ืคื”.
09:24
One poll found that 41 percent of Muslims in this country support jihad,
226
564129
3887
ืกืงืจ ืื—ื“ ื’ื™ืœื” ืฉ-41 ืื—ื•ื–ื™ื ืžื”ืžื•ืกืœืžื™ื ื‘ืžื“ื™ื ื” ืชื•ืžื›ื™ื ื‘ื’'ื™ื”ืื“,
09:28
which is obviously pretty scary,
227
568040
1525
ืฉื–ื” ื‘ื‘ืจื•ืจ ื“ื™ ืžืคื—ื™ื“,
09:29
and it was reported everywhere in 2015.
228
569589
2642
ื•ื–ื” ื“ื•ื•ื— ื‘ื›ืœ ืžืงื•ื ื‘-2015.
09:32
When I want to check a number like that,
229
572255
2615
ื›ืฉืื ื™ ืจื•ืฆื” ืœื‘ื“ื•ืง ืžืกืคืจ ื›ืžื• ื–ื”,
09:34
I'll start off by finding the original questionnaire.
230
574894
2501
ืื ื™ ืืชื—ื™ืœ ื‘ืœืžืฆื•ื ืืช ื”ืฉืืœื•ืŸ ื”ืžืงื•ืจื™.
09:37
It turns out that journalists who reported on that statistic
231
577419
2926
ืžืกืชื‘ืจ ืฉืขื™ืชื•ื ืื™ื ืฉื“ื™ื•ื•ื—ื• ืขืœ ื”ืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ื”ืืœื”
09:40
ignored a question lower down on the survey
232
580369
2231
ื”ืชืขืœืžื• ืžืฉืืœื” ื‘ืžื•ืจื“ ื”ืกืงืจ
09:42
that asked respondents how they defined "jihad."
233
582624
2346
ืฉืฉืืœื” ืžืฉื™ื‘ื™ื ืื™ืš ื”ื ืžื’ื“ื™ืจื™ื "ื’'ื™ื”ืื“."
09:44
And most of them defined it as,
234
584994
1981
ื•ืจื•ื‘ื ื”ื’ื“ื™ืจื• ืื•ืชื•,
09:46
"Muslims' personal, peaceful struggle to be more religious."
235
586999
3942
ื›"ืžืื‘ืง ื”ืื™ืฉื™ ื”ืจื’ื•ืข ืฉืœ 'ืžื•ืกืœืžื™' ืœื”ื™ื•ืช ื™ื•ืชืจ ื“ืชื™."
09:50
Only 16 percent defined it as, "violent holy war against unbelievers."
236
590965
4194
ืจืง 16 ืื—ื•ื– ื”ื’ื“ื™ืจื• ืื•ืชื• ื›"ืžืœื—ืžื” ืืœื™ืžื” ืงื“ื•ืฉื” ื ื’ื“ ืœื ืžืืžื™ื ื™ื."
09:55
This is the really important point:
237
595183
2430
ื–ื• ื ืงื•ื“ื” ื‘ืืžืช ื—ืฉื•ื‘ื”:
09:57
based on those numbers, it's totally possible
238
597637
2155
ื‘ื”ืชื‘ืกืก ืขืœ ื”ืžืกืคืจื™ื ื”ืืœื”, ื–ื” ืœื’ืžืจื™ ืืคืฉืจื™
09:59
that no one in the survey who defined it as violent holy war
239
599816
3105
ืฉืืฃ ืื—ื“ ื‘ืกืงืจ ืฉื”ื’ื“ื™ืจ ืื•ืชื• ื›ืžืœื—ืžื” ืงื“ื•ืฉื” ืืœื™ืžื”
10:02
also said they support it.
240
602945
1332
ื’ื ืืžืจื• ืฉื”ื ืชื•ืžื›ื™ื ื‘ื”.
10:04
Those two groups might not overlap at all.
241
604301
2208
ืฉืชื™ ื”ืงื‘ื•ืฆื•ืช ื”ืืœื” ืื•ืœื™ ืœื ื—ื•ืคืคื•ืช ื‘ื›ืœืœ.
10:06
It's also worth asking how the survey was carried out.
242
606942
2637
ื–ื” ื’ื ืฉื•ื•ื” ืœืฉืื•ืœ ืื™ืš ื”ืกืงืจ ื ืขืจืš.
10:09
This was something called an opt-in poll,
243
609603
1998
ื–ื” ื”ื™ื” ืžืฉื”ื• ืฉื ืงืจื ืกืงืจ ืœืคื™ ื‘ื—ื™ืจื”,
10:11
which means anyone could have found it on the internet and completed it.
244
611625
3402
ืžื” ืฉืื•ืžืจ ืฉื›ืœ ืื—ื“ ื”ื™ื” ื™ื›ื•ืœ ืœืžืฆื•ื ืื•ืชื• ื‘ืื™ื ื˜ืจื ื˜ ื•ืœืžืœื ืื•ืชื•.
ืื™ืŸ ื“ืจืš ืœื“ืขืช ืื ื”ืื ืฉื™ื ื”ืืœื” ืืคื™ืœื• ืžื–ื“ื”ื™ื ื›ืžื•ืกืœืžื™ื.
10:15
There's no way of knowing if those people even identified as Muslim.
245
615051
3339
10:18
And finally, there were 600 respondents in that poll.
246
618414
2612
ื•ืœื‘ืกื•ืฃ, ื™ืฉ 600 ืžืฉื™ื‘ื™ื ื‘ืกืงืจ.
10:21
There are roughly three million Muslims in this country,
247
621050
2654
ื™ืฉ ื‘ืขืจืš ืฉืœื•ืฉื” ืžืœื™ื•ืŸ ืžื•ืกืœืžื™ื ื‘ืžื“ื™ื ื”,
10:23
according to Pew Research Center.
248
623728
1607
ืœืคื™ ืžืจื›ื– ื”ืžื—ืงืจ ืคื™ื•.
ื–ื” ืื•ืžืจ ืฉื”ืกืงืจ ื“ื™ื‘ืจ ื‘ื”ืขืจื›ื” ื’ืกื” ืœืื—ื“ ืžืชื•ืš 5,000 ืžื•ืกืœืžื™ื ื‘ืžื“ื™ื ื”.
10:25
That means the poll spoke to roughly one in every 5,000 Muslims
249
625359
2993
10:28
in this country.
250
628376
1168
10:29
This is one of the reasons
251
629568
1266
ื–ื• ืื—ืช ื”ืกื™ื‘ื•ืช
10:30
why government statistics are often better than private statistics.
252
630858
3607
ืฉืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ืžืžืฉืœืชื™ื•ืช ื”ืจื‘ื” ืคืขืžื™ื ื˜ื•ื‘ื•ืช ื™ื•ืชืจ ืžืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ืคืจื˜ื™ื•ืช.
10:34
A poll might speak to a couple hundred people, maybe a thousand,
253
634489
3035
ืกืงืจ ืื•ืœื™ ื™ื“ื‘ืจ ืœื›ืžื” ืžืื•ืช ืื ืฉื™ื, ืื•ืœื™ ืœืืœืฃ,
10:37
or if you're L'Oreal, trying to sell skin care products in 2005,
254
637548
3058
ืื• ืื ืืชื ืœื•ืจื™ืืœ, ืžื ืกื™ื ืœืžื›ื•ืจ ืžื•ืฆืจื™ ื˜ื™ืคื•ืœ ื‘ืขื•ืจ ื‘-2005,
10:40
then you spoke to 48 women to claim that they work.
255
640630
2417
ืื– ื“ื™ื‘ืจืชื ืขื 48 ื ืฉื™ื ื›ื“ื™ ืœื˜ืขื•ืŸ ืฉื”ื ืขื•ื‘ื“ื™ื.
10:43
(Laughter)
256
643071
1026
(ืฆื—ื•ืง)
10:44
Private companies don't have a huge interest in getting the numbers right,
257
644121
3556
ืœื—ื‘ืจื•ืช ืคืจื˜ื™ื•ืช ืื™ืŸ ืื™ื ื˜ืจืก ื’ื“ื•ืœ ืœื”ื’ื™ืข ืœืžืกืคืจ ื ื›ื•ืŸ,
10:47
they just need the right numbers.
258
647701
1755
ื”ืŸ ืคืฉื•ื˜ ืจื•ืฆื•ืช ืžืกืคืจื™ื ืžืชืื™ืžื™ื.
10:49
Government statisticians aren't like that.
259
649480
2020
ืกื˜ื˜ื™ืกื˜ื™ืงืื™ื ืžืžืฉืœืชื™ื™ื ืœื ืขื•ื‘ื“ื™ื ื›ื›ื”.
10:51
In theory, at least, they're totally impartial,
260
651524
2447
ื‘ืชืื•ืจื™ื”, ืœืคื—ื•ืช, ื”ื ืœื’ืžืจื™ ืœื ืžื•ื˜ื™ื,
10:53
not least because most of them do their jobs regardless of who's in power.
261
653995
3501
ื‘ื™ืŸ ื”ื™ืชืจ ื‘ื’ืœืœ ืฉื”ื ืขื•ืฉื™ื ืืช ื”ืขื‘ื•ื“ื” ืฉืœื”ื ื‘ืœื™ ืงืฉืจ ืœืžื™ ืžื—ื–ื™ืง ื‘ืฉืœื˜ื•ืŸ.
10:57
They're civil servants.
262
657520
1162
ื”ื ืขื•ื‘ื“ื™ ืฆื™ื‘ื•ืจ.
10:58
And to do their jobs properly,
263
658706
1964
ื•ื›ื“ื™ ืœืขืฉื•ืช ืืช ื”ืขื‘ื•ื“ื” ืฉืœื”ื ื˜ื•ื‘,
11:00
they don't just speak to a couple hundred people.
264
660694
2363
ื”ื ืœื ืžื“ื‘ืจื™ื ืจืง ืœื›ืžื” ืžืื•ืช ืื ืฉื™ื.
11:03
Those unemployment numbers I keep on referencing
265
663081
2318
ืžืกืคืจื™ ื”ืชืขืกื•ืงื” ื”ืืœื” ืฉืื ื™ ืžืžืฉื™ื›ื” ืœื”ืชื™ื™ื—ืก ืืœื™ื”ื
11:05
come from the Bureau of Labor Statistics,
266
665423
2004
ืžื’ื™ืขื™ื ืžืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ืฉืœ ืœืฉื›ืช ื”ืชืขืกื•ืงื”,
11:07
and to make their estimates,
267
667451
1335
ื•ื›ื“ื™ ืœืขืฉื•ืช ืืช ื”ื”ืขืจื›ื•ืช ืฉืœื”ื,
11:08
they speak to over 140,000 businesses in this country.
268
668810
3489
ื”ื ืžื“ื‘ืจื™ื ืขื ื™ื•ืชืจ ืž-140,000 ืขืกืงื™ื ื‘ืžื“ื™ื ื”.
11:12
I get it, it's frustrating.
269
672323
1725
ืื ื™ ืžื‘ื™ื ื”, ื–ื” ืžืชืกื›ืœ.
11:14
If you want to test a statistic that comes from a private company,
270
674072
3115
ืื ืืชื ืจื•ืฆื™ื ืœื‘ื—ื•ืŸ ืกื˜ื˜ื™ืกื˜ื™ืงื” ืฉืžื’ื™ืขื” ืžื—ื‘ืจื” ืคืจื˜ื™ืช,
ืืชื ื™ื›ื•ืœื™ื ืœืงื ื•ืช ืืช ืงืจื ื”ืคื ื™ื ืœื›ื ื•ืœื›ืžื” ื—ื‘ืจื™ื, ืœื ืกื•ืช ืื•ืชื•,
11:17
you can buy the face cream for you and a bunch of friends, test it out,
271
677211
3361
ืื ื–ื” ืœื ืขื•ื‘ื“, ืืชื ื™ื›ื•ืœื™ื ืœื”ื’ื™ื“ ืฉื”ืžืกืคืจื™ื ื”ื™ื• ืฉื’ื•ื™ื™ื.
11:20
if it doesn't work, you can say the numbers were wrong.
272
680596
2591
ืื‘ืœ ืื™ืš ืืชื ืžืชืฉืืœื™ื ืกื˜ื˜ื™ืกื˜ื™ืงื•ืช ืžืžืฉืœืชื™ื•ืช?
11:23
But how do you question government statistics?
273
683211
2146
ืืชื ืจืง ืžืžืฉื™ื›ื™ื ืœื‘ื“ื•ืง ื”ื›ืœ.
11:25
You just keep checking everything.
274
685381
1630
ื’ืœื• ืื™ืš ื”ื ืื•ืกืคื™ื ืืช ื”ืžืกืคืจื™ื.
11:27
Find out how they collected the numbers.
275
687035
1913
ื’ืœื• ืื ืืชื ืจื•ืื™ื ื›ืœ ืžื” ืฉืืชื ืฆืจื™ื›ื™ื ืขืœ ื”ืชืจืฉื™ื.
11:28
Find out if you're seeing everything on the chart you need to see.
276
688972
3125
ืื‘ืœ ืืœ ืชืชื™ื™ืืฉื• ืžื”ืžืกืคืจื™ื ืœื’ืžืจื™ ื‘ื’ืœืœ ืฉืื ืชืขืฉื• ื–ืืช,
11:32
But don't give up on the numbers altogether, because if you do,
277
692121
2965
ื ืขืฉื” ื”ื—ืœื˜ื•ืช ืฉืœ ืžื“ื™ื ื™ื•ืช ืฆื™ื‘ื•ืจื™ืช ื‘ื—ืฉื™ื›ื”,
11:35
we'll be making public policy decisions in the dark,
278
695110
2439
ื‘ืฉื™ืžื•ืฉ ื‘ืœื ื™ื•ืชืจ ืžืื™ื ื˜ืจืกื™ื ืคืจื˜ื™ื™ื ืœื”ื ื—ื•ืช ืื•ืชื ื•.
11:37
using nothing but private interests to guide us.
279
697573
2262
11:39
Thank you.
280
699859
1166
ืชื•ื“ื” ืœื›ื.
(ืžื—ื™ืื•ืช ื›ืคื™ื™ื)
11:41
(Applause)
281
701049
2461
ืขืœ ืืชืจ ื–ื”

ืืชืจ ื–ื” ื™ืฆื™ื’ ื‘ืคื ื™ื›ื ืกืจื˜ื•ื ื™ YouTube ื”ืžื•ืขื™ืœื™ื ืœืœื™ืžื•ื“ ืื ื’ืœื™ืช. ืชื•ื›ืœื• ืœืจืื•ืช ืฉื™ืขื•ืจื™ ืื ื’ืœื™ืช ื”ืžื•ืขื‘ืจื™ื ืขืœ ื™ื“ื™ ืžื•ืจื™ื ืžื”ืฉื•ืจื” ื”ืจืืฉื•ื ื” ืžืจื—ื‘ื™ ื”ืขื•ืœื. ืœื—ืฅ ืคืขืžื™ื™ื ืขืœ ื”ื›ืชื•ื‘ื™ื•ืช ื‘ืื ื’ืœื™ืช ื”ืžื•ืฆื’ื•ืช ื‘ื›ืœ ื“ืฃ ื•ื™ื“ืื• ื›ื“ื™ ืœื”ืคืขื™ืœ ืืช ื”ืกืจื˜ื•ืŸ ืžืฉื. ื”ื›ืชื•ื‘ื™ื•ืช ื’ื•ืœืœื•ืช ื‘ืกื ื›ืจื•ืŸ ืขื ื”ืคืขืœืช ื”ื•ื•ื™ื“ืื•. ืื ื™ืฉ ืœืš ื”ืขืจื•ืช ืื• ื‘ืงืฉื•ืช, ืื ื ืฆื•ืจ ืื™ืชื ื• ืงืฉืจ ื‘ืืžืฆืขื•ืช ื˜ื•ืคืก ื™ืฆื™ืจืช ืงืฉืจ ื–ื”.

https://forms.gle/WvT1wiN1qDtmnspy7