The price of a "clean" internet | Hans Block and Moritz Riesewieck

[This talk contains mature content]

Moritz Riesewieck: On March 23, 2013,

users worldwide discovered in their news feed

a video of a young girl being raped by an older man.

Before this video was removed from Facebook,

it was already shared 16,000 times,

and it was even liked 4,000 times.

This video went viral and infected the net.

Hans Block: And that was the moment we asked ourselves

how could something like this get on Facebook?

And at the same time, why don't we see such content more often?

After all, there's a lot of revolting material online,

but why do we so rarely see such crap on Facebook, Twitter or Google?

MR: While image-recognition software

can identify the outlines of sexual organs,

blood or naked skin in images and videos,

it has immense difficulties to distinguish pornographic content

from holiday pictures, Adonis statues

or breast-cancer screening campaigns.

It can't distinguish Romeo and Juliet dying onstage

from a real knife attack.

It can't distinguish satire from propaganda

or irony from hatred, and so on and so forth.

Therefore, humans are needed to decide

which of the suspicious content should be deleted,

and which should remain.

Humans whom we know almost nothing about,

because they work in secret.

They sign nondisclosure agreements,

which prohibit them from talking and sharing

what they see on their screens and what this work does to them.

They are forced to use code words in order to hide who they work for.

They are monitored by private security firms

in order to ensure that they don't talk to journalists.

And they are threatened by fines in case they speak.

All of this sounds like a weird crime story,

but it's true.

These people exist,

and they are called content moderators.

HB: We are the directors of the feature documentary film "The Cleaners,"

and we would like to take you

to a world that many of you may not know yet.

Here's a short clip of our film.

(Music)

(Video) Moderator: I need to be anonymous, because we have a contract signed.

We are not allowed to declare whom we are working with.

The reason why I speak to you

is because the world should know that we are here.

There is somebody who is checking the social media.

We are doing our best to make this platform

safe for all of them.

Delete.

Ignore.

Delete.

Ignore.

Delete.

Ignore.

Ignore.

Delete.

HB: The so-called content moderators

don't get their paychecks from Facebook, Twitter or Google themselves,

but from outsourcing firms around the world

in order to keep the wages low.

Tens of thousands of young people

looking at everything we are not supposed to see.

And we are talking about decapitations, mutilations,

executions, necrophilia, torture, child abuse.

Thousands of images in one shift --

ignore, delete, day and night.

And much of this work is done in Manila,

where the analog toxic waste from the Western world

was transported for years by container ships,

now the digital waste is dumped there via fiber-optic cable.

And just as the so-called scavengers

rummage through gigantic tips on the edge of the city,

the content moderators click their way through an endless toxic ocean

of images and videos and all manner of intellectual garbage,

so that we don't have to look at it.

MR: But unlike the wounds of the scavengers,

those of the content moderators remain invisible.

Full of shocking and disturbing content,

these pictures and videos burrow into their memories

where, at any time, they can have unpredictable effects:

eating disorders, loss of libido,

anxiety disorders, alcoholism,

depression, which can even lead to suicide.

The pictures and videos infect them,

and often never let them go again.

If they are unlucky, they develop post-traumatic stress disorders,

like soldiers after war missions.

In our film, we tell the story of a young man

who had to monitor livestreams of self-mutilations and suicide attempts,

again and again,

and who eventually committed suicide himself.

It's not an isolated case, as we've been told.

This is the price all of us pay

for our so-called clean and safe and "healthy"

environments on social media.

Never before in the history of mankind

has it been easier to reach millions of people around the globe

in a few seconds.

What is posted on social media spreads so quickly,

becomes viral and excites the minds of people all around the globe.

Before it is deleted,

it is often already too late.

Millions of people have already been infected

with hatred and anger,

and they either become active online,

by spreading or amplifying hatred,

or they take to the streets and take up arms.

HB: Therefore, an army of content moderators

sit in front of a screen to avoid new collateral damage.

And they are deciding, as soon as possible,

whether the content stays on the platform -- ignore;

or disappears -- delete.

But not every decision is as clear

as the decision about a child-abuse video.

What about controversial content, ambivalent content,

uploaded by civil rights activists or citizen journalists?

The content moderators often decide on such cases

at the same speed as the [clear] cases.

MR: We will show you a video now,

and we would like to ask you to decide:

Would you delete it,

or would you not delete it?

(Video) (Air strike sounds)

(Explosion)

(People speaking in Arabic)

MR: Yeah, we did some blurring for you.

A child would potentially be dangerously disturbed

and extremely frightened by such content.

So, you rather delete it?

But what if this video could help investigate the war crimes in Syria?

What if nobody would have heard about this air strike,

because Facebook, YouTube, Twitter would have decided to take it down?

Airwars, a nongovernmental organization based in London,

tries to find those videos as quickly as possible

whenever they are uploaded to social media,

in order to archive them.

Because they know, sooner or later,

Facebook, YouTube, Twitter would take such content down.

People armed with their mobile phones

can make visible what journalists often do not have access to.

Civil rights groups often do not have any better option

to quickly make their recordings accessible to a large audience

than by uploading them to social media.

Wasn't this the empowering potential the World Wide Web should have?

Weren't these the dreams

people in its early stages had about the World Wide Web?

Can't pictures and videos like these

persuade people who have become insensitive to facts

to rethink?

HB: But instead, everything that might be disturbing is deleted.

And there's a general shift in society.

Media, for example, more and more often use trigger warnings

at the top of articles

which some people may perceive as offensive or troubling.

Or more and more students at universities in the United States

demand the banishment of antique classics

which depict sexual violence or assault from the curriculum.

But how far should we go with that?

Physical integrity is guaranteed as a human right

in constitutions worldwide.

In the Charter of Fundamental Rights of the European Union,

this right expressly applies to mental integrity.

But even if the potentially traumatic effect

of images and videos is hard to predict,

do we want to become so cautious

that we risk losing social awareness of injustice?

So what to do?

Mark Zuckerberg recently stated that in the future,

the users, we, or almost everybody,

will decide individually

what they would like to see on the platform,

by personal filter settings.

So everyone could easily claim to remain undisturbed

by images of war or other violent conflicts, like ...

MR: I'm the type of guy who doesn't mind seeing breasts

and I'm very interested in global warming,

but I don't like war so much.

HB: Yeah, I'm more the opposite,

I have zero interest in naked breasts or naked bodies at all.

But why not guns? I like guns, yes.

MR: Come on, if we don't share a similar social consciousness,

how shall we discuss social problems?

How shall we call people to action?

Even more isolated bubbles would emerge.

One of the central questions is: "How, in the future,

freedom of expression will be weighed against the people's need for protection."

It's a matter of principle.

Do we want to design an either open or closed society

for the digital space?

At the heart of the matter is "freedom versus security."

Facebook has always wanted to be a "healthy" platform.

Above all, users should feel safe and secure.

It's the same choice of words

the content moderators in the Philippines used

in a lot of our interviews.

(Video) The world that we are living in right now,

I believe, is not really healthy.

(Music)

In this world, there is really an evil who exists.

(Music)

We need to watch for it.

(Music)

We need to control it -- good or bad.

(Music)

[Look up, Young man! --God]

MR: For the young content moderators in the strictly Catholic Philippines,

this is linked to a Christian mission.

To counter the sins of the world

which spread across the web.

"Cleanliness is next to godliness,"

is a saying everybody in the Philippines knows.

HB: And others motivate themselves

by comparing themselves with their president, Rodrigo Duterte.

He has been ruling the Philippines since 2016,

and he won the election with the promise: "I will clean up."

And what that means is eliminating all kinds of problems

by literally killing people on the streets

who are supposed to be criminals, whatever that means.

And since he was elected,

an estimated 20,000 people have been killed.

And one moderator in our film says,

"What Duterte does on the streets,

I do for the internet."

And here they are, our self-proclaimed superheroes,

who enforce law and order in our digital world.

They clean up, they polish everything clean,

they free us from everything evil.

Tasks formerly reserved to state authorities

have been taken over by college graduates in their early 20s,

equipped with three- to five-day training --

this is the qualification --

who work on nothing less than the world's rescue.

MR: National sovereignties have been outsourced to private companies,

and they pass on their responsibilities to third parties.

It's an outsourcing of the outsourcing of the outsourcing,

which takes place.

With social networks,

we are dealing with a completely new infrastructure,

with its own mechanisms,

its own logic of action

and therefore, also, its own new dangers,

which had not yet existed in the predigitalized public sphere.

HB: When Mark Zuckerberg was at the US Congress

or at the European Parliament,

he was confronted with all kinds of critics.

And his reaction was always the same:

"We will fix that,

and I will follow up on that with my team."

But such a debate shouldn't be held in back rooms of Facebook,

Twitter or Google --

such a debate should be openly discussed in new, cosmopolitan parliaments,

in new institutions that reflect the diversity of people

contributing to a utopian project of a global network.

And while it may seem impossible to consider the values

of users worldwide,

it's worth believing

that there's more that connects us than separates us.

MR: Yeah, at a time when populism is gaining strength,

it becomes popular to justify the symptoms,

to eradicate them,

to make them invisible.

This ideology is spreading worldwide,

analog as well as digital,

and it's our duty to stop it

before it's too late.

The question of freedom and democracy

must not only have these two options.

HB: Delete.

MR: Or ignore.

HB: Thank you very much.

(Applause)