Algorithms of Oppression

As I read through a discussion of how search and local law intersect in Noble’s Algorithms of Oppression, I was reminded of a discussion a friend and I had over dinner the other day.

He’d suggested that artificial intelligence is responsible for the content moderation at places like Facebook. Citing the apparently instant start to streaming for services like Facebook Live, he posited there was no way his streaming could be moderated in any way, at least not before he’d had the chance to broadcast. Were he to share something that violated policy, everyone would see what he broadcast before it was discontinued.

Since I have done a lot of manual work in AI, and it’s well-established that every company whose primary product is user generated content relies on an army of manual content moderators working around the clock, I suggested it was actually possible there was no AI in the system at all. This seemed impossible to him: a numbers game, at scale.

Why it’s interesting

We were quickly able to agree that the numbers scaled down considerably for his example of streaming video: on any given day, regardless of the number of daily active users, nearly none would be streaming live that day, and even fewer would be streaming at any given moment.

This is the important fact I wanted to discuss: Even still, with billions of daily active users, any streaming media site would still need tens of thousands of moderators monitoring incoming streams. Those workers are rarely considered when we’re talking about something that “just works.”

(Caveat: I don’t have any idea how many users use streaming video or how many content moderators there are, just that the services can’t exist without them.)

Why it matters

My friend was happy to attribute beheading- and revenge-porn free experiences using social media apps to AI. The very existence of the service depends on high-quality human judgement to make decisions though.

The people actually making the product feel safe and fast and amazing, are real. Humans create the emotional power of what’s marketed as AI though manual labor, mostly in the global south. Especially when, in the recent cultural narrative of The Social Dilemma, the narrative has started to change to frame the product being you. This is an alarming and emotional message.

The people doing the emotional labor and the toughest part of the job being invisible isn’t new. We can’t forget it though.