AI isn’t enough to protect social media communities from AI - Ars Technica

Skip to content

AI

Biz & IT

Cars

Culture

Gaming

Health

Policy

Science

Security

Space

Tech

Forum

Subscribe

Story text

Size

Small<br>Standard<br>Large

Width

Standard<br>Wide

Links

Standard<br>Orange

* Subscribers only

Learn more

Pin to story

Theme

Search

Sign In

Sign in dialog...

Text<br>settings

Story text

Size

Small<br>Standard<br>Large

Width

Standard<br>Wide

Links

Standard<br>Orange

* Subscribers only

Learn more

Minimize to nav

Consumer Checkpoint

The tech products we rely on are constantly changing. Sometimes those changes bring improvements; other times, they frustrate users or make beloved products worse.<br>Each month, we’ll examine the biggest controversial changes to tech products: what changed, why it’s happening, how users are affected, and how companies can do better.

Sometimes you have to fight fire with fire. But when it comes to AI slop and hateful content threatening the safety and value of social media platforms, adding more fire—in this case, more AI—can make the problem worse.

At its best, social media can be a haven for people who want to share their experiences and knowledge. It gets closest to this ideal when users contribute authentic, valuable content, whether that’s a uniquely thoughtful blog post or a helpful video on how to build a PC. Relying primarily on AI tools to preserve that authenticity misses what makes social media worthwhile in the first place: the people behind it.

Erroneous erasures

In April, a Slack channel for moderators of the r/AskHistorians Reddit community was usually busy. The channel, which automatically receives links to modmail messages, was flooded with alerts after dozens of comments and posts dating back 10 years were automatically removed from the subreddit.

“And there was nothing we or the experts [who posted the deleted content] could do about it,” Dr. Sarah Gilbert, one of the mods, told me.

This was particularly damaging to the subreddit because its users view the community as an archive of detailed responses that continue to educate people long after content is posted.

Reddit’s recently revamped AI moderation tools were apparently responsible for the removals, the moderators believe. After recovering the text of some posts, one of AskHistorians’ mods noticed that all the removed content linked to Rare Historical Photos, a historical image-sharing website. The mods think Reddit might have designated the website—and thus any post using its content for explanatory illustrations—as spam.

Reddit has not responded to a request for comment.

The deletion of the content erased valuable information that had taken time to aggregate (Gilbert tells me some people spend hours, “sometimes over the course of days,” researching and writing responses to questions submitted to the subreddit). Yet it’s possible that those erroneous removals, and others like them, have contributed to metrics intended to demonstrate how effective AI modding is on Reddit.

Reddit says that thanks to AI, it has “increased enforcement actions on hate and violent content by more than 200 percent” and that AI drives “faster, higher volume enforcement.” AI has “helped reduce exposure to potentially harmful content by more than 40 percent,” Reddit said this month. It also said that it uses large language models (LLMs) to catch “the highly subtle, coordinated patterns of fake behavior and artificial hype.”

But as the AskHistorians ordeal illustrates, more enforcement doesn’t necessarily mean better enforcement.

The false positives problem

The growth of generative AI has created new obstacles for social media moderation. Gilbert noted, for instance, that large language models “have made spam detection a lot harder,” as they seek to mimic real human voices. “Over the last two to three months, we’ve been absolutely flooded by LLM-powered spambots,” she said.

Marketing agencies are creating social media content designed to get brands cited by generative AI chatbots. Marketers have long used inauthentic social media posts to boost visibility, but the rise of chatbots has opened a new front. Startup ReachLLM, for example, focuses specifically on marketing through chatbots. As part of that effort, company representatives have created and moderate subreddits on Reddit.

These challenges have led some social media companies to explore new AI-based moderation techniques. Reddit, for example, says its AI tools have “revoked nearly [2 million] fake votes daily” and that it uses LLMs “to catch the highly subtle, coordinated patterns of fake behavior and artificial hype that older systems once missed.”

But many social media platforms have become overly reliant on AI modding tools that have been quick to penalize users for innocuous content.

Recently, Discord admitted that its AI mod system wrongfully banned about 8,400 accounts in May to early July. The AI mistakenly labeled images containing square grids,...