Google sues Chinese cybercrime network that used Gemini to automate scams - Ars Technica

Skip to content

AI

Biz & IT

Cars

Culture

Gaming

Health

Policy

Science

Security

Space

Tech

Forum

Subscribe

Story text

Size

Small<br>Standard<br>Large

Width

Standard<br>Wide

Links

Standard<br>Orange

* Subscribers only

Learn more

Pin to story

Theme

Search

Sign In

Sign in dialog...

Text<br>settings

Story text

Size

Small<br>Standard<br>Large

Width

Standard<br>Wide

Links

Standard<br>Orange

* Subscribers only

Learn more

Minimize to nav

Google loves telling us all the ways people are using its generative AI products to build new things, grow businesses, and save the world. Supposedly. Of course, people are also using AI for crime. Google has announced a new legal salvo aimed at a Chinese group called Outsider Enterprise, which is allegedly responsible for a massive AI-powered scam campaign. Google says it’s working with law enforcement and mobile carriers to fight back.

According to Google’s legal filing, Outsider Enterprise operates through Telegram. The group offers phishing-as-a-service to individuals who may not be technically savvy enough to set up fraudulent websites and text campaigns on their own. In its Telegram channels, Outsider Enterprise reportedly provided instructions on how to use Google’s Gemini AI to create websites that imitate those of Google, YouTube, and government agencies such as New York’s E-ZPass. The group offered nearly 300 scam templates.

Google says that scams enabled by Outsider Enterprise resulted in more than 2.5 million text messages being sent to Android users. About 55,000 of those messages happened in a two-week period last month. In all, Google has tracked 9,000 fake websites and 1 million URLs connected to the scam network.

The text messages often made claims about account problems or issues with a package delivery. When users clicked on the links, they ended up on one of those fraudulent websites, designed by Gemini to look legitimate. The cybercriminals used these sites to steal personal data and banking details. Google’s filing does not estimate the amount of money stolen through Outsider Enterprise scams, but the blog post notes that hundreds of people have lost some amount of money.

Google worked with AT&T, Verizon, and T-Mobile to block many of these malicious text messages, and Google notes that its on-device scam detection in Google Messages probably helped reduce the number of successful phishing attempts, too. This AI-powered feature apparently stops 10 billion scam texts every month, so it’s fair to expect it caught at least some Outsider Enterprise activity.

Laws for AI threats

Google has filed lawsuits against scammers before, but this is the first time it has taken direct action against a group alleged to be using Gemini as part of its scams. Google discusses the security measures it has baked into Gemini every time it announces a new model, but these can clash with the overarching need for chatbots to follow instructions and please users. And then you end up with thousands of scammers using Gemini to build fake websites.

In addition to its civil lawsuit, Google is assisting the FBI’s cybercrime division with a parallel criminal investigation. However, no one knows who’s behind Outsider Enterprise, and even if Google did have names, there’s little to be done when the perpetrators are in China. The company can go after fraudulent domains and Telegram accounts in hopes of disrupting the Outsider Enterprise operation, but the scams may simply change form.

Google believes that the era of AI calls for new approaches to law enforcement, so it’s taking this opportunity to renew its public support for a spate of legislation. The company has called out seven different potential federal laws, like the National Strategy for Combating Scams Act, the Strategic Task Force on Scam Prevention Act, and the AI Plan Act.

Most of the legislation Google promotes calls on one or more federal law enforcement agencies to set up task forces to counter the threat of AI-assisted scams and market manipulation. One of them (Artificial Intelligence Public Awareness and Education Campaign Act) is aimed at improving the public’s ability to spot malicious uses of AI. However, the industry-wide goal of attaining human-like intelligence in AI systems will only make this content harder for people to spot, even with all the well-meaning government legislation in the world.

Ryan Whitwam

Senior Technology Reporter

Ryan Whitwam

Senior Technology Reporter

Ryan Whitwam is a senior technology reporter at Ars Technica, covering the ways Google, AI, and mobile technology continue to change the world. Over his 20-year career, he's written for Android Police, ExtremeTech, Wirecutter, NY Times, and more. He has reviewed more phones than most people will ever own. You can follow him on Bluesky, where you will see photos of his dozens of mechanical keyboards.

9 Comments

Comments

Forum...