New method aims to keep kids safe from illegal AI-generated content | MIT News | Massachusetts Institute of Technology
Skip to content ↓
Massachusetts Institute of Technology
Search websites, locations, and people
See More Results
Suggestions or feedback?
Enter keywords to search for news articles:
Submit
Browse By
Topics
View All →
Explore:
Machine learning
Sustainability
Startups
Black holes
Classes and programs
Departments
View All →
Explore:
Aeronautics and Astronautics
Brain and Cognitive Sciences
Architecture
Political Science
Mechanical Engineering
Centers, Labs, & Programs
View All →
Explore:
Abdul Latif Jameel Poverty Action Lab (J-PAL)
Picower Institute for Learning and Memory
Media Lab
Lincoln Laboratory
Schools
School of Architecture + Planning
School of Engineering
School of Humanities, Arts, and Social Sciences
Sloan School of Management
School of Science
MIT Schwarzman College of Computing
View all news coverage of MIT in the media →
Listen to audio content from MIT News →
Subscribe to MIT newsletter →
Close
Breadcrumb
MIT News
New method aims to keep kids safe from illegal AI-generated content
New method aims to keep kids safe from illegal AI-generated content
Researchers developed an auditing technique to test generative AI models for malicious capabilities, without prompting them for illegal outputs.
Adam Zewe<br>MIT News
Publication Date:
July 13, 2026
Press Inquiries
Press Contact:
Abby
Abazorius
Email:<br>abbya@mit.edu
Phone:<br>617-253-2709
MIT News Office
Media Download
↓ Download Image
Caption:
Researchers developed an evaluation procedure that tests generative AI models for harmful capabilities without generating outputs. This could enable auditors to identify open-source models that have been adapted to produce illegal content.
Credits:
Credit: Christine Daniloff, MIT; iStock
*Terms of Use:
Images for download on the MIT News office website are made available to non-commercial entities, press and the general public under a<br>Creative Commons Attribution Non-Commercial No Derivatives license.<br>You may not alter the images provided, other than to crop them to size. A credit line must be used when reproducing images; if one is not provided<br>below, credit the images to "MIT."
Close
Caption:
Researchers developed an evaluation procedure that tests generative AI models for harmful capabilities without generating outputs. This could enable auditors to identify open-source models that have been adapted to produce illegal content.
Credits:
Credit: Christine Daniloff, MIT; iStock
Previous image<br>Next image
With the exploding popularity of generative artificial intelligence, many open-source models are now available online for anyone to adapt for their task, such as generating product renderings in a certain artistic style.<br>But these models also find their way into the hands of nefarious actors who may optimize them to produce illegal content, like hate speech or child sexual abuse material (CSAM). This is a growing problem — the National Center for Missing and Exploited Children received more than 1.5 million reports of AI-generated CSAM in 2025, an increase from 67,000 in 2024.<br>Engineers usually test AI for harmful capabilities by prompting the model and inspecting its outputs, but this is impossible for CSAM, since it is illegal in the U.S to generate such content, regardless of intent.<br>To avoid this dilemma and improve AI safety, a team of MIT scientists, led by graduate student Vinith Suriyakumar and associate professors Ashia Wilson and Marzyeh Ghassemi, joined forces with researchers from Thorn to develop a new auditing approach that determines whether a model can produce CSAM, without prompting it. Thorn is a child safety nonprofit whose mission is to transform how children are protected from sexual abuse and exploitation in the digital age.<br>Their technique examines how the inner workings of a model have been adapted, but it never generates an output. By examining hidden representations, it can reliably infer whether a model has been specialized to produce harmful imagery.<br>When tested, the auditing procedure identified model variations that had been specialized to generate CSAM with 100 percent accuracy. A hosting platform could use this technique to flag unsafe models and quickly remove them or prevent them from being uploaded in the first place.<br>“This unlocks a new avenue for platforms that host open-source models and for law enforcement to actually test whether a model is capable of generating CSAM. Before, we had no way of measuring this. It was a huge blind spot that some people were taking advantage of. Now, we can address an AI safety problem that is having severe negative impacts,” says Vinith Suriyakumar, an MIT electrical engineering and computer science (EECS) graduate student and lead author of a paper on this technique.<br>Suriyakamur and Wilson, the Lister Borthers Career Develop Professor in EECS and a principal...