A nonspeaking person is the star of this opera. AI made that possible | Scientific American
SubscribeSign In
JOIN US ON THE MONARCH MIGRATION. LEARN MORE.<br>August 14, 2026<br>Add Us On GoogleAdd SciAm<br>The unlikely opera star: How AI helped a nonspeaking performer find his voice
What if artificial intelligence is used to amplify human creativity rather than replace it?
By Rachel Feltman, Sushmita Pathak & Alex Sugiura
Jill Steinberg
SUBSCRIBE TO Science Quickly<br>Apple | Spotify | YouTube | RSS
Rachel Feltman: For Scientific American’s Science Quickly, I’m Rachel Feltman. Today we’ll be hearing from a few folks, one of whom is a young man named Jakob Jordan.<br>Jakob Jordan: Hi, my name is Jakob.<br>Feltman: Jakob has autism and apraxia, which is a disorder of the brain and nervous system that keeps people from carrying out physical actions at their own command. In Jakob’s case, it can be difficult or even impossible to speak his thoughts aloud.<br>On supporting science journalism<br>If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.<br>So it might surprise you to learn that Jakob just starred in an opera called Sensorium Ex.<br>[CLIP from a song from the Sensorium Ex album entitled “What I Am.”]<br>Feltman: In a moment, you’ll hear Jakob using a custom-built AI speech synthesis device that made this opera sing—one that was specifically designed to sound like him and, crucially, to give him expressive control over his own voice.<br>Jordan: Hearing my voice produce my desired words feels so exhilarating. My mouth is constantly spouting off repetitive loops that I get stuck in. It is not my intended speech, so it can be frustrating when I can’t express my intentions. Sensorium Ex gave me the gift of hearing my trapped thoughts come to life. It’s like my voice was released from prison after a wrongful conviction. I’m so grateful to the Sensorium Ex team for making dreams that felt impossible come true.<br>Feltman: That sound bite was processed with the AI tech that Jakob was referring to, but later on we’ll hear how it contrasts with his more traditional text-to-speech device.<br>Along with Jakob, today we’ll be chatting with New York University professor Luke DuBois, who developed Jakob’s AI speech synthesis device, and composer Paola Prestini, who wrote the opera this tech was designed for. Their work together raises a radical question about AI: What if we built a tool not to replace human creativity but to amplify it?<br>Feltman: Paola, how did the idea of this opera first arise?<br>Paola Prestini: It started with a commission from an opera company that then didn’t end up participating in the final run. But they wanted me to do something in relation, actually, to blindness, and at that point I was very, you know, determined to only do something that I had lived experience in or collaborate with someone who had. And I was already working with this amazing poet Brenda Shaughnessy, and Brenda has lived experience as a mother to a nonverbal and nonambulatory child. And so I went back to the opera company, and I was like, “Listen, you know, this is something that we really feel we could, a story we could tell.”<br>And from there we refined it, and eventually my company VisionIntoArt really took over because we learned in taking this kind of intersectional approach to disability that we really had to create the whole framework, uh, you know, not just the opera.<br>Feltman: I’m really curious what the reaction was to your proposal to work with a different group.<br>Prestini: So I think what I can say with certainty is that a work like Sensorium Ex doesn’t really exist in the opera field. Even though the word “opera” means “to work” and historically was really the place of innovation, I think in contemporary society, it doesn’t exist that way anymore. It’s really about, you know, preserving the canon. And when it is done to commission new work, it generally doesn’t treat the stage as an R & D [research and development] lab, which is what we were essentially doing. And so I wanna be graceful in saying that, you know, my desire for the piece was to honor what was in the libretto—and the libretto was asking questions that I didn’t know how to answer without collaborating with someone, you know, like Luke or like Jakob. And to honor, also, the fact that the characters were written with certain disabilities, I wanted to really make sure that the context for disability would thrive. And so what I was being asked to do was not just create an opera but create a framework for success for the performers.<br>And so that’s when I learned, well, I have to be my own opera company, you know? Like, I have to collaborate with...