Silent speech with ultrasound — Aleph

AlephSilent speech with ultrasound<br>Vadims Casecnikovs✉, Gimran Abdullin, Raffi Hotter, Lev Chizhov<br>July 7, 2026<br>For comparison, lip-reading achieves 12.5% word error rate on a 1M hour dataset. We're excited that our system approaches existing methods despite being an early investigation trained on a 50-hour dataset and done in just a month.<br>We place an ultrasound probe behind the chin and capture videos like this of the tongue:<br>Your browser does not support the video tag.

And then we turn them into words. Here's a quick demo:<br>Your browser does not support the video tag.

A few days ago, we found out that the model also generalizes to new people (as long as they speak with an American accent 🇺🇸🦅11Our Eastern European friends were less thrilled.). Our friends were able to walk in, pick up a probe, and start using the system right away. We did not expect this to work with so little data.<br>Your browser does not support the video tag.

Speaking silently<br>Speech is roughly four times faster than typing, making it one of the fastest ways to communicate with a computer. As AI gets better at understanding natural language, we're beginning to interact with computers the way science fiction always imagined: by simply talking to them. We're already seeing the beginning of this with tools like Wispr Flow and ChatGPT voice.<br>But regular speech has one major limitation: most of the time, we're around other people. We're sitting next to coworkers and strangers in coffee shops or on the subway, and we don't want other people overhearing our conversations with our computers.<br>Just as earphones made listening private, silent speech could make speaking private.<br>There are several ways of detecting silent speech: EMG, radar, lip reading, etc.22Andy Matuschak has a great brief review here and on his Patreon. What's special about ultrasound is that you can see the tongue directly and cleanly in the image — no need to infer what's going on from noisy indirect measurements like EMG or radar. And it enables invisible silent speech, where, in principle, you wouldn't even be able to tell that someone is speaking silently.<br>Moreover, the tongue carries so much information about speech. The tongue forms around 34 distinct phoneme classes across the 40 English phonemes,33Some phoneme pairs like /t/ and /d/ or /s/ and /z/ differ mainly in voicing rather than tongue position, but they still produce subtle differences in the root of the tongue that ultrasound can detect. compared with only about 10–14 visually distinguishable lip shapes.

A submental ultrasound pulse propagating up through the floor of the mouth into the tongue. Drag to rotate.<br>Data & Infra<br>We collected our own dataset of ultrasound tongue imaging with silent speech.<br>To train a good model, the data has to satisfy two conditions: (1) an ultrasound recording should actually show the tongue, and (2) a person should actually say the text they were supposed to say. Both were surprisingly challenging, as people got tired, mumbled, and didn't hold their probes correctly.<br>When setting up data collection, the first question was: should people speak silently or out loud?<br>We ultimately care about decoding silent speech, but audible speech allows us to easily check that people said the words properly. By looking at tongue recordings, we noticed that silent and vocalized speech show similar tongue movements. That led us to believe that we could collect vocalized ultrasound speech data, use the audio to quality-check the data, and still train a model that generalizes to silent speech.<br>To quality-check the vocalized data, we asked people to speak audibly, transcribed the audio, and checked whether it matched the sentence they were supposed to read. We also used a real-time ultrasound quality classifier to detect poor probe positioning or poor coupling, and had a supervisor with a real-time dashboard monitoring the sessions.<br>We collected 50 hours of data of people reading synthetically generated short stories out loud.

We used stories instead of isolated phrases because they are easier to read and produce more natural speech. People can stay in the flow of reading, while we still control what appears in the data: broad vocabulary, different sentence structures, and hard cases like articles and contractions.<br>Training<br>The task is simple to describe but hard to solve: given an ultrasound video of the tongue, predict the spoken text.<br>Instead of training everything from scratch, we started with models that already knew something useful: ResNet-18 2+1d for video and Whisper Base for decoding speech-like embeddings into text.<br>Whisper is a strong speech-to-text model. We found its decoder useful as it is a pretrained model that can convert speech embeddings into text. We just needed to transfer our video embeddings into the embedding space of Whisper. To do that, we trained the tongue-video encoder's embeddings to be as close as possible to the embeddings of...