Can AI Read the Room? USC Study Finds AI Is Better at Reading Than Listening - USC Viterbi | School of Engineering

Can AI Read the Room? (Credit: Midjourney)

Today, users can do more than exchange text with artificial intelligence (AI).

From holding conversations and asking questions aloud to sending voice messages, AI can now process audios and respond in real time.

But can AI really listen as well as it can read?

A new study led by USC researchers suggests the answer is no. The team found that even today’s most advanced audio large language models (LLMs) struggle to interpret information beyond the spoken words, often taking language too literally and missing the nonverbal cues that have an equally important role in human communication.

In everyday conversations, meaning extends far beyond words. Tone of voice, emotion, emphasis, pitch and other paralinguistic information provide important social and emotional context that helps people interpret what someone truly means.

USC professor Mohammad Soleymani led the research project to uncover why audio LLMs struggle to interpret these listening cues and developed new techniques that significantly improve their ability to understand the "how" of speech—not just the spoken words.

The project, which began last August, resulted in the paper, "Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox," which was accepted to the International Conference on Machine Learning (ICML) 2026.

Soleymani is a research associate professor at USC Viterbi School of Engineering and the USC Mark and Mary Stevens School of Computing and AI‘s Thomas Lord Department of Computer Science, with joint appointment at the USC Institute for Creative Technologies. He also leads the USC Intelligent Human Perception (IHP) Lab.

The research team also included Soleymani’s PhD students Ashutosh Chaubey, and Jiacheng Pang, a former master’s student from his lab.

Why AI Models Struggle to Listen

Audio LLMs are multimodal AI models that can process not only text but also audio inputs, such as speech and environmental sounds. Embedded in conversational AI tools like ChatGPT and Gemini, Audio LLMs allow users to speak directly to AI, receive spoken responses in real time and analyze uploaded audio recordings.

Audio LLMs work by converting speech into numerical representations, known as vectors, that are translated into a format the model’s internal "language brain" can understand.

While these models are designed to be multimodal and process both spoken content and acoustic information, they suffer from a strong language bias, "treating text as a first-class citizen and audio cues as secondary," Soleymani explained.

Because the models are so focused on language, they often ignore the acoustic reality of an audio clip.

As a result, audio LLMs excel at transcribing what is said, accurately converting spoken words into text, but often struggle to understand how something is said—such as the speaker’s tone, emotion or pitch, making them functionally "tone-deaf."

For example, if a speaker sounds clearly happy but says the words, "I am sad," the model will incorrectly conclude that the speaker is sad.

This can lead to misinterpretations of a speaker’s intent, emotional state and social context. In sensitive applications such as health assessments or human-AI interactions, failing to "hear" a user’s true emotion or condition could result in incorrect or even harmful responses.

Testing AI With Contradictory Audio Tasks

After identifying this blind spot in Audio LLMs, Soleymani’s team designed a series of "stress tests" to evaluate how well the models understood paralinguistic information. The researchers introduced a benchmark called VoxParadox, which evaluates models across 10 different paralinguistic tasks. These tasks include biometric and identity-related challenges, such as estimating a speaker’s age or gender and counting how many people are speaking, as well as prosodic and acoustic tasks, including identifying emotion, intonation, pitch and volume.

For each task, the researchers presented the models with audio clips where spoken words intentionally contradicted the accompanying audio content and asked them to determine whether the uttered words were true or false.

If the model answered incorrectly, it suggested the model was prioritizing the text over the audio. If it answered correctly, it demonstrated that the model had incorporated the nonverbal audio information into its reasoning.

The team evaluated NVIDIA’s Audio Flamingo 3 and Alibaba’s Qwen2-Audio models and found that both struggled to accurately interpret paralinguistic information.

Looking Inside AI’s Brain

Soleymani’s team then set out to understand why Audio LLMs fail at interpreting paralinguistic information by probing the models’ internal layers—almost like performing "brain surgery" on AI.

By examining the models’ internal computations in real time, the researchers...