Choosing a real-time voice AI stack when your language barely exists in the training data

KamalG

SubscribeSign in

Choosing a real-time voice AI stack when your language barely exists in the training data<br>OpenAI Realtime vs Gemini Live vs the pipeline you build yourself — evaluated under Azerbaijani, generalized to any low-resource language.<br>KamalG<br>Jul 06, 2026

Share

The demo works until you switch the language

Every real-time voice AI demo you’ve seen runs in English. The latency is magic, the turn-taking feels human, and the vendor’s pricing page has a calculator that makes your CFO happy. Then you change one parameter — the language your users actually speak — and the entire stack evaluation you did last month is worthless.<br>I found this out building an AI interview platform for a financial holding in Azerbaijan. Azerbaijani has around 24 million speakers, a Latin script, and clean Turkic morphology. It is not an obscure language. It is, however, a low-resource language in the only sense that matters here: the amount of transcribed speech data in the training corpora of every major STT and speech-to-speech model.<br>That single fact reshapes the whole architecture decision:<br>Speech-to-speech models (OpenAI Realtime, Gemini Live) give you the best latency and the most natural conversation — but you’re locked into whatever language coverage the model has, and you can’t swap components when it doesn’t.

Cascaded pipelines (LiveKit, Pipecat, Vapi with custom providers) let you plug in any STT/TTS you can find — but now you own the latency budget, the turn-detection logic, and the integration of whatever Azerbaijani speech models exist.

In high-resource languages this is a preference. In low-resource languages it’s a fork: one path may simply not work at all, and the other may work badly in ways you only discover in production, with a real candidate on a real phone line.

This post is the evaluation I wish had existed before I started: what actually breaks, in which stack, and the decision framework that generalizes to any language the demos weren’t made for.<br>The speech-to-speech path — great, until it isn’t

Start with the option everyone wants to work: a single model that takes audio in and produces audio out. OpenAI Realtime and Gemini Live. No pipeline to assemble, sub-second responses, natural interruption handling for free.<br>The appeal is real. In English, both are the closest thing to a solved problem this space has. So the first question isn’t “which is better” — it’s “does either one actually speak Azerbaijani, or does it just claim to?”<br>There are distinct failure levels, and vendor docs only tell you about the first:<br>Not listed. The language isn’t on the support page. Easy — you’re done.

Listed but degraded. The language is “supported,” and comprehension is fine on clean, slow speech — then falls apart on phone-quality audio, regional accents, code-switching into Russian mid-sentence (which every real Azerbaijani speaker does), or domain vocabulary. This is the level that kills you in production, because your demo will pass.

Asymmetric. The model understands the language but speaks it with a foreign accent, wrong stress patterns, or drifting phonology. For an interview platform, a robot that sounds foreign is a credibility problem with candidates.

What I found:<br>OpenAI Realtime: Failed at level 3. Comprehension was workable, but the spoken Azerbaijani came out with a distorted accent — wrong phonology, wrong stress. For casual use, tolerable. For a system conducting job interviews at a financial institution, disqualifying: candidates hear the accent in the first sentence, and it sets the tone for everything after. There’s no voice tuning or pronunciation control that fixes this — the failure is in the model, and the model isn’t yours.

Gemini Live: The surprise. Comprehension and output quality were genuinely good — it passed all three levels. What killed it was a fourth failure mode my own taxonomy missed: latency . Response delays were long enough to break conversational rhythm. In a chat interface that’s a spinner; in voice it’s the candidate wondering whether the call dropped, talking over the model, and the turn-taking collapsing from there. Real-time voice has a hard perceptual budget — cross it and quality elsewhere stops mattering.

So the scoreboard for speech-to-speech and Azerbaijani, mid-2026: one model that speaks the language badly, and one that speaks it well but too slowly for conversation. Neither failure is fixable from my side of the API. That’s the structural bet you make with speech-to-speech — when it works, it’s the best option available; when it misses your language on any dimension, you’re waiting on someone else’s roadmap.<br>Which forces you down the other path: assembling the pipeline yourself.<br>The cascaded path — you own every millisecond

If speech-to-speech fails your language, the fallback is the classic pipeline: voice activity detection → speech-to-text → LLM →...