Voice Orchestration Benchmarks by Cekura
Voice Orchestration Benchmarks<br>Everyone benchmarks the model. We benchmark the stack around it.<br>An independent benchmark isolating the orchestration layer: turn-taking, interruptions, latency and tool execution. One byte-identical agent, run across 6 platforms. The differences are the platforms, not the model. Note: these are platform defaults, expect higher peaks once you tune to your use case.
Performance vs latency<br>75%80%85%90%95%100%1.60s1.85s2.10s2.35s2.60s2.85s3.10sP50 turn latency · faster ←pass^3 (%) · better ↑↖ upper-left = bestRetell: 96.6% pass^3 at 1.96sRetellVapi: 94.9% pass^3 at 2.34sVapiPipecat: 89.8% pass^3 at 3.15sPipecatLiveKit: 84.7% pass^3 at 2.46sLiveKitSynthflow: 81.4% pass^3 at 3.16sSynthflowElevenLabs: 76.3% pass^3 at 1.73sElevenLabsFigures are comparisons under one fixed harness · see methodology.
Highest reliability<br>Retell96.6%<br>pass^3
Fastest responses<br>ElevenLabs1.73s<br>median turn latency
Best interruption handling<br>Pipecat4.90<br>out of 5 · interruption score
Most consistent latency<br>Vapi1.66s-2.95s<br>P5-P95 turn latency
By metric<br>Insights<br>1s2s3s4s5sElevenLabs: p50 1730ms · 1125 turns (p5 1090 · p25 1290 · p75 2260 · p95 3194)ElevenLabs1.73sRetell: p50 1960ms · 1143 turns (p5 1230 · p25 1525 · p75 2480 · p95 3789)Retell1.96sVapi: p50 2340ms · 1205 turns (p5 1660 · p25 2070 · p75 2550 · p95 2950)Vapi2.34sLiveKit: p50 2460ms · 1053 turns (p5 1910 · p25 2130 · p75 3040 · p95 3870)LiveKit2.46sPipecat: p50 3150ms · 1218 turns (p5 1860 · p25 2242 · p75 3660 · p95 5419)Pipecat3.15sSynthflow: p50 3160ms · 1569 turns (p5 1960 · p25 2470 · p75 3910 · p95 5080)Synthflow3.16sPer-turn latency · line = median (P50), box P25–P75, whiskers P5–P95 · ~1,100–1,570 turns/platform
Latency<br>Per-turn response latency<br>With the model, prompt and TTS fixed, per-turn latency reflects the platform’s turn-taking: endpointing, VAD, buffering and network path. ElevenLabs is fastest (median 1.73 s per turn) but carries a long tail (P95 3.19 s, max 10.4 s). Vapi is the most consistent (P5–P95 1.66–2.95 s). LiveKit now sits near Vapi at 2.46 s median; Pipecat and Synthflow remain slower, around 3.15 s.
Average interruption score (0–5) · interruption scenarios only
Interruption<br>Interruption handling<br>Turn-taking is set by the platform, not the prompt: Voice Activity Detection and end-of-turn detection decide when the agent yields. Across the interruption scenarios (barge-in, cough, mid-turn silence), Pipecat scores highest at 4.90, with LiveKit close at 4.89 . Vapi is lowest at 4.63, and the field is tight overall.<br>See example →<br>How Interruption Score is determined⌄Cekura uses voice-activity detection on the stereo recording to find every point where the Main Agent starts speaking while the caller is still talking, then scores 5 × (1 − interruptions ÷ turns), clamped to 0–5, so 5/5 means it never talked over the caller. We report it on the interruption scenarios (those that inject barge-in, coughs, or mid-turn silence) because a whole-suite average is diluted by calls with nothing to interrupt.
Appropriate end-call rate (% of calls)
Call handling<br>Appropriate end call<br>Whether a call ends cleanly is handled by the platform, not the model. Retell ended every call correctly ; Vapi, Pipecat and LiveKit were next at 99.4%. Synthflow was lowest at 94.9%, usually failing to close after an emergency redirect or a late second request. Native end-call was disabled; ending was specified in the prompt.<br>See example →<br>How Appropriate End Call is determined⌄A per-call pass/fail from an LLM judge that reads the transcript and the reason the call ended, and decides whether the Main Agent wrapped up properly: resolving the request before hanging up, and not looping past a clean goodbye. Native end-call was disabled for the benchmark, so this reflects how each platform executes a prompt-driven hangup rather than a built-in one.
Average repetition score (0–5)
Conversation<br>Repetition<br>Loops surface when a platform mishandles silence or overlapping speech: a turn-taking failure, not a prompt one. Scores run 4.61 to 4.74, Retell highest. The metric sits near its ceiling, so read the low tail, not the mean.<br>See example →<br>How Repetition Score is determined⌄An LLM flags each time the Main Agent re-confirms the same, unchanged information twice or more; the score is 5 × (1 − repetitions ÷ turns), clamped to 0–5, so 5/5 means no unnecessary repetition. Most calls sit near the ceiling; the signal is the low tail, where a platform loops instead of recovering with one clean re-ask or a clean close.
Average voice-tone score (0–5)
Experience<br>Voice tone<br>Audio quality is set by the platform’s media pipeline (encoding, jitter, packet handling), not the model. ElevenLabs is highest at 4.60/5 ; Retell, Synthflow and Pipecat cluster from 4.45 to 4.51, while Vapi and the newer LiveKit run land at 4.20 and 4.04.<br>See example →<br>How Voice Tone is determined⌄A dedicated ML model analyzes the Main Agent's audio...