Pushing the Speed-Cost Frontier for Qwen3-TTS | Nari Labs

NEW POST: PUSHING THE SPEED-COST FRONTIER FOR QWEN3-TTS

Listen to this post

-->

TL;DR

Our Qwen3-TTS 1.7B CustomVoice implementation achieves 10 requests per second (RPS) and sub-50 ms p95 time-to-first-audio (TTFA) while maintaining real-time playback on a single NVIDIA H100 SXM.

×

We compare five implementations: ours, vLLM-Omni, SGLang-Omni△, VoxServe, and M*, under Poisson open-loop traffic. After tuning each implementation for low-latency streaming, ours is the only one to achieve sub-50 ms p95 TTFA . We maintain sub-50 ms p95 TTFA through 10 RPS and keep it below 100 ms even at 20 RPS .

Our system produces approximately 630 characters per second at 10 RPS. At $4.29 per hour for a 1× H100 SXM instance, this translates to ~$2 per 1M characters at full utilization1. For comparison, ElevenLabs V3 is $100 / 1M and Cartesia Sonic 3.5 is $49 / 1M at a higher TTFA.

We open source the implementation and benchmark. Our methodology is explained below.

Interested in custom deployments or want to optimize your multimodal inference workload?

Let’s chat

Defining “Real-time” TTS

Let’s start by discussing what a real-time TTS server needs to achieve. We think it’s a four-part problem:

Low Audible TTFA: Time from request dispatch to the first audible sample must be low.

Zero underruns: Once playback starts, the client must not run out of buffered audio.

Capacity: 1 and 2 must hold as RPS increases.

Non-malformed output: Speech must be intelligible.

We choose Qwen3-TTS CustomVoice 1.7B because it is one of the most popular TTS models with a permissive license.

Based on the above definition, we target low p95 audible TTFA with zero underruns while maintaining high RPS on a single NVIDIA H100 SXM.

All benchmarks run for five minutes under Poisson open-loop traffic to approximate real workloads, following Fireworks AI’s LLM benchmark. Each engine receives the complete text in a single HTTP request, while audio output remains streamed. We detect audible TTFA, reconstruct playback from received PCM, and evaluate the completed audio using Deepgram STT.

How Do Other Engines Perform?

The table below shows the upstream/default result at 1 RPS for each engine. We only apply changes for compatibility in this run.

Enginep95 audible TTFAp95 leading silenceRequests with underruns

vLLM-Omni277.883 ms90 ms100%<br>SGLang-Omni△1,140.69 ms80 ms0%<br>VoxServe315.064 ms30 ms0%<br>M*1,159.956 ms90 ms0%

These defaults have substantial room for improvement. We tune each serving engine for its own latency, continuity, quality, and capacity requirements.

1. Remove leading silence

The first PCM returned by a model can contain tens of milliseconds of silence before the first sustained sound. This gap pushes audible TTFA back like so:

We add a dynamic trim. It detects sustained speech from short RMS windows, removes samples before onset, and streams the remaining audio normally. This change improves TTFA by ~80ms but does not make model inference itself faster.

2. Tune frame accumulation

We also tune how many codec frames are collected before decoding and releasing an audio chunk.

Smaller initial chunks reduce TTFA, but provide less playback headroom and create more frequent decoder work. Larger chunks are easier to batch and make continuous playback safer, but delay the first audible output. A useful configuration therefore starts with a small chunk and increases the chunk size for later output.

The exact knobs differ by engine: vLLM-Omni exposes settings such as codec_chunk_frames and codec_chunk_ramp; the other engines provide equivalent chunk or stride controls. We iterate over these values to find the config that best matches: low p95 TTFA, zero underruns and stable behavior as load increases.

Performance after tuning existing serving engines

The following table shows the selected no-underrun profile for each engine after leading-silence and frame-accumulation tuning.

Enginep95 TTFA (1 RPS)p95 TTFA (6 RPS)

vLLM-Omni56.815 ms93.451 ms<br>SGLang-Omni△120.879 ms273.700 ms<br>VoxServe49.3 ms363.2 ms<br>M*104.035 ms179.501 ms

VoxServe reaches sub-50 ms p95 TTFA at 1 RPS, while the other three engines do not. By around 6 RPS, every engine is at roughly 100 ms p95 TTFA or higher2.

How We Optimized Qwen3-TTS

We first need to understand Qwen3-TTS architecture. It is a 3-part model performing hierarchical multi-codebook generation. The Talker predicts the first codebook token for each audio frame, the Code Predictor generates the remaining 15 codebook tokens, and the causal Codec converts codebook tokens into waveform samples.

Each module has its own compute profile, batching behavior, and latency requirements. Rather than optimizing each module in isolation, we focus on a broader question: how should a serving system coordinate these heterogeneous tasks?

1. Bringing three modules under one scheduler

Most Qwen3-TTS serving...