ttsproof · PyPI

Skip to main content<br>Switch to mobile version

Warning

You are using an unsupported browser, upgrade to a newer version.

Warning

Some features may not work without JavaScript. Please try enabling it if you encounter problems.

Search PyPI

Search

ttsproof 0.3.1

Automated failure-mode QA for text-to-speech systems: structural audio checks, equivalence-aware WER/CER, and ASR-uncertainty quarantine.

pip install ttsproof

Copy PIP instructions

TTSProof

Automated failure-mode QA for text-to-speech systems.

Your TTS pipeline can produce a clip that is empty, half-silent, clipped, stuck<br>in a loop, or three times longer than it should be — and a WER score alone will<br>miss most of it, while plain WER also fails perfectly good audio because the<br>input said 3:30 PM and the transcript said three thirty pee em.

TTSProof runs the checks that catch what actually breaks:

Structural audio checks (no model needed): empty/truncated audio,<br>duration explosions, long internal silences, clipping, repeated-chunk loop<br>detection, end-of-clip artifacts. Just numpy + soundfile.

Equivalence-aware WER/CER: expected text and ASR transcript are both<br>canonicalized to spoken form (numbers, decimals, dates, clock times,<br>acronyms, single letters) before scoring — so formatting differences don't<br>count as pronunciation errors.

ASR-uncertainty quarantine: when audio is structurally clean but ASR<br>disagrees on a very short utterance (a letter, an acronym, "ahh"), the sample<br>is quarantined for human review instead of counted as a failure — because at<br>that length, the ASR is as likely to be wrong as the TTS.

The method was evaluated on a production TTS service — 130 edge cases × 3<br>voices (390 samples), with a blinded human validation of the quarantine zone —<br>and published as a citable technical report:

An Automated Failure-Mode QA Framework for Neural Text-to-Speech Systems<br>DOI: 10.5281/zenodo.20757553 (CC-BY-4.0)

Install

pip install ttsproof # structural checks + metrics + benchmark corpus<br>pip install "ttsproof[asr]" # + faster-whisper for pronunciation gating

Benchmark any TTS engine in one command

TTSProof ships a built-in corpus of 817 curated edge cases across 39<br>categories — numbers, decimals, currencies, dates, ISO timestamps, clock<br>times, time zones, phone numbers, URLs, emails, IP/MAC addresses, file paths,<br>Roman numerals, ordinals, units, abbreviations, acronyms, single letters,<br>pronunciation torture words (Worcestershire, synecdoche, colonel…),<br>proper names (Reykjavík, Nguyễn, Tchaikovsky…), scientific/medical<br>vocabulary, tongue twisters, homographs, Greek, Norwegian, mixed-language<br>lines, math, punctuation abuse, hallucination traps, emoji, SQL/JSON/markup,<br>and more.

The corpus is versioned independently of the software (this release:<br>Benchmark Corpus 1.0 ) — published scores stay comparable across tool<br>updates, and every report records both versions:

# your engine as a command template ({text} in, {out} wav path out):<br>ttsproof benchmark --cmd "mytts --text {text} --wav {out}"

# or score audio you already generated (files named .wav):<br>ttsproof generate --out cases.jsonl # export the corpus, synthesize it your way<br>ttsproof benchmark --wav-dir ./my_audio

You get a category scoreboard in the terminal…

numbers 98.3% 59/60 decided<br>dates 96.7% 29/30 decided<br>urls 88.9% 8/9 decided (+0 quarantined)<br>norwegian 95.0% 19/20 decided<br>OVERALL 96.1% pass=485 fail=20 quarantine=23

…plus report.html — a self-contained page with score bars, every failure's<br>waveform, an audio player, and what the ASR actually heard.

Each category is scored by an honest policy : strict (unambiguous spoken<br>form — equivalence-aware WER), keywords (URLs/currencies have many valid<br>readings — key tokens must survive the round trip), or structural (emoji and<br>punctuation storms have no meaningful transcript — the audio just has to<br>survive). No fake failures from formatting differences.

CI regression gate:

ttsproof regress baseline/report.json current/report.json --tolerance 1.0<br># exit 1 + category-level diff when quality drops:<br># REGRESSION DETECTED:<br># OVERALL: 96.2% -> 94.7% (-1.5 pp)<br># numbers: 99.1% -> 95.0% (-4.1 pp)

Compare engines:

ttsproof compare xtts/report.json fish/report.json kokoro/report.json

Testing closed-source models (ElevenLabs, OpenAI, …) via SpeechSDK

You can benchmark commercial/closed-source models using SpeechSDK. See our SpeechSDK integration example for a ready-to-use wrapper script.

Integration suggested by u/pmarks98 (Jellypod / SpeechSDK).

Quickstart

Check one file (CLI):

ttsproof check output.wav --text "Hello there"

QA a folder of generated audio against a manifest:

# cases.jsonl — one case per line:<br># {"id": "case_001", "text": "Meet me at 3:30 PM", "wav": "case_001.wav"}<br>ttsproof run --manifest cases.jsonl --wav-dir ./audio --out ./reports --asr

You get report.csv + report.json with one verdict per sample:<br>pass / hard_fail / quarantine.

Gate any TTS system in CI (Python):

import ttsproof

def...