Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce INSV-A, an automated screening framework for low-resource, non-Latin-script TTS evaluation, which decomposes the evaluation into synthesis completion, ASR WER/CER, transcript Script Fidelity Rate, and audio language identification. They instantiate this framework as PashtoTTS-Bench, a benchmark for Pashto TTS, evaluating several TTS systems including Edge and OmniVoice models. Results show that OmniVoice auto achieves the lowest WER, but language identification models struggle to differentiate Pashto TTS outputs from Urdu, highlighting the challenges in evaluating TTS for such languages.
Current TTS evaluation metrics can be misleading for low-resource languages, as systems might "succeed" by speaking a neighboring language or producing unnatural audio that still yields a low WER.
Text-to-speech (TTS) evaluation for low-resource non-Latin-script languages can fail when it relies on a single ASR round-trip word error rate (WER). A system may produce no audio, speak a neighbouring language, preserve target script text only in an ASR transcript, or sound unnatural to native listeners. We introduce INSV (Intelligibility, Naturalness, Script fidelity, and Verification), a reporting framework that separates these cases. This paper reports INSV-A, the automated screening subset: synthesis completion, ASR WER/CER, transcript Script Fidelity Rate, and audio language identification. Native MOS and phonetic annotation are specified but not claimed in this release. We instantiate INSV-A as PashtoTTS-Bench, a dated benchmark for Pashto TTS. The April-May 2026 run evaluates Edge GulNawaz, Edge Latifa, OmniVoice clone, OmniVoice auto, and an Urdu negative control on 200 FLEURS and 200 filtered Common Voice 24 prompts. Under the independent omniASR_CTC_300M_v2, OmniVoice auto has the lowest WER (24.1% FLEURS, 27.4% CV24), followed by Edge GulNawaz (32.8%, 39.5%), Edge Latifa (35.6%, 47.7%), and OmniVoice clone (45.4%, 34.8%). WER below the natural-speech baseline reflects clean synthetic audio and should not be read as better than native speech. Whisper Large V3 returns 0.0% Pashto labels on checked Pashto TTS audio, while MMS-LID-4017 and SpeechBrain VoxLingua107 separate Pashto outputs from the Urdu control. The release provides provider metadata, per-sentence scores, LID audits, failure logs, and scripts for adding systems.