Search papers, labs, and topics across Lattice.
This paper introduces PreferenceASR, a novel test set designed to benchmark automatic speech recognition (ASR) systems based on their adherence to user preferences in output style, addressing the limitations of existing ASR benchmarks that fail to account for format distinctions. The authors employ a two-stage LLM-assisted pipeline for dataset creation, supplemented by human verification, and evaluate ASR models using a preference-aware normalizer that selectively applies normalization steps based on user instructions. Results reveal that model rankings vary significantly across different preference types, highlighting quality differences that traditional evaluation methods overlook, and the dataset is made publicly available for further research.
ASR models can exhibit drastically different performance depending on user preferences, revealing hidden quality disparities in traditional benchmarks.
Popular ASR test sets adopt inconsistent conventions for numbers, disfluencies, entities, and casing, while standard normalizers erase the format distinctions users care about. Current benchmarks therefore cannot measure whether a model follows user preferences for output style. We introduce PreferenceASR, a test set evaluating ASR systems on their ability to follow natural-language preference instructions across four categories: normalization, entities, disfluencies, and case. Built from seven open-source corpora via a two-stage LLM-assisted pipeline with human verification, it is evaluated with a preference-aware normalizer that selectively skips steps matching the active instruction. Benchmarking four models shows rankings shift across preference types, exposing quality differences traditional evaluation obscures. We publicly release the dataset.
NVIDIA