Search papers, labs, and topics across Lattice.
This paper challenges the assumption that low Word Error Rate (WER) in speech language models (SLMs) equates to high-quality representations suitable for both understanding and generation. They introduce a dynamic compression tokenizer that achieves ultra-low frame rates with minimal WER, enabling the isolation of "pure" semantic tokens. Experiments conditioning generative models with these tokens reveal severe articulation blur and unintelligible speech, demonstrating the orthogonality of semantic categorization (low WER) and the continuous phonetic trajectories needed for synthesis.
Low Word Error Rate can be a mirage: compressing speech to "pure" semantic tokens, even with near-perfect WER, produces unintelligible speech when used for generation.
The pursuit of a"unified"discrete token for both speech understanding and generation has led the Speech Language Model (SLM) community to heavily rely on Word Error Rate (WER) -- the core metric for Whisper-style tokenizers -- as the definitive proxy for representation quality. This fosters the assumption that low-WER tokens inherently preserve the information necessary for intelligible acoustic synthesis. We argue this is fundamentally deceptive. While high-frequency tokens succeed in generation tasks due to implicit information leakage, isolating pure semantic information at ultra-low frame rates strips away the finegrained articulation and micro-dynamics essential for ODE-based generation. Empirically validating this requires extreme compression without sacrificing WER -- a methodological bottleneck, as standard fixed-stride downsampling arbitrarily truncates phonetic boundaries. To overcome this, we develop a dynamic compression tokenizer that intelligently aligns representations with semantic boundaries, achieving ultra-low frame rates with exceptionally low WER. Using these isolated"pure"semantic tokens, we expose the WER trap: when conditioning generative models -- even with oracle duration alignments -- the reconstructed speech suffers from severe articulation blur and is rendered acoustically unintelligible. Our findings demonstrate that semantic categorization rewarded by low WER is inherently orthogonal to the continuous phonetic trajectories required for synthesis, shattering the illusion of the unified token and advocating for explicitly decoupled speech representations.
