Search papers, labs, and topics across Lattice.
FlexiSLM introduces a novel approach to spoken language modeling by utilizing dynamic and controllable frame rates for both speech input and output, addressing the limitations of fixed frame rates in existing models. This method leverages recent advancements in dynamic frame rate speech coding, allowing for significant reductions in inference time while maintaining high-quality speech generation. Experimental results show that FlexiSLM outperforms existing 7B models at high-quality settings and can effectively operate at frame rates as low as 4.0 Hz, halving inference time at 6.25 Hz compared to traditional 12.5 Hz models.
FlexiSLM can operate at frame rates as low as 4.0 Hz while maintaining high-quality speech, effectively halving inference time compared to traditional models.
Spoken language models (SLMs) extend LLMs to speech input and output. Existing SLMs represent speech at fixed frame rates (e.g., 25 or 12.5 Hz), ignoring the time-varying information density of speech and offering no flexibility to trade off quality for speed at inference time. Recent audio tokenizer research has proposed dynamic frame rate speech coding, which exploits this non-uniformity and enables two new capabilities: very low average frame rates and frame rate controllability. However, this technique has not yet been applied to SLMs. We introduce Flexible Spoken Language Model (FlexiSLM), the first SLM that supports dynamic and controllable frame rates on both speech input and output. Using dynamic frame rate representations, FlexiSLM outperforms fixed-frame-rate 7B models including Qwen2.5-Omni and Kimi-Audio at its high-quality operating points. We further verify that FlexiSLM can be accurately steered down to 4.0 Hz; at 6.25 Hz, it roughly halves inference time relative to 12.5 Hz while retaining strong speech-to-speech quality. Audio samples are available at https://flexislm.github.io .
