Search papers, labs, and topics across Lattice.
This paper introduces PRIME-Speech, a framework that maintains the capabilities of strong speech-to-text (S2T) models while enabling efficient speech-to-speech (S2S) generation. By utilizing a frozen backbone and synchronizing a causal audio post-decoder with the model's intermediate hidden states, PRIME-Speech effectively generates codec tokens based on evolving reasoning rather than completed text. The approach demonstrates that it can produce accurate spoken responses with low word error rates (WER) across various tasks, preserving the original S2T performance without the need for extensive multi-turn training data.
PRIME-Speech achieves low-latency, accurate speech-to-speech generation without sacrificing the robust performance of existing speech-to-text models.
Strong speech-to-text (S2T) LLMs already provide robust speech perception and text reasoning, but adding speech-to-speech (S2S) output is challenging: fine-tuning the backbone can degrade the original S2T performance, while attaching a downstream talker reintroduces a serial text-to-speech bottleneck. We present PRIME-Speech, a frozen-backbone S2S conversion framework that trains only speech-generation modules. PRIME-Speech synchronizes a causal audio post-decoder with intermediate hidden states of the frozen backbone, so codec tokens are generated from the model's evolving reasoning trajectory rather than from completed text chunks. The post-decoder uses mixed hidden-state, text, and audio-history conditioning, and a training-time packing strategy with turn-level audio KV-cache and position reset stabilizes multi-turn spoken interaction without additional multi-turn S2S training data. Multi-token prediction further reduces the effective codec prediction rate and improves first-audio latency without modifying the reasoning path. Across speech translation, spoken QA, speech understanding, and multi-turn dialogue, PRIME-Speech preserves the S2T behavior of the frozen backbone while producing accurate, low-WER spoken responses.
Microsoft Research