Search papers, labs, and topics across Lattice.
This paper introduces SpeechKV, a novel approach that compresses the key-value (KV) cache of speech tokens within Speech LLMs, addressing the inefficiencies caused by lengthy speech sequences during autoregressive decoding. By applying learned pooling techniques, SpeechKV maintains fine-grained information while achieving compression to text-level granularity, resulting in significant performance improvements. The method was trained on 71K hours of speech data, yielding a 6.6% relative gain in out-of-domain entity recognition and a 2.3% gain on OpenASR, alongside a decoding speedup of at least 1.49 times that scales with audio length.
Compressing the KV cache of speech tokens can enhance decoding speed by over 1.49 times while improving performance on key benchmarks.
Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.
