Search papers, labs, and topics across Lattice.
This paper introduces a vLLM-based inference pipeline that enhances both speech understanding and generation by integrating a novel approach to autoregressive decoding with delay-pattern de-interleaving and multi-stream sampling. The method effectively maintains high throughput while implementing Classifier-Free Guidance (CFG), achieving 80% of the throughput of non-CFG processes despite the added complexity of dual-request handling. By open-sourcing this framework, the authors provide a valuable resource for advancing multimodal audio applications in AI.
Achieving 80% throughput with Classifier-Free Guidance challenges the assumption that CFG drastically reduces efficiency in multimodal audio processing.
While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.
CMU ML