Search papers, labs, and topics across Lattice.
This paper introduces AudioCALM, a novel framework for universal audio generation that integrates speech, sound, and music generation through continuous autoregressive modeling. By employing a thin flow-matching head for next-token prediction and an Asymmetric Mixture-of-Modality-Experts architecture, AudioCALM effectively addresses the challenges of fidelity and attention alignment across diverse audio tasks. Experimental results show that AudioCALM not only matches the state-of-the-art in modality-specific tasks but also surpasses previous unified models, demonstrating its effectiveness in generating high-quality audio across various domains.
AudioCALM achieves state-of-the-art performance in speech, sound, and music generation by seamlessly integrating diverse audio modalities into a single autoregressive framework.
Unifying speech, sound, and music generation in one model is hindered by tradeoffs between fidelity, end-to-end training, in-context conditioning, and variable-length synthesis that no current paradigm fully resolves. To address this challenge, we present AudioCALM, a universal audio generation framework that extends autoregressive (AR) next-token prediction from discrete tokens to continuous audio latents: a thin flow-matching head replaces the softmax to predict rectified-flow velocities at each position, and a block-causal AR-Flow attention pattern produces arbitrary-length output. Joint training of multiple audio generation tasks faces an asymmetric text--audio mismatch: speech transcripts align to specific time spans and demand tight, time-aligned attention, whereas sound and music captions describe only overall semantics and rely on diffuse, holistic attention; mixing the two disproportionately degrades sound and music generation. We address this asymmetry at two levels: a data reformulation strategy that unifies all three tasks under a single description-style conditioning interface, and a novel architecture Asymmetric Mixture-of-Modality-Experts (A-MoME), which adds a dedicated residual expert for speech while sound and music share the backbone, incurring no inference overhead on non-speech inputs. Experimental results demonstrate that AudioCALM matches modality-specific state-of-the-art and outperforms prior unified baselines on speech, sound, and music generation benchmarks.
DAMO