Search papers, labs, and topics across Lattice.
This paper introduces STAR-VAE, a novel approach that addresses the Rate-Distortion-Regularity Trilemma in continuous VAEs by implementing Structured Topology-Aware Regularization (STAR), which reshapes latent space geometry to better accommodate the hierarchical nature of audio data. By imposing a growth-based constraint field, STAR effectively routes structural and textural information into appropriately matched channel subspaces, enhancing both reconstruction fidelity and semantic information preservation. Experimental results demonstrate that STAR-VAE outperforms existing methods in audio reconstruction and generation, particularly in text-to-audio tasks, while maintaining linear complexity in global context modeling.
STAR-VAE achieves state-of-the-art audio reconstruction fidelity by aligning latent space geometry with the hierarchical structure of audio signals.
Continuous Variational Autoencoders (VAEs) serve as the fundamental continuous tokenizer for modern neural audio generation systems, enabling high-fidelity reconstruction while providing a compact, smooth latent space for downstream generative priors. However, continuous VAEs face a fundamental conflict among compression rate, reconstruction fidelity, and latent space topology, which we formalize as the Rate-Distortion-Regularity Trilemma. This trilemma stems from a topological mismatch: the isotropic Gaussian prior in standard VAEs imposes a flat latent geometry that fails to accommodate audio's hierarchical nature, where low-frequency components are structured and compressible while high-frequency components are stochastic and incompressible, leading to disordered information packing in which crucial semantic features are interleaved with high-entropy noise. To address this challenge, we propose Structured Topology-Aware Regularization (STAR), a general training strategy that reshapes latent space geometry by imposing a growth-based constraint field, routing structural and textural information into channel subspaces with matching capacities. STAR is applicable to any VAE architecture and effectively resolves the trilemma, as demonstrated in CNN-based VAEs. We further present STAR-VAE, which combines STAR with a hybrid CNN-Mamba architecture for local feature extraction and linear-complexity global context modeling, and STAR-Gen, an LLM-based Flow Matching framework that leverages STAR-VAE's structured latent space for high-fidelity generation without vector quantization artifacts. Experiments across diverse audio domains show that STAR-VAE achieves state-of-the-art reconstruction fidelity and enhanced semantic information preservation, while the structured latent space improves both traditional diffusion models and STAR-Gen for text-to-audio generation.
DAMO