Search papers, labs, and topics across Lattice.
The paper introduces LoSATok, a low-dimensional audio tokenizer designed to unify audio understanding and generation by compressing high-dimensional semantic features into a 128-dimensional latent space using a semantic bottleneck. This bottleneck is regularized by a novel time-relation loss to maintain temporal feature consistency, and a dual-level semantic supervision method is used to capture both semantic and acoustic details. Experiments across speech, music, and general audio domains demonstrate that LoSATok maintains competitive understanding performance while significantly improving Diffusion Transformer (DiT) modeling performance for audio generation.
Compressing audio semantics into just 128 dimensions doesn't just reduce DiT modeling burden; it actually *improves* audio generation quality across diverse domains.
Audio tokenizers are fundamental to unifying audio understanding and generation. Understanding requires high-level semantics, while generation demands semantic and acoustic details. Existing unified tokenizers jointly encode both in high-dimensional continuous latents, which increases the modeling burden of Diffusion Transformers (DiTs) for generation. We propose LoSATok, a low-dimensional audio tokenizer for cross-domain audio understanding and generation. Motivated by the observation that 1280-dimensional semantic encoder features are compressible, we introduce a Semantic Bottleneck that compresses them into 128 dimensions, regularized by the proposed time-relation loss for temporal feature consistency. We further design a dual-level semantic supervision method that leverages both high- and low-dimensional semantic signals, enabling the tokenizer to jointly capture semantics and acoustic details within a compact latent space. Experiments on speech, music, and general audio show that SemBo preserves strong low-dimensional semantic capacity and LoSATok retains competitive understanding performance compared with several semantic representations, while consistently improving DiT modeling performance on speech, music, and audio generation. These results demonstrate that LoSATok's low-dimensional representations can effectively support audio understanding and generation. Our code is provided at https://github.com/wxzyd123/LoSATok.
Tsinghua AI