Search papers, labs, and topics across Lattice.
This paper investigates the latent space properties of FSQ tokenization within the continuous diffusion framework for generating categorical data, specifically focusing on text-to-speech applications. By analyzing Kullback-Leibler divergence on diffusion path measures, the authors demonstrate that FSQ tokens optimize the structure of the latent space, leading to superior performance in token prediction accuracy. The empirical results reveal that FSQ-based text-to-speech models outperform traditional LLM-based approaches, achieving better performance while being smaller and faster.
FSQ tokenization not only optimizes latent space for continuous diffusion but also outperforms LLM-based text-to-speech models in both efficiency and effectiveness.
Continuous diffusion for categorical data is a framework belonging to the diffusion family and aiming at generating discrete data. The scientific interest to such models has been constantly increasing these days because researchers try to achieve a challenging goal of finding reasonable alternatives to autoregressive large language models. In this paper, we study the properties of the structure of the latent space corresponding to discrete tokens expressed in terms of Kullback-Leibler divergence on diffusion path measures and accuracy of the correct token prediction by the optimally trained diffusion model. We find that FSQ tokenization scheme has the latent space structure with the properties that make it best suited for continuous diffusion for categorical data as verified through rigorous theoretical analysis and numerical experiments. To validate our findings in real-life scenario, we train several text-to-speech diffusion models having speech tokens as intermediate acoustic features, and show that the one based on FSQ tokens indeed performs the best, and, moreover, it outperforms its strong LLM-based counterpart, at the same time being significantly smaller and faster.
