Search papers, labs, and topics across Lattice.
This paper organizes audio generative modeling by examining the interplay between representation design and distribution modeling, focusing on how different latent representations鈥攄iscrete, continuous, and hybrid鈥攁ffect performance across four key objectives. It introduces two diagnostic dimensions, dependency horizon and conditional ambiguity, to refine the understanding of how to allocate modeling capacity based on contextual relevance and uncertainty. The findings reveal that existing systems like RVQ and AudioLM illustrate the practical implications of this framework, highlighting the trade-offs between modeling strategies and representation types in audio generation tasks.
The interplay between representation and modeling strategies can redefine how we optimize audio generative systems for context and uncertainty.
Every audio generative system makes two coupled decisions: what representation to generate, and how to model its distribution. This paper organizes audio generative modeling around this coupling. For representation design, we compare discrete, continuous, and hybrid latents through four objectives: representation burden, distortion, empirical modelability, and streaming compatibility. For distribution modeling, rather than treating a latent's difficulty as an intrinsic scalar, we use two diagnostic dimensions: dependency horizon, how far useful context extends, and conditional ambiguity, how much uncertainty remains after conditioning. These dimensions refine the common semantic-versus-acoustic intuition: variables with a long dependency horizon should receive global modeling capacity. Conditionally ambiguous detail may be delegated to a local or iterative generator. Applied to representative systems, this view shows that RVQ's residual order gives ordered capacity but not ordered semantics, that AudioLM's semantic-versus-acoustic cascade is one explicit placement of this boundary rather than a universal template, and that autoregression, iterative refinement, and hybrid designs differ chiefly in how they trade dependency horizon against critical-path generation cost. The distinction between discrete and continuous latents describes the output interface; dependency horizon, conditional ambiguity, and streaming determine how that interface should be modeled. Rather than cataloguing individual systems, we provide an evaluation and design framework for comparing representation-model pairs.
