Search papers, labs, and topics across Lattice.
This paper introduces a novel mechanism that allows a neural audio codec (NAC) to operate at multiple token temporal resolutions (TTRs) without requiring separate training for each resolution. By utilizing sampling-frequency-independent convolutional layers, the method generates TTR-dependent convolutional kernels from a shared parameter set, effectively balancing the representation of rapid acoustic events with token-sequence length. Experimental results demonstrate that this approach significantly outperforms traditional models that rely on switching TTR-specific layers, enhancing the efficiency and versatility of audio generation and understanding tasks.
A single neural audio codec can now flexibly operate across multiple temporal resolutions, outperforming traditional models that require separate training for each resolution.
Discrete tokens obtained from neural audio codecs (NACs) have been used as compact representations in audio generation and understanding models. In such token-based systems, token temporal resolution (TTR), defined as the time interval between adjacent token frames, is important because it controls the trade-off between representing rapid acoustic events and reducing token-sequence length. However, most NACs are trained at a single TTR and require separate training for each TTR. This paper proposes a mechanism that enables a single NAC to operate at multiple TTRs using sampling-frequency-independent convolutional layers. The mechanism regards TTR as the sampling period of the token sequence and generates TTR-dependent convolutional kernels from a shared parameter set, while adjusting the kernel size and stride for each TTR. We incorporate the mechanism into Descript Audio Codec, leaving the quantizer unchanged. Experiments on environmental sound reconstruction show that the proposed model outperforms a single-model baseline that switches TTR-specific layers for each TTR.
