Search papers, labs, and topics across Lattice.
This paper introduces NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework that employs a causal Transformer to predict the next patch embedding of a log-mel spectrogram from previous embeddings, leveraging causal masking and stop-gradient as its sole training signal. By adopting a minimalist approach that avoids complex architectures and auxiliary losses, NAPE demonstrates superior performance across six audio and speech benchmarks, achieving state-of-the-art results in fine-tuning and strong linear-probing outcomes. The findings suggest that autoregressive prediction can effectively harness audio's temporal structure, leading to scalable and efficient audio learners.
NAPE achieves state-of-the-art performance in audio representation learning by simplifying the pre-training process to a single autoregressive prediction task.
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
