Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces BEST-RQ-2, a two-step contextualize-then-predict pretraining scheme for self-supervised audio representations that builds on the original BEST-RQ framework. By employing a ViT context encoder to process unmasked spectrogram regions and a lightweight predictor for masked regions, the method achieves improved performance across music and environmental sound tasks, albeit with a slight reduction in speech performance. The separation of contextualization and prediction stages leads to consistent improvements in transfer learning benchmarks, demonstrating the efficacy of this approach while maintaining inference compute efficiency.
BEST-RQ-2 achieves superior audio representation transfer by decoupling contextualization from prediction, challenging the efficiency of traditional one-stage models.
Self-supervised learning enables audio representations that transfer across domains and tasks. We present BEST-RQ-2, an evolution of BEST-RQ that retains frozen randomprojection-based discrete targets while introducing a two-step contextualize-then-predict pretraining scheme. A ViT context encoder processes only the unmasked spectrogram regions, and a lightweight predictor infers targets for the masked regions; the predictor is discarded after pretraining. Replacing the original Conformer encoder with a ViT shifts performance across domains, slightly reducing speech performance while improving music and environmental sounds, with comparable average scores. The main improvement comes from decomposing masked prediction into separate contextualization and prediction stages. On the X-ARES and XARES-LLM benchmarks, BEST-RQ-2 consistently outperforms one-stage baselines in overall transfer while keeping inference compute unchanged. Code and model checkpoints are publicly available.