Search papers, labs, and topics across Lattice.
Amazon Science
CMU ML
Meta AI
Stanford HAIColumbiaJames Silberrad BrownJames Silberrad Brown Center for AINortheasternNYUJun 17, 2026arXiv:2606.19398This paper introduces S-JEPA, a self-supervised speech representation learning method that utilizes a continuous optimization trajectory to match soft posteriors from a Gaussian Mixture Model (GMM) instead of relying on discrete hard cluster IDs. By eliminating the need for offline re-clustering and hand-tuned layer selection, S-JEPA achieves the lowest word error rate (WER) among self-supervised learning methods with fewer than 90 million parameters, while also matching the performance of HuBERT-Base in emotion recognition at half the parameter count. The findings highlight that the soft-target objective preserves acoustic ambiguity, which is critical for effective speech representation learning.
S-JEPA sets a new standard in speech representation learning by achieving top performance with fewer parameters and without the cumbersome offline re-clustering process.
Self-supervised speech encoders are predominantly trained by predicting discrete hard cluster IDs at masked positions, a recipe that collapses acoustic ambiguity at category boundaries and requires interrupting training to re-cluster the entire corpus between iterations. We introduce S-JEPA, a JEPA-style encoder-predictor pair trained to match the soft posteriors of a Gaussian Mixture Model at masked positions via KL divergence. Training runs as one continuous optimization trajectory in two phases: a fixed GMM over MFCC features, then an online GMM over encoder features, with the input layer selected adaptively from a label-free signal, removing both the offline re-cluster step and the hand-tuned choice of which transformer layer to cluster on. Under the SUPERB protocol, S-JEPA achieves the lowest WER among evaluated SSL methods below 90M parameters and matches HuBERT-Base on emotion recognition at roughly half its parameter count, establishing a new Pareto frontier without offline re-clustering or teacher distillation. An analysis of the predictor's per-frame entropy on held-out speech reveals a bimodal distribution with a substantial minority of frames near the entropy of a perfect two-cluster tie, providing direct empirical evidence that the soft-target objective preserves the acoustic ambiguity that hard targets would collapse. Code is available at https://github.com/gioannides/s-jepa.
