Search papers, labs, and topics across Lattice.
This paper introduces InsideSSL, a model-centric framework that analyzes self-supervised learning (SSL) models for speech representation by examining their layer-wise dynamics through compression, geometry, and robustness. The study reveals that different training objectives lead to unique regimes of acoustic compression and manifold unfolding, while the Generative Compatibility Matrix (GCM) highlights the functional transferability across layers, identifying stable phonetic cores and deep-layer semantic pruning. Ultimately, the findings link layer topology to downstream task performance, showing how it influences phoneme, pitch, and speaker encoding.
Distinct training objectives in self-supervised speech models create unique acoustic compression regimes that impact downstream task performance.
Self-supervised learning (SSL) models, such as Wav2Vec2, HuBERT, and WavLM, have become foundational across a wide range of speech and audio tasks. Despite their success, understanding their internal layer-wise dynamics remains an ongoing challenge. To address this, we propose a two-part model-centric framework called InsideSSL. First, we establish a task-agnostic analysis from three intrinsic per-layer perspectives: compression (entropy), geometry (curvature), and robustness to perturbations. We show that varying training objectives induce distinct regimes of acoustic compression and manifold unfolding. Second, we introduce the cross-layer Generative Compatibility Matrix (GCM) to evaluate functional transferability, exposing stable phonetic cores, identity volatility, and deep-layer semantic pruning. In addition to these evaluations, linear probing connects the model-centric perspective to downstream tasks, demonstrating how layer topology dictates phoneme, pitch, and speaker encoding.
