Search papers, labs, and topics across Lattice.
This paper investigates the limitations of state-of-the-art strided convolutional encoders in end-to-end neural audio models, revealing two structural bottlenecks that hinder access to frequency-localized primitives: alias equivalence class collapse and restricted frequency resolution. Through theoretical analysis and controlled experiments, the authors quantify these bottlenecks, demonstrating significant degradation in representational capacity and filter bandwidth under realistic signal conditions. They introduce Gabor Latent Refactorization (GLRF), a novel intervention that effectively mitigates these issues, enhancing the model's ability to control and interpret audio features without requiring retraining.
State-of-the-art audio encoders may obscure critical frequency-localized features, but a simple post-hoc intervention can recover access and improve interpretability.
End-to-end neural audio models achieve high-fidelity compression and generation. We might read that performance as evidence they directly represent interpretable features such as pitch and timbre, but a model can produce plausible outputs without doing so. A model may encode these features in any reachable basis, but regardless of which, the features are well described as compositions of time-frequency-localized primitives. Whether state-of-the-art encoders preserve access to these primitives, and thus to compositions of them, remains unclear. Through theoretical analysis and controlled experiments, we show that several state-of-the-art strided convolutional encoders impose two structural bottlenecks, both predictable from architecture and signal structure, on access to these primitives: (1) they collapse primitives into alias equivalence classes, establishing a bound on representational capacity, and (2) they limit the frequency resolution available to learned filters, restricting separability. For well structured data, we find collapse rates of 31-35% and filter bandwidths 10-35x above the theoretical resolution bound, confirming that both bottlenecks arise under realistic signal conditions. We then introduce Gabor Latent Refactorization (GLRF), a lightweight post-hoc intervention that re-expresses encoder latents in a frequency-localized basis, reducing filter bandwidths from 10-35x to 1.5-3x of the theoretical resolution bound while preserving reconstruction fidelity and improving control over attributes like pitch. These results show that the encoders in question predictably degrade access to frequency-localized primitives, entangling the features that depend on them, and that a lightweight, retraining-free intervention can recover much of that access, improving steerability and interpretability.
