Search papers, labs, and topics across Lattice.
This paper introduces PRISM, a novel framework for adapting Audio-Text Foundation Models (ATMs) to severe acoustic noise without requiring additional training or privileged noise annotations. By leveraging the Affine Noise Hypothesis, PRISM effectively estimates and reverses distortion in the multimodal latent space using frozen text prototypes, achieving rapid adaptation through a single matrix-vector multiplication. The method demonstrates significant performance improvements on the UrbanSound8K dataset, outperforming both zero-shot and oracle-assisted baselines by substantial margins, while also addressing the Polyphonic Trap failure mode through Confidence-Aware Regression.
PRISM achieves a remarkable 12.94 percentage point improvement in noisy audio classification without any additional training, redefining adaptation strategies for ATMs.
Audio-Text Foundation Models (ATMs) fail catastrophically under severe acoustic noise, yet existing adaptation strategies either rely on gradient-based Test-Time Adaptation (TTA), which reinforces noise rather than signal, or on prompt tuning that requires privileged noise annotations unavailable at inference. We address these failures with PRISM (Prototype-Rectified Iterative Self-supervised Manifold Denoising), a training-free, source-free TTA framework grounded in the Affine Noise Hypothesis: severe acoustic noise induces a low-rank affine shift in the multimodal latent space, with more than 90% of distortion energy confined to the leading 60 principal components. PRISM estimates and reverses this distortion from an unlabeled target batch using frozen text prototypes as geometric anchors via three closed-form geometric corrections compiled into a single static projection matrix by Affine Bias Regression. At inference, adaptation reduces to one matrix-vector multiplication in 0.0009 ms, making it substantially faster than gradient-based TTA while requiring no additional training. On UrbanSound8K, PRISM improves over the zero-shot baseline by 12.94 percentage points and surpasses an oracle-assisted TTA baseline by 9.41 percentage points, despite never observing its privileged augmented noise prompts. We further identify the Polyphonic Trap, a principled failure mode of subspace deflation for broadband classes, and resolve it via Confidence-Aware Regression (CAR), recovering up to 8.16 percentage points for the worst-affected class.
