Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the extent to which visual models can infer physical damping properties from audio cues, finding that audio embeddings achieve a Spearman rank correlation of 0.73 for the quality factor $Q$, significantly outperforming single video frames (0.28). Distillation from an audio teacher to a CLIP backbone improves visual inference, particularly under a leave-one-material-out protocol, recovering a substantial portion of the audio advantage. Expanding visual input to four frames further enhances performance, suggesting that dynamic visual information is crucial for capturing damping characteristics.
Audio embeddings capture physical damping properties far better than single visual frames, and this gap is partially bridgeable through audio-to-vision distillation.
Physical understanding is crucial for decision making, yet vision models struggle to infer properties such as damping that govern how materials behave. Audio encodes such properties directly, because an object's acoustic response to impact reveals its damping through measurable decay. We probe frozen encoders for the quality factor $Q$, a dimensionless measure of how many oscillations an object completes before its ring dies away, and report how well each embedding orders impacts by $Q$ using the Spearman rank correlation $\rho$, for which $\rho=1$ is a perfect ordering and $\rho=0$ an unrelated one. Audio embeddings reach $\rho=0.73$ and retain most of that advantage on materials withheld from training, while a single video frame taken at the moment of impact, encoded by CLIP, supports only $\rho=0.28$. Having established where the signal lives, we ask how much of that gap vision can recover. We distill a frozen audio teacher into a frozen CLIP backbone through a trained projection head and find that this recovers a useful fraction of the audio advantage. Because pooled correlations on this task are dominated in practice by category identity, we also evaluate under a leave-one-material-out protocol in which the held-out material is withheld from the projection head as well as the probe. Distillation with a four-frame input improves the correlation there from 0.234 to 0.338 (+0.053 over the matched four-frame baseline, 14 of 17 materials), a gain that no category-level strategy can produce. Widening the visual input to four frames spanning the impact supplies half of that improvement at no acoustic cost, which indicates that part of what a single frame misses is visible dynamics.