Search papers, labs, and topics across Lattice.
The paper introduces FlatSounds, a benchmark for evaluating the physical reasoning capabilities of video-to-audio (V2A) models using counterfactual pairs and single-video pattern tests that control for specific physical factors. Evaluation of state-of-the-art V2A models reveals a reliance on text captions over visual information for inferring physics and semantics, leading to a trade-off where captions improve physical accuracy but degrade temporal alignment. The study demonstrates the need for V2A models to learn physical processes directly from pixels, rather than relying on textual priors.
V2A models prioritize text captions over visual cues when generating audio, resulting in physically plausible but often temporally misaligned sounds.
Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes. Existing evaluations emphasize perceptual realism and overlook physical correctness under controlled interventions. In this paper, we introduce FlatSounds, a benchmark that audits the physical reasoning of V2A models through: 1) controlled counterfactual pairs in which a single physical factor is varied, and 2) single-video pattern tests that probe internal consistency and directional trends. These settings test whether the generated audio correctly reflects specific physical properties and timings. Our evaluation of state-of-the-art models reveals a consistent trade-off: models rely more on text captions than the visual stream to infer physics and semantics. Captions generally improve physical and semantic accuracy, but paradoxically degrade temporal alignment. Our results highlight the need to move beyond audio quality toward learning physical processes directly from pixels. Finally, we find that our physics-based metrics correlate strongly with human preference tests on our own data. Project webpage: https://research.nvidia.com/labs/cosmos-lab/flatsounds/
