Search papers, labs, and topics across Lattice.
The paper introduces the Distilled Audio-Visual State-Space (DAVSS) model, which leverages knowledge distillation from transformer teachers to create a multimodal state-space model that is significantly smaller and more efficient. Despite having only 14 million parameters, DAVSS outperforms larger transformer-based models like CAV-MAE by optimizing input resolution and enhancing joint audio-visual processing. Key improvements include using finer input resolutions with smaller patch sizes and allocating a larger portion of the model for cross-modal interactions, leading to better performance without a substantial increase in computational cost.
A 14M parameter model that outperforms larger transformers while being 12 times smaller, reshaping the efficiency landscape of audio-visual processing.
State-space models (SSMs) distilled from transformer teachers combine the performance of transformers with the efficiency of SSMs. We extend the Transformer-SSM knowledge distillation to a multimodal setting and propose the Distilled Audio-visual State-Space (DAVSS) model. The DAVSS model, 14M parameters, is 12 times smaller compared to transformer-based models such as CAV-MAE, and still outperforms them. DAVSS improves over the existing audio-visual models by: 1) Finer input resolution: using smaller patch sizes process the input, compensating for the smaller model size by increasing input sequence lengths. This is supported by the observation that a larger patch size results in lower performance. 2) Deeper joint modeling: utilizing a larger portion of the model (30%) for joint audio-visual processing, compared to<5% in CAV-MAE, enabling deeper cross-modal interaction without significantly increasing the computational cost associated with the concatenated audio-visual tokens.
