Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
X-AVDD addresses memory and cross-modal semantic alignment limitations in audio-visual dataset distillation by employing a cross-attention mechanism to explicitly model inter-modal relationships during synthetic data generation. This approach significantly reduces synthesis costs ($\sim 8\times$ GPU-hours) and yields highly aligned data, outperforming existing methods on benchmarks like VGGSound and AVE. X-AVDD also demonstrates strong generalization across architectures and scales effectively to complex models like AudioCLIP, achieving competitive accuracy with a fraction of the original dataset and reduced training time.
Synthesized audio-visual datasets now retain strong cross-modal semantic correspondence, boosting downstream model accuracy by over 15 percentage points and slashing distillation costs.
Audio-visual dataset distillation aims to synthesize a small dataset from a large paired audio-visual dataset, while nearly preserving the training utility for neural networks. Existing approaches such as DM and AVDD face two critical limitations: 1) they are memory-intensive, making them difficult to scale to complex architectures and higher samples-per-class (SPC) settings, and 2) they often fail to preserve strong cross-modal semantic correspondence. To tackle these issues, we propose X-AVDD, a cross-attention-based distillation method that explicitly captures inter-modal relationships during synthesis. By decoupling cross-modal alignment from joint optimization, X-AVDD is substantially more memory-efficient, reducing end-to-end synthesis cost by $\sim 8\times$ GPU-hours, while producing highly aligned synthetic data. Empirically, X-AVDD significantly outperforms other audio-visual distillation methods. On VGGSound, an audio-visual dataset of short YouTube clips, X-AVDD improves top-1 test accuracy from $8.2\%\rightarrow24.0\%$ at SPC=$10$ and $9.8\%\rightarrow26.5\%$ at SPC=$20$. Furthermore, X-AVDD improves cross-architecture generalization (e.g., $25.5\%\rightarrow35.3\%$ for a ViT target on AVE dataset at SPC $=50$) and successfully scales to complex architectures such as AudioCLIP, achieving $62.3\%$ accuracy using only $5.65\%$ of the full dataset, against the full dataset accuracy of $72.4\%$ while completing in $9\times$ lower training time compared to the full dataset.