Search papers, labs, and topics across Lattice.
This paper introduces Aligning Hierarchical Standardized Embedding (AHSE), a novel approach for Audio-visual Generalized Zero-shot Learning (AV-GZSL) that addresses the limitations of existing methods by standardizing and hierarchically aligning audio-visual and textual embeddings. By applying Z-score standardization and a multi-level alignment strategy, AHSE effectively minimizes distributional mismatches and enhances the robustness of the embedding space. Experimental results on benchmark datasets show that AHSE achieves competitive performance in zero-shot learning tasks, highlighting its potential to improve classification of both seen and unseen objects or scenes.
Hierarchical alignment of standardized embeddings can significantly enhance zero-shot learning performance by addressing the inherent mismatches between audio-visual and textual modalities.
Audio-visual Generalized Zero-shot Learning (AV-GZSL) is a challenging task that aims to classify both seen and unseen objects or scenes by integrating data from audio and visual modalities. Recent studies primarily focus on fusing or aligning audio and visual features to generate more informative audio-visual embeddings. Also, aligning the audio-visual and textual features of most existing methods relies solely on the optimization objectives. However, those methods neglect the inherent distributional and structural differences between audio-visual and textual modalities. To address this limitation, we propose a method termed Aligning Hierarchical Standardized Embedding (AHSE), which enables hierarchical alignment of standardized audio-visual and textual embeddings within a shared embedding space. Specifically, we first apply Z-score standardization to the fused audio-visual and textual embeddings to reduce distributional mismatches. We then introduce a hierarchical alignment strategy that minimizes discrepancies at the semantic, class, and batch levels, thereby constructing a more robust and well-structured embedding space. This strategy not only preserves semantic and inter-class relationships but also maintains spatial consistency within each batch. Extensive experiments on three benchmark datasets: VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL, demonstrate that AHSE achieves competitive performance in zero-shot learning.
