Search papers, labs, and topics across Lattice.
This paper addresses the challenge of out-of-distribution (OOD) detection in multi-modal action recognition by introducing a post-hoc detector that leverages the relationship between multi-modal and uni-modal predictions. By combining a new information signal with a feature-space score, the proposed hybrid approach enhances OOD detection by normalizing these scores with multi-modal logits. Experimental results across various datasets demonstrate that this method consistently surpasses existing state-of-the-art techniques, highlighting the critical role of modality awareness during inference.
Ignoring modality-specific signals during inference can lead to significant performance drops in multi-modal OOD detection, but this new hybrid approach turns that insight into a robust solution.
The incorporation of additional modalities into action recognition models increases their performance across a wide range of settings. However, how this additional information can contribute to making the models more robust remains underexplored, particularly for the case of multi-modal out-of-distribution (OOD) detection. While methods exist that regularize the multi-modal training process with OOD detection in mind, they still apply off-the-shelf OOD detectors designed for the uni-modal case during inference, discarding important information. Based on an interesting relationship we find between the multi-modal and uni-modal predictions, we propose to use this signal to build a post-hoc detector explicitly designed for the multi-modal scenario. We combine this new source of information with a feature-space score, which detects off-manifold samples in the multi-modal space, and normalize them by the multi-modal logits. In doing so, the proposed hybrid detector is compatible with existing training-time approaches and consistently improves performance. Experiments on a wide range of established datasets from the MultiOOD benchmark show that, on average, our approach outperforms the state of the art. Our results show the importance of explicitly considering the different modalities at inference time for multi-modal OOD detection.
