Search papers, labs, and topics across Lattice.
This paper introduces the Multimodal Concept Bottleneck Model (MM-CBM), which enhances the interpretability of deep learning networks by aligning image and text embeddings with natural concepts through dual Concept Bottleneck Layers (CBLs). By addressing the limitations of traditional Concept Bottleneck Models, such as fixed class generalization and information leakage, MM-CBM enables interpretable performance on new vision tasks like zero-shot classification and image retrieval. The results show an impressive average accuracy improvement of 51.26% across four benchmarks, while maintaining high accuracy within 5% of black-box models.
Achieving over 51% accuracy improvement, MM-CBM redefines interpretability in multimodal deep learning by aligning image and text features with natural concepts.
Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts. However, existing CBMs are constrained in their ability to generalize beyond a fixed set of predefined classes and the risk of non-concept information leakage, where predictive signals outside the intended concepts are inadvertently exploited. In this paper, we propose Multimodal Concept Bottleneck Model (MM-CBM) to address these issues and extend CBMs into CLIP. MM-CBM utilizes dual Concept Bottleneck Layers (CBLs) to align both the image and text embeddings into interpretable features. This allows us to perform new vision tasks like zero-shot classification or image retrieval in an interpretable way. Compared to existing methods, MM-CBM achieves up to 51.26% accuracy improvement on average across four standard benchmarks. Our method maintains high accuracy, staying within ~5% of black-box performance while offering greater interpretability.
