Search papers, labs, and topics across Lattice.
This paper introduces Caption Bottleneck Models (CaBM), which enhance interpretability in machine learning by utilizing free-form natural language captions instead of predefined concept sets. By training classifiers on LMM-generated captions, CaBM eliminates information leakage and allows for the autonomous discovery of dataset-specific concepts, addressing the limitations of traditional Concept Bottleneck Models. The results show that CaBM achieves competitive accuracy while maintaining interpretability, without relying on costly expert annotations or static concept sets.
Free-form natural language captions can replace rigid concept layers in models, enabling autonomous discovery of high-quality concepts without the risk of information leakage.
Concept Bottleneck Models (CBMs) provide interpretability by routing predictions through a layer of human-understandable concepts. However, defining an optimal concept set for a specific dataset remains an open challenge. Existing approaches rely on expensive expert annotations or LLM-generated lists based solely on class names. Even "open-vocabulary" variants typically depend on static concept sets, which restrict discovery and introduce label bias. Furthermore, traditional CBMs often suffer from information leakage, where unmodeled visual features bypass the bottleneck and compromise the integrity of the explanations. To overcome these limitations, we propose Caption Bottleneck Models (CaBM), a framework that circumvents the need for predefined concept sets by replacing rigid concept layers with free-form natural language. By representing images via LMM-generated captions and training a classifier strictly on this text, CaBM ensures a leakage-free architecture by construction. Additionally, by analyzing the text classifier post-training, CaBM autonomously discovers high-quality, dataset-specific concepts. Our results across fine- and coarse-grained benchmarks demonstrate that CaBM achieves competitive accuracy while preserving interpretability without the constraints of external dictionaries or manual labeling.
