Search papers, labs, and topics across Lattice.
This paper introduces a label-mediated method to generate natural language descriptions of human brain microscopy images, addressing the scarcity of paired image-text data in cytoarchitectural analysis. The method leverages existing cytoarchitectonic vision foundation models (CytoNet) and large language models, linking them through automatically mined area descriptions from literature based on image labels. Results demonstrate the model's ability to generate plausible area-level descriptions, achieve high accuracy in matching reference labels, and maintain discriminative power even with masked area labels.
Unlock natural language descriptions for brain microscopy images without paired image-text data, using a clever label-mediated approach that mines captions from existing literature.
Foundation models increasingly offer potential to support interactive, agentic workflows that assist researchers during analysis and interpretation of image data. Such workflows often require coupling vision to language to provide a natural-language interface. However, paired image-text data needed to learn this coupling are scarce and difficult to obtain in many research and clinical settings. One such setting is microscopic analysis of cell-body-stained histological human brain sections, which enables the study of cytoarchitecture: cell density and morphology and their laminar and areal organization. Here, we propose a label-mediated method that generates meaningful captions from images by linking images and text only through a label, without requiring curated paired image-text data. Given the label, we automatically mine area descriptions from related literature and use them as synthetic captions reflecting canonical cytoarchitectonic attributes. An existing cytoarchitectonic vision foundation model (CytoNet) is then coupled to a large language model via an image-to-text training objective, enabling microscopy regions to be described in natural language. Across 57 brain areas, the resulting method produces plausible area-level descriptions and supports open-set use through explicit rejection of unseen areas. It matches the cytoarchitectonic reference label for in-scope patches with 90.6% accuracy and, with the area label masked, its descriptions remain discriminative enough to recover the area in an 8-way test with 68.6% accuracy. These results suggest that weak, label-mediated pairing can suffice to connect existing biomedical vision foundation models to language, providing a practical recipe for integrating natural-language in domains where fine-grained paired annotations are scarce.
