Search papers, labs, and topics across Lattice.
This study investigates the localization of X-ray fluorescence (XRF) images within optical microscopy fields of view using training-free vision language models (VLMs). The research addresses the challenge of aligning images from different modalities, particularly when structural correspondence is low, by employing a proposal-and-verify workflow that combines VLM predictions with image similarity measures. The key finding reveals that while direct VLM prompting alone is unreliable, the proposed workflow significantly enhances localization accuracy in low-correspondence scenarios, outperforming traditional methods.
A novel proposal-and-verify approach using vision language models can recover accurate localization in challenging low-correspondence imaging scenarios where traditional methods fail.
Registering images acquired with different microscopy modalities is essential for relating complementary measurements of the same specimen. In correlative X-ray fluorescence (XRF) and optical microscopy, the XRF map often covers only a small region of an optical image acquired from the same or an adjacent tissue section. Field-of-view (FOV) localization is necessary but can be difficult when appearance and structure differ across modalities. Here we evaluate training-free vision language model (VLM) localization on two datasets representing same-section high-correspondence and adjacent-section low-correspondence imaging. We test unconstrained and metadata-constrained search and compare VLMs with geometric controls, classical template matching, and two alternative training-free approaches (DINOv2 and multiGradICON). Direct VLM prompting produced content-dependent spatial signals but was not reliable alone. Classical matching was most accurate when cross-modal structure was preserved but failed in the low-correspondence collection. A proposal-and-verify workflow used repeated VLM predictions as candidates and image-based similarity to select the final location. This workflow recovered useful localization in the low-correspondence regime.
