Search papers, labs, and topics across Lattice.
The paper introduces QuPAINT, a physics-aware instruction tuning approach for characterizing 2D quantum materials from optical microscopy images. They address the limitations of existing vision models by generating synthetic training data (Synthia) based on thin-film interference physics and creating a large-scale instruction dataset (QMat-Instruct) of physics-informed question-answer pairs. QuPAINT incorporates a Physics-Informed Attention module to fuse visual embeddings with optical priors, achieving improved performance on the newly established QF-Bench benchmark.
By injecting physics-based priors into multimodal LLMs, QuPAINT achieves robust quantum material characterization from optical microscopy, even with limited real-world data.
Characterizing two-dimensional quantum materials from optical microscopy images is challenging due to the subtle layer-dependent contrast, limited labeled data, and significant variation across laboratories and imaging setups. Existing vision models struggle in this domain since they lack physical priors and cannot generalize to new materials or hardware conditions. This work presents a new physics-aware multimodal framework that addresses these limitations from both the data and model perspectives. We first present Synthia, a physics-based synthetic data generator that simulates realistic optical responses of quantum material flakes under thin-film interference. Synthia produces diverse and high-quality samples, helping reduce the dependence on expert manual annotation. We introduce QMat-Instruct, the first large-scale instruction dataset for quantum materials, comprising multimodal, physics-informed question-answer pairs designed to teach Multimodal Large Language Models (MLLMs) to understand the appearance and thickness of flakes. Then, we propose Physics-Aware Instruction Tuning (QuPAINT), a multimodal architecture that incorporates a Physics-Informed Attention module to fuse visual embeddings with optical priors, enabling more robust and discriminative flake representations. Finally, we establish QF-Bench, a comprehensive benchmark spanning multiple materials, substrates, and imaging settings, offering standardized protocols for fair and reproducible evaluation.
