Search papers, labs, and topics across Lattice.
This study leverages visual language models (VLMs) to enhance the performance of an astronomy foundation model, Zoobot, in recognizing galaxy morphology with limited human supervision. By introducing a survey-oriented visual question answering (VQA) benchmark and evaluating state-of-the-art VLMs, the authors demonstrate that these models can provide meaningful weak supervision, improving classification accuracy while highlighting their limitations in replacing human annotators. The findings indicate that integrating a general-purpose VLM as a teacher can significantly boost the label efficiency of morphology classification tasks, paving the way for better adaptation to future astronomical surveys.
A general-purpose VLM can enhance an astronomy model's galaxy morphology recognition, achieving better results with minimal human labeling.
Existing astronomy foundation models provide strong galaxy representations, but adapting them to new survey conditions and survey-specific morphology recognition tasks still requires substantial human supervision. We show that VLM-based VQA systems contain meaningful visual-semantic priors that can serve as weak supervision for downstream morphology classifiers and improve morphology classification under limited human-label budgets. We first introduce a survey-oriented VQA benchmark spanning two representative imaging regimes and evaluate state-of-the-art VLMs on galaxy morphology questions. The results show that these models capture useful morphology signals and informative uncertainty, but are not sufficiently reliable to replace human annotators. Motivated by this finding, we use a general-purpose VLM as a morphology teacher for Zoobot, an astronomy foundation model pretrained on large-scale Galaxy Zoo annotations. Across two survey domains and multiple annotation budgets, the VLM teacher consistently improves Zoobot's downstream morphology classification. These results demonstrate that a general-purpose VLM provides knowledge complementary to an astronomy foundation model and can teach it to better recognize galaxy morphology under limited human supervision. The resulting pipeline is designed for label-efficient adaptation to forthcoming large-scale surveys, including the Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) and the Nancy Grace Roman Space Telescope. The benchmark and code are publicly available at https://github.com/fw-ic/VLM-morphology-teacher.
