Search papers, labs, and topics across Lattice.
This paper introduces MM-IssueLoc, a controlled benchmark designed to evaluate the effectiveness of visual evidence in repository-level issue localization, addressing the limitations of existing text-only evaluations. The benchmark includes 652 issue-pull request instances across 23 programming languages, with detailed annotations for image categories and relevance levels, allowing for nuanced assessments of multimodal systems. Results indicate that current systems struggle with multimodal localization, with the best-performing model achieving only 38.96% accuracy at the file level, highlighting the need for improved integration of visual evidence in issue localization tasks.
Existing multimodal systems falter in repository-level localization, with the best performance still falling short of reliable accuracy thresholds.
Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.
Tsinghua AI