Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the limitations of vision-language models (VLMs) in accurately assessing fine-grained vehicle damage by introducing TinyDamage, a hybrid architecture that combines a dedicated segmentation model for spatial grounding with a VLM for semantic reasoning. The study reveals that while the Qwen-VL model achieves high semantic classification accuracy, it struggles with spatial grounding, leading to significant hallucinations and missed detections in vehicle damage assessment. By integrating a segmentation model that employs a supervised contrastive objective, the authors demonstrate a dramatic reduction in report hallucination rates from 92% to 31%, showcasing the effectiveness of their approach in enhancing the reliability of automated assessments.
Segmentation strategies can drastically cut hallucination rates in VLM-generated reports, transforming vehicle damage assessment accuracy.
Vision-language models (VLMs) are increasingly deployed as reasoning agents in real-world visual assessment pipelines, yet their spatial grounding remains unreliable for fine-grained, visually ambiguous targets. We study this gap in the context of automated vehicle damage assessment, where fine-grained defects such as scratches and hairline cracks occupy few pixels, produce weak gradient signal, and are easily confused with reflections and surface texture. We show that a state-of-the-art VLM (Qwen-VL) achieves strong semantic classification accuracy (87.3%) on this task but is systematically ungrounded at the spatial level: it hallucinates damage in reflective regions, misses elongated scratches entirely, and produces spatially inconsistent outputs when prompted for localization. We propose TinyDamage, a hybrid architecture that delegates spatial grounding to a dedicated multi-task segmentation model while reserving the VLM for semantic reasoning and report generation. On the segmentation side, we find that the choice of loss function has an outsized and underexplored effect on tiny-object grounding: focal loss, widely used for class imbalance, collapses tiny-damage detection to zero, while a supervised contrastive objective measurably improves damage/background separability. We integrate the segmentation model into a 7-node LangGraph agent pipeline that grounds every VLM generation step in the segmentation output, and show that this grounding reduces the report hallucination rate from 92% (text-only) and 78% (image-only) to 31% in a controlled evaluation on 100 human-verified reports. We introduce DET_l, a permissive per-category detection metric for evaluating tiny-object grounding under class imbalance, and report latency and reliability characteristics of the deployed pipeline.