Search papers, labs, and topics across Lattice.
This paper introduces a vision-language pipeline for estimating disaster severity scores from multimodal social media data by combining domain-adapted captioning with BLIP2, semantic alignment filtering using CLIPScore and ITM, and chain-of-thought reasoning with a large multimodal language model. The approach addresses the challenge of noisy social media data by filtering semantically misaligned image-text pairs and leveraging LLMs for reasoning. Experiments on the CrisisMMD dataset show that the proposed pipeline achieves up to 80.2% precision and strong F1 performance for high-confidence severe damage cases, outperforming traditional and unimodal fusion baselines.
Achieve trustworthy disaster severity scoring by fusing vision-language models with semantic alignment and chain-of-thought reasoning, enabling precise damage assessment from noisy social media data.
Accurate and timely assessment of the severity of the damage is vital for effective disaster response and prioritization of resources. However, the unstructured and noisy nature of multimodal social media data poses challenges for reliable automated analysis. This paper introduces a robust vision-language pipeline that combines domain-adapted caption generation with BLIP2, semantic alignment filtering using CLIPScore and ImageText Matching (ITM), and chain-of-thought reasoning with a large multimodal language model (LLM) to estimate quantitative severity scores. Evaluations of the CrisisMMD dataset demonstrate significant improvements over the baselines of traditional and unimodal fusion, achieving up to 80.2% precision and strong F1 performance for high-confidence severe damage cases. To address uncertain or low-confidence instances, we advocate a human-in-the-loop design that routes ambiguous cases for manual verification, ensuring trustworthy decision support for emergency response operations.
