Search papers, labs, and topics across Lattice.
To evaluate multimodal foundation models in high-stakes emergency domains, this work establishes SAFIRE, a benchmark spanning 83K captioned images and 193K multiple-choice visual questions across 20 real-world fire and smoke scenarios. Evaluating ten open-source MLLMs (8B–38B) reveals severe deficiencies in safety-critical visual reasoning, with models averaging only 61.9% accuracy across 10 evaluation dimensions. Crucially, the authors show that fine-tuning vision encoders on just 7% of curated domain data surges classification accuracy from 20.1% to 64.5%, highlighting that data quality, not scale, is the bottleneck for hazard understanding.
Leading open-source MLLMs achieve an alarming 61.9% average accuracy on fine-grained fire and smoke reasoning, but adapting vision encoders on just 7% targeted domain data more than triples classification performance.
Multimodal Large Language Models (MLLMs) show strong progress on vision-language tasks, yet their reliability in safety-critical settings remains underexplored. Fire-smoke understanding is central to public safety and disaster response, but most existing benchmarks lack diverse real-world scenarios and context-aware evaluation. We introduce SAFIRE, a large-scale benchmark for fire-smoke understanding in MLLMs, comprising 83K captioned images from 20 scenarios and 193K multiple-choice VQA (MCVQA) generated from a 9.7K-image subset, spanning 10 evaluation dimensions from basic perception to higher-order reasoning. A GPT-5.4-assisted multi-stage verification pipeline with MLLM majority voting ensures annotation quality. Evaluating ten open-source MLLMs (8B-38B) yields an average accuracy of 61.9%, exposing major gaps in safety-critical reasoning. We further show that adapting vision encoders with only 7% of our domain-specific data boosts fire-scene classification accuracy from 20.1% to 64.5%, indicating that carefully curated data can yield substantial gains even when data volume is limited. All datasets, models, and code are available at https://risys-lab.github.io/SAFIRE/.
