Search papers, labs, and topics across Lattice.
This paper introduces C3-Bench, a novel benchmark designed to evaluate Context-aware Change Captioning systems across diverse real-world scenarios, featuring 4,996 human-labeled image pairs from 51 change contexts in four domains. The authors employ an innovative LLM-as-Judge framework to assess models on fine-grained dimensions such as correctness and fluency, revealing that conventional models and even advanced LLMs struggle significantly when faced with changes outside their training regimes. The findings highlight critical blind spots in current change captioning approaches, underscoring the need for improved generalization and reliability in these systems.
Conventional change captioning models falter dramatically in unfamiliar contexts, exposing a critical vulnerability in the current paradigm.
While Change Captioning systems have garnered substantial attention to respond to our evolving world, their true performance on diverse real-world change contexts remains largely unexplored due to the lack of comprehensive evaluation frameworks. To fill this gap, we propose C3-Bench, a comprehensive benchmark for evaluating Context-aware Change Captioning. C3-Bench features: (1) 4,996 human-labeled image pairs of 51 real-world change contexts across four domains (e.g., natural scenes, remote sensing imagery, image editing, and anomalies), each with diverse, carefully curated scenarios derived from multiple change-centric communities; and (2) the first LLM-as-Judge evaluation framework in the change captioning task that measure fine-grained dimensions (e.g., correctness, specificity, fluency, and relevance), along with a novel reversibility metric exploring whether models understand changes with symmetric consistency. Based on C3-Bench, we benchmark 32 models -- including conventional change captioning models, proprietary Large Multimodal Models (LMMs), and 2B-90B open-source LMMs. We reveal a fundamental blind spot in the prevailing change captioning paradigm: Once the change context departs from training-style regimes, conventional models collapse, and even state-of-the-art LMMs such as GPT-5.2 exhibit systematic domain- and position-dependent errors that distort reliable change understanding. By making these hidden failure modes explicit and measurable, we delineate the next frontier for building generalizable and trustworthy change captioning systems. All codes and datasets are publicly available on the project page.
