Search papers, labs, and topics across Lattice.
This paper introduces a retrieval-augmented image captioning framework that enhances traditional methods by integrating external knowledge to generate richer, context-aware descriptions. By employing a hierarchical multi-modal article retrieval system, the approach effectively utilizes structured article features and multi-faceted similarity computations to inform caption generation. The authors achieved a competitive 5th place in the ACM Multimedia EVENTA 2025 Challenge, demonstrating the efficacy of their method on the OpenEvent-V1 dataset with a score of 0.2824.
Context-rich image captions are now possible by leveraging structured external knowledge, transforming how we understand image description generation.
Traditional image captioning methods often struggle to generate comprehensive, context-rich descriptions, especially for details not directly observable from visual cues. To overcome this, we propose a novel retrieval-augmented image captioning framework that generates captions with deeper insights, such as object attributes, event context, and underlying significance, by leveraging external knowledge. Our approach features a hierarchical multi-modal article retrieval mechanism that moves beyond monolithic text entities. This retrieval considers article structure-aware features, including weighted textual components (e.g., headlines, body sections) and visual placement patterns, alongside multi-faceted similarity computations (content--visual, visual--visual, and discourse positioning). A subsequent contextual relevance refinement stage further enhances the retrieved information. The retrieved articles then serve as the knowledge base for caption generation: first, a VLM generates a concise image description; second, we segment relevant information from the retrieved articles based on this description; and finally, an LLM utilizes both the description and extracted knowledge to generate a comprehensive, contextually detailed caption. We participated in the ACM Multimedia EVENTA 2025 Challenge and achieved 5th place with an overall score of 0.2824 on the private test set of the OpenEvent-V1 dataset. Source code is publicly released at https://github.com/mf0212/EVENTA-Challange.
