Search papers, labs, and topics across Lattice.
The paper introduces Graph-to-Frame RAG (G2F-RAG), a training-free method for improving video reasoning in LMMs by integrating external knowledge in the visual space. G2F-RAG constructs a video knowledge graph and renders relevant subgraphs as reasoning frames appended to the video, allowing the LMM to reason in a unified visual domain. Experiments on diverse benchmarks demonstrate consistent gains, especially in knowledge-intensive settings, highlighting the importance of knowledge representation and delivery.
Forget long-context attention: G2F-RAG shows you can boost video reasoning in LMMs by fusing retrieved knowledge directly into the visual space.
When video reasoning requires external knowledge, many systems with large multimodal models (LMMs) adopt retrieval augmentation to supply the missing context. Appending textual or multi-clip evidence, however, forces heterogeneous signals into a single attention space. We observe diluted attention and higher cognitive load even on non-long videos. The bottleneck is not only what to retrieve but how to represent and fuse external knowledge with the video backbone.We present Graph-to-Frame RAG (G2F-RAG), a training free and auditable paradigm that delivers knowledge in the visual space. On the offline stage, an agent builds a problem-agnostic video knowledge graph that integrates entities, events, spatial relations, and linked world knowledge. On the online stage, a hierarchical multi-agent controller decides whether external knowledge is needed, retrieves a minimal sufficient subgraph, and renders it as a single reasoning frame appended to the video. LMMs then perform joint reasoning in a unified visual domain. This design reduces cognitive load and leaves an explicit, inspectable evidence trail.G2F-RAG is plug-and-play across backbones and scales. It yields consistent gains on diverse public benchmarks, with larger improvements in knowledge-intensive settings. Ablations further confirm that knowledge representation and delivery matter. G2F-RAG reframes retrieval as visual space knowledge fusion for robust and interpretable video reasoning.
