Search papers, labs, and topics across Lattice.
The paper introduces TAU-Agent, a retrieval-augmented framework designed for effective traffic anomaly understanding by integrating a central retrieval agent with two visual perception tools: a Video Captioning Tool and an Open-Vocabulary Tracking Tool. This framework enables the selection of relevant evidence based on task queries, which is then processed by a fine-tuned vision-language model for reasoning and answer generation. Evaluated on the AI City Challenge 2026 benchmarks, TAU-Agent demonstrates competitive performance, achieving notable rankings across multiple tracks, underscoring its efficacy in handling complex traffic scenarios.
TAU-Agent leverages a novel retrieval mechanism to enhance traffic anomaly detection, achieving impressive benchmark results that challenge existing paradigms in video analysis.
Traffic Anomaly Understanding (TAU) requires models and systems to detect, reason about, and explain anomalous events in transportation videos. To address this challenge, we propose TAU-Agent, an agentic retrieval-augmented framework for traffic anomaly understanding. Given a task query, a central retrieval agent orchestrates two visual perception tools, namely a Video Captioning Tool and an Open-Vocabulary Tracking Tool, to retrieve and select query-relevant evidence, including captions, temporal intervals, and object trajectories. The selected evidence, together with sampled video frames and the input query, is provided to a supervised fine-tuned vision-language model for final reasoning and answer generation. We evaluate TAU-Agent on both the in-domain and the out-of-domain benchmarks from the AI City Challenge 2026. TAU-Agent achieves scores of 0.6779 on Track 3, 0.3998 on Track 7, and 67.9275 on Track 8, ranking second, twelfth, and fifth, respectively. Code is available at: https://github.com/siri-rouser/TAU-Agent.
