A Multimodal Video Understanding Agent Based on Video-Audio Multi-Task Joint Fine-Tuning and State Machine Scheduling

Multimodal video understanding requires the joint processing of visual and auditory information, as well as reasoning over temporal and logical relationships. This paper presents a LangGraph-based multimodal video understanding agent that integrates audio, video frames, and subtitles within a state-machine architecture to enable intelligent routing and multi-task processing. We construct a dedicated multimodal video understanding dataset and fine-tune the Qwen2.5-VL model for multi-task learning. The dataset covers a variety of understanding tasks, including perception, temporal reasoning, and higher-level inference on audio–video pairs. By reformulating tasks into prompts and training them jointly, the model learns to share knowledge across tasks. At the input stage, the agent employs a vision–language model to automatically classify the query type, and subsequently routes the request to appropriate nodes for perception, temporal reasoning, inference, or multimodal fusion. During processing, visual information is analyzed by the VLMTool, while audio information is processed by the ALMTool; the results are then jointly reasoned and synthesized in the final fusion stage. Experimental results demonstrate that our approach significantly improves performance on video question answering and temporal localization tasks, thereby validating the effectiveness of multi-task multimodal learning and intelligent routing.