Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel multi-agent framework called Divide, Deliberate, Decide for fine-grained egocentric action recognition in videos, addressing the limitations of Vision-Language Models (VLMs) that often exhibit bias towards specific visual cues. The framework operates fully locally and employs an orchestrator to segment the video and propose candidate labels, while an ensemble of diverse VLM specialists engages in a structured deliberation process to refine predictions. Experimental results demonstrate significant improvements in zero-shot action recognition performance, primarily attributed to the benefits of heterogeneous model deliberation rather than increased computational resources.
A multi-agent framework that leverages diverse VLMs can dramatically enhance zero-shot action recognition by overcoming inherent model biases.
Fine-grained action recognition in egocentric video is challenging for Vision-Language Models (VLMs): actions often differ only in small visual cues, and a single model tends to be biased toward a subset of these cues. We propose Divide, Deliberate, Decide, a fully-local, zero-shot multi-agent framework in which (i) a VLM orchestrator chunks the video and proposes a top-k candidate label list per segment, (ii) an ensemble of heterogeneous VLM specialists, drawn from different open model families, engages in a structured deliberation that includes a peer-consultation round of questions, and (iii) agent rankings are aggregated with a Borda count and the orchestrator re-ranks its own prediction in light of the specialists' evidence. The entire pipeline runs locally with no fine-tuning. Experiments show that our method positively improves zero-shot action recognition performance over the baseline, highlighting the influence of a heterogeneous deliberation step, showing that the gain stems from decorrelated model priors rather than from additional compute.