Search papers, labs, and topics across Lattice.
The paper introduces Critic in the Loop, a hierarchical framework for robotic manipulation that combines a VLM for high-level reasoning, a VLA model for reactive control, and a visual Critic for dynamic scheduling. The Critic monitors the workspace and adaptively switches between the VLA for routine tasks and the VLM for replanning when anomalies are detected. This approach reduces VLM query frequency while improving robustness and autonomy, achieving state-of-the-art performance on long-horizon manipulation benchmarks.
A lightweight visual Critic can dynamically schedule VLMs and VLAs to achieve state-of-the-art performance in long-horizon manipulation while minimizing expensive VLM queries.
Balancing high-level semantic reasoning with low-level reactive control remains a core challenge in visual robotic manipulation. While Vision-Language Models (VLMs) excel at cognitive planning, their inference latency precludes real-time execution. Conversely, fast Vision-Language-Action (VLA) models often lack the semantic depth required for complex, long-horizon tasks. To bridge this gap, we introduce Critic in the Loop, an adaptive hierarchical framework driven by dynamic VLM-Expert scheduling. At its core is a bionic Tri-System architecture comprising a VLM brain for global reasoning, a VLA cerebellum for reactive execution, and a lightweight visual Critic. By continuously monitoring the workspace, the Critic dynamically routes control authority. It sustains rapid closed-loop execution via the VLA for routine subtasks, and adaptively triggers the VLM for replanning upon detecting execution anomalies such as task stagnation or failures. Furthermore, our architecture seamlessly integrates human-inspired rules to intuitively break infinite retry loops. This visually-grounded scheduling minimizes expensive VLM queries, while substantially enhancing system robustness and autonomy in out-of-distribution (OOD) scenarios. Comprehensive experiments on challenging, long-horizon manipulation benchmarks reveal that our approach achieves state-of-the-art performance.
