Search papers, labs, and topics across Lattice.
This paper introduces a Multi-Agent System (MAS) architecture that integrates compact Vision Language Models (VLMs) for robotic control, specifically designed to operate on onboard hardware without relying on external compute resources. The system effectively manages a multi-purpose autonomous mobile manipulator in a simulated industrial warehouse, successfully executing tasks such as safety inspections and responding to human requests. Key findings reveal that the onboard MAS architecture not only enhances explainability and generalization but also offers a cost-efficient alternative to traditional cloud-dependent systems, validated through hardware-in-the-loop simulations.
A fully onboard multi-agent robotic system can outperform cloud-dependent models in efficiency and task execution while maintaining strong generalization capabilities.
Vision Language Models (VLMs) and Vision Language Action (VLA) models have shown promise in robotic control. Yet, they face significant challenges regarding explainability, generalization, and compute requirements. This paper presents a Multi-Agent System (MAS) architecture that addresses these limitations by deploying specialized agents on onboard hardware - eliminating dependence on external compute. The system controls a multi-purpose autonomous mobile manipulator in a simulated industrial warehouse, fulfilling five task categories: safety inspection, warehouse maintenance, warehouse search, package quality verification, and responding to human requests. Compact VLMs (3-20B parameters) are used throughout, with fine-tuning applied to improve package inspection accuracy. A novel"Megamind"orchestration agent mitigates context retention issues inherent to long-horizon planning with smaller models. The system was validated in a hardware-in-the-loop simulation using an AMD Ryzen(TM) AI mini PC. Results demonstrate that a fully onboard MAS architecture is a viable, cost-efficient alternative to cloud-dependent deployments, with strong potential for real-world transfer. The simulation environment has been released as open source under the Apache 2.0 licence.
