Search papers, labs, and topics across Lattice.
This paper introduces a modular framework for the CMU Vision-Language-Action Challenge, enabling autonomous agents to execute complex tasks from natural language instructions. The system integrates a perception pipeline that utilizes OwlViT embeddings to create a semantic voxel map and a language pipeline that classifies commands using a Vision-Language Model, operating under a strict 500-second exploration limit. Key results show that the framework effectively translates user queries into actionable robotic tasks, showcasing significant advancements in human-robot interaction and task execution in indoor environments.
Bridging the gap between human language and robotic action, this framework allows agents to understand and execute complex tasks in real-time, even with partial environmental maps.
This paper presents an integrated system for the CMU Vision-Language-Action (VLA) Challenge, designed to enable an autonomous agent to perform complex tasks based on natural language instructions. Our framework employs a modular architecture that orchestrates environment mapping, question processing, and navigation. The system operates in two parallel streams: a perception pipeline that constructs a semantic voxel map from real-time camera feeds using OwlViT embeddings, and a language pipeline that classifies user commands with a Vision-Language Model. The mapping is time-constrained; the system proceeds with a partial map if a 500-second exploration limit is reached. The classified query is then grounded in the geometric and semantic context of the map to generate a detailed prompt for the VLM. This yields an actionable output, demonstrating a capable solution for bridging the gap between human language and robotic action.
