Search papers, labs, and topics across Lattice.
This paper introduces a modular framework for zero-shot robotic manipulation that integrates high-level reasoning with low-level kinematics, enabling robots to perform tasks guided by natural language in semi-structured environments. The architecture consists of three stages: visual perception using FastSAM and Set-of-Mark prompting for generating grounded visual anchors, semantic interpretation through a Large Language Model (LLM) that translates human directives, and task execution via a Task Orchestrator that creates collision-free trajectories. Evaluated in two zero-shot scenarios, the framework achieved a 62% success rate, showcasing its effectiveness in executing complex actions without requiring prior training or manual programming.
Achieving a 62% success rate in zero-shot robotic manipulation, this framework effectively translates natural language into actionable tasks without any prior training.
This paper presents a modular training-free framework for zero-shot, language-guided robotic manipulation in semi-structured environments. The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution. To overcome the spatial ambiguity and semantic hallucinations inherent in standard Vision-Language Models (VLMs), the perception module employs FastSAM and Set-of-Mark (SoM) prompting to dynamically generate grounded, alphanumeric visual anchors. The same foundation model then operates purely as a Large Language Model (LLM) to act as a semantic router, translating unconstrained human directives into verifiable, reconfigurable configurations. Finally, these configurations are dynamically parsed by a Task Orchestrator into MoveIt Task Constructor (MTC) to generate collision-free trajectories. The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning, achieving a 62% end-to-end task success rate across both scenarios, demonstrating its capacity to reliably execute complex physical actions without domain-specific training or manual coordinate programming.
