Search papers, labs, and topics across Lattice.
This paper introduces a distributed conversational framework that enables human-robot collaborative manipulation by integrating local language and vision-language models with a ROS 2 execution stack. The system effectively translates free-form user commands into structured action requests while maintaining a responsive control loop, allowing for flexible deployment across various hardware setups. Experimental results on a Franka FR3 platform demonstrate the framework's reliability and latency performance under varying scene ambiguities, showcasing the advantages of different LLM/VLM configurations in enhancing task execution.
Free-form user commands can be transformed into precise robotic actions, improving collaborative manipulation in dynamic environments.
This paper presents a distributed conversational framework for human-robot collaborative manipulation that integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack. Language understanding, visual grounding, orchestration, and motion execution run as separate ROS 2 nodes, enabling flexible deployment across distributed hardware while maintaining a responsive control loop. From free-form user commands, the system generates structured action requests for pick, place, and handover. It uses a VLM to return image-space targets, which are converted into metric robot-frame goals using depth and calibration. A web dashboard exposes intermediate intent and grounding overlays (pixel, depth, and robot-frame) and requires explicit operator confirmation before any motion is executed. Experiments on a Franka FR3 platform evaluate end-to-end task reliability and latency under increasing working table scene ambiguity and compare alternative LLM/VLM configurations in the same pipeline. Code and full documentation are available at [github.com/cogrob-tuni/franka-llm](https://github.com/cogrob-tuni/franka-llm).
