Search papers, labs, and topics across Lattice.
The paper introduces RoboSolver, a multi-agent framework leveraging LLMs and VLMs to solve robotic arm problems by integrating textual and visual inputs with computational tools. RoboSolver automates tasks such as forward and inverse kinematics, velocity/acceleration computation, 3D simulation, and motion control based on user queries. Benchmarking against GPT-4o, DeepSeek-V3.2, Claude-Sonnet-4.5, and Gemini 2.5 Pro VLM, RoboSolver consistently outperformed the raw models, achieving up to 0.97 accuracy in various robotic tasks.
LLMs plus a multi-agent framework can solve complex robotics problems with near-perfect accuracy, far surpassing the capabilities of raw LLMs alone.
This study proposes an intelligent multi-agent framework built on LLMs and VLMs and specifically tailored to robotics. The goal is to integrate the strengths of LLMs and VLMs with computational tools to automatically analyze and solve problems related to robotic manipulators. Our developed framework accepts both textual and visual inputs and can automatically perform forward and inverse kinematics, compute velocities and accelerations of key points, generate 3D simulations of the robot, and ultimately execute motion control within the simulated environment, all according to the user's query. To evaluate the framework, three benchmark tests were designed, each consisting of ten questions. In the first benchmark test, the framework was evaluated while connected to GPT-4o, DeepSeek-V3.2, and Claude-Sonnet-4.5, as well as their corresponding raw models. The objective was to extract the forward kinematics of robots directly from textual descriptions. The results showed that the framework integrated with GPT-4o achieved the highest accuracy, reaching 0.97 in computing the final solution, whereas the raw model alone attained an accuracy of only 0.30 for the same task. Similarly, for the other two models, the framework consistently outperformed the corresponding raw models in terms of accuracy. The second benchmark test was identical to the first, except that the input was provided in visual form. In this test, the GPT-4o LLM was used alongside the Gemini 2.5 Pro VLM. The results showed that the framework achieved an accuracy of 0.93 in obtaining the final answer, which is approximately 20% higher than that of the corresponding raw model. The third benchmark test encompassed a range of robotic tasks, including simulation, control, velocity and acceleration computation, as well as inverse kinematics and Jacobian calculation, for which the framework achieved an accuracy of 0.97.
