Search papers, labs, and topics across Lattice.
The paper introduces NativeEmbodied, a new benchmark for VLM-driven embodied agents that uses a unified, native low-level action space to address limitations of existing benchmarks that rely on high-level commands or discretized action spaces. NativeEmbodied includes three high-level tasks and four low-level tasks targeting fundamental embodied skills, enabling joint evaluation across task and skill granularities. Experiments using state-of-the-art VLMs on NativeEmbodied reveal deficiencies in fundamental embodied skills that limit performance on high-level tasks, highlighting key challenges for current VLM-driven embodied agents.
Current VLM-driven embodied agents struggle with fundamental skills like navigation and object manipulation when evaluated in realistic, low-level action spaces, severely hindering their performance on complex tasks.
Recent advances in vision-language models (VLMs) have shown promise for human-level embodied intelligence. However, existing benchmarks for VLM-driven embodied agents often rely on high-level commands or discretized action spaces, which are non-native settings that differ markedly from real-world control. In addition, current benchmarks focus primarily on high-level tasks and lack joint evaluation and analysis at both low and high levels. To address these limitations, we present NativeEmbodied, a challenging benchmark for VLM-driven embodied agents that uses a unified, native low-level action space. Built on diverse simulated scenes, NativeEmbodied includes three representative high-level tasks in complex scenarios to evaluate overall performance. For more detailed analysis, we further decouple the skills required by complex tasks and construct four types of low-level tasks, each targeting a fundamental embodied skill. This joint evaluation across task and skill granularities enables fine-grained assessment of embodied agents. Experiments with state-of-the-art VLMs reveal clear deficiencies in several fundamental embodied skills, and further analysis shows that these bottlenecks significantly limit performance on high-level tasks. NativeEmbodied highlights key challenges for current VLM-driven embodied agents and provides insights to guide future research.
Tsinghua AI