Search papers, labs, and topics across Lattice.
This paper introduces Guava, a harness framework for embodied manipulation that integrates high-level reasoning with external modules for perception, planning, and control. Through systematic exploration, the authors identify three essential components for effective embodied agents: iterative perception-reasoning-action loops, semantic action abstractions, and multimodal observations. Experimental results demonstrate that Guava enables a 4B open-source model to achieve performance on par with leading proprietary models, showcasing strong generalization capabilities with minimal training data.
A well-designed harness can unlock powerful embodied manipulation capabilities in compact models, achieving frontier performance with just 2K simulation trajectories.
Language models trained on large-scale vision-language data have demonstrated strong potential for embodied agents. Harnessing models through embodied tools use offers a promising alternative to end-to-end vision-language-action systems by combining high-level reasoning with external modules for perception, planning, and control. However, it remains unclear what makes an effective harness for embodied manipulation, and to what extent such a harness can unlock embodied capabilities in a wide range of reasoning models. In this work, we present Guava, a harness framework for embodied tool use developed through systematic exploration of the design space of agent workflows, action spaces, and observation spaces. Our study identifies three key ingredients for effective embodied agents: iterative perception-reasoning-action loops, semantic action abstractions, and multimodal observations. To understand whether these design principles are universal even to small models, we develop an end-to-end training pipeline that distills embodied manipulation capabilities into a 4B open-source model using fewer than 2K trajectories collected entirely in simulation. Experimental results in both simulation and real-world environments show performance comparable to frontier proprietary models while exhibiting strong generalization to unseen objects, novel instructions, and long-horizon tasks. Results suggest that a well-designed harness can serve as a scalable, model-agnostic interface for embodied manipulation, enabling strong emergent embodied capabilities in compact open-source models with minimal training data.
