Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To overcome the high-dimensional and heterogeneous action space of humanoid loco-manipulation, the authors present Holo-M, a discrete vision-language-action (VLA) model that directly incorporates action tokens into the LLM vocabulary. The framework decomposes the action space into four distinct body-part tokenizers—end-effector, body, hand, and kinematics—allowing scalable co-training across human egocentric videos, simulation, and humanoid teleoperation without relying on decoupled continuous action heads. Evaluated on the SIMPLE humanoid benchmark, Holo-M achieves state-of-the-art task success rates in both specialist and generalist settings while maintaining real-time inference latency via grouped discrete diffusion decoding.
Full-body humanoid loco-manipulation no longer requires isolated continuous action heads: factoring discrete action tokens across body parts directly within an LLM's native vocabulary outperforms specialized continuous policies when paired with grouped discrete diffusion decoding.
Vision-language-action (VLA) models using discrete action tokens have proven effective for controling robotic arms on manipulation tasks. For a humanoid, however, the whole-body action space -- legs, torso, arms, and hands -- is far higher-dimensional and heterogeneous, raising tokenization, training, and real-time inference challenges that the previous VLA models do not address. We present Holo-M, to our knowledge the first discrete VLA model for humanoid loco-manipulation that intrinsically exploits the language model by extending its vocabulary with action tokens. In this model, we devise a unified action tokenizer that decomposes the humanoid action space into four body-part-specific tokenizers -- end-effector, body, hand, and kinematics -- enabling training across drastically different embodiments and data sources, including humanoid teleoperation, ego-centric human video, and simulation. By extending the language model's vocabulary with these action tokens, we avoid the knowledge-insulation problem inherent to the models that use separate continuous action experts. To meet real-time control requirements, we decode each body part's action tokens through grouped discrete diffusion decoding, rather than using autoregression on the action tokens. We have conducted extensive experiments on the SIMPLE humanoid loco-manipulation benchmark, in which Holo-M achieves the highest success rates in both the generalist and specialist evaluations, leading the second best by significant margins. We will release all the code and model weights.