Search papers, labs, and topics across Lattice.
This paper introduces Transformer Transformer, a novel diffusion transformer model designed for motion-conditioned robot co-design, which generates optimized robot designs that can track target trajectories based on human demonstrations. By utilizing a unified tokenization approach called RoboTokens, the model effectively operates across various robot embodiments and applications, enabling reward-agnostic state and action predictions that can be adapted to specific reward functions. Experimental results demonstrate significant improvements in performance and efficiency, including a 70% reduction in tracking error for a fabricated ALOHA design compared to its predecessor.
Zero-shot optimization of robot designs leads to a staggering 70% reduction in tracking error, showcasing the potential of motion-conditioned co-design.
An often overlooked factor of robot manipulation performance is the embodiment of the robot itself. Motivated by this problem, we study motion-conditioned robot co-design, where the goal is to generate complete robot designs that track target end-effector trajectories (from human demonstrations) while optimizing user-defined rewards. We introduce Transformer Transformer, a diffusion transformer trained on RoboTokens, a unified tokenization of robot embodiments, states, and actions. The same architecture can be used across embodiment spaces (e.g., wheeled bimanual, quadrupeds, humanoids) and use cases (embodiment generation, cross embodiment controller). Rather than overfitting to one reward function, Transformer Transformer is a dynamics model, whose reward-agnostic state and action predictions can be converted into reward-specific value predictions. These value predictions are used to steer embodiment diffusion towards high value robot designs, through a procedure we call Dynamics Self-Guidance. Experiments across multiple design spaces show zero-shot optimization of unseen rewards and trajectories, improving performance and runtime over the evolutionary baseline. Finally, we fabricated an optimized ALOHA design, which reduced tracking error by over 70% compared to the original design.
Stanford HAI