Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors present P2P-T, an object-centric framework that learns precise robotic tool use directly from unpaired human video demonstrations without requiring cross-domain alignment data. By pretraining an object-centric world model on automated pose extractions from foundation models, the approach provides structured geometric priors to a low-level execution policy. With minimal task-specific fine-tuning, the system achieves a 73% performance improvement over prior state-of-the-art baselines on complex, contact-rich real-world tool manipulation tasks.
Robots can master delicate tool manipulation directly from unpaired human videos, achieving a 73% performance leap over SOTA by anchoring control in object-centric 3D pose priors rather than raw end-to-end pixels.
Scaling up robotic manipulation is primarily bottlenecked by the scarcity of real-world robot data. While recent approaches leverage human video demonstrations to mitigate this shortage, they remain computationally expensive and still rely on paired human-robot data for domain alignment. Although current state-of-the-arts excel at long-horizon tasks, they struggle with the delicate and precise control required for complex tool manipulation. To overcome these limitations, we introduce P2P-T, from Pixel to Poses for Tool Manipulation, a data-efficient, object-centric framework that learns tool use directly from human demonstrations. P2P-T bridges the cognitive and physical execution gap through a two-stage approach. First, pretraining an object-centric world model to extract stable pose priors; second, integrating these priors into an efficient, pose-aware low-level policy. By utilizing a robust automated data processing pipeline powered by modern foundation models, P2P-T completely bypasses the need for human-robot aligned data. This reduces overall training overhead drastically. With minimal per-task fine-tuning, our framework achieves a 73% improvement over the previous state of the art in execution performance on complex, real-world tool manipulation tasks that currently remain out of reach for standard large-scale pretrained models.