Search papers, labs, and topics across Lattice.
This paper introduces Sparse2Act, a framework that leverages task-space end-effector actions as geometric supervision to pretrain sparse point-cloud encoders for robot manipulation. By aligning sparse 3D representations with action data, the method enables effective cross-domain transfer and retains flexibility in downstream policy architectures. The approach achieves significant performance improvements on both the LIBERO-10 and Meta-World-5 benchmarks, with a notable average success rate of 72.5% in real-world tasks after simulation pretraining.
Robot actions can serve as powerful geometric supervision, enabling sparse 3D representations that are both reusable and effective across diverse manipulation tasks.
Explicit 3D representations are attractive for manipulation because they expose object shape, workspace geometry, and robot-object relations in metric coordinates. However, sparse 3D encoders are often learned through downstream task objectives, tying the representation to a particular data distribution, policy architecture, and action parameterization. We introduce Sparse2Act, an observation-action alignment framework for pretraining sparse point-cloud encoders. The key idea is to use task-space end-effector actions as geometric supervision: masked sparse 3D tokens are trained to organize scene features around the workspace motion paired with the observation. After pretraining, only the encoder initialization is reused by downstream policies, allowing them to retain their own architectures and action spaces, including joint-space commands. On the LIBERO-10 benchmark, our method achieves 86.9% average success after 500 fine-tuning steps. The same pretrained encoder supports LIBERO-to-Meta-World cross-domain transfer, achieving 73.4% average success on the Meta-World-5 benchmark. Ablations on the objective and decoder capacity show that the gains come from the masked action-alignment signal and remain useful across downstream action decoders. In real-world experiments, simulation pretraining followed by limited real-data fine-tuning achieves an average success rate of 72.5% across four tasks, demonstrating effective sim-to-real transfer. These results suggest that robot actions can provide compact geometric supervision for reusable sparse 3D representations.
