Search papers, labs, and topics across Lattice.
This paper introduces See2Act, an innovative imitation learning framework that integrates action prediction with actively inferred viewpoints to enhance robot performance in environments with occluded objects. By conditioning action decisions on a sequence of viewpoints derived from offline demonstrations, the model effectively learns to navigate and manipulate objects even when visibility is compromised. Empirical results show that See2Act improves task performance by up to 34% in simulated environments and achieves successful zero-shot sim-to-real transfer in real-world pick-and-place tasks, demonstrating its robustness in partial observability scenarios.
Robots can now learn to see and act simultaneously, achieving up to 34% better performance in occluded environments.
Most imitation learning methods assume full observability in table-top settings. In practice, objects are often occluded, requiring robots to both search and act, and learning this coupled behavior from limited demonstrations remains challenging. We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively-inferred viewpoints at test time, by coupling action denoising with viewpoint refinement. The policy is trained using camera poses anchored to keyframe actions from offline demonstrations, enabling implicit learning of where to see, while learning how to act. We empirically demonstrate that in Ravens the policy recovers informative viewpoints under severe occlusions, and on RLBench tasks it improves performance by up to 34% over prior methods. In the real world, we collect 50 demonstrations in a digital twin and achieve zero-shot sim-to-real transfer on pick-and-place tasks using depth observations. The policy handles significant occlusions, showing that learned viewpoint reasoning enables robust manipulation under partial observability.
