Search papers, labs, and topics across Lattice.
This paper introduces SSI-Policy, a modular framework that utilizes a Structured Scene Interface (SSI) to enhance vision-language robotic manipulation in low-data scenarios. By integrating monocular depth features, language-grounded object layouts, and instruction-conditioned 2D motion trajectories into a single RGB-only representation, SSI-Policy enables effective learning from minimal demonstrations. The framework outperforms the best prior method by nearly 15% on the LIBERO benchmark with only 10 demonstrations per task, showcasing its efficiency and robustness across various real-world manipulation tasks.
SSI-Policy achieves a nearly 15% improvement in robotic manipulation performance with just 10 demonstrations, challenging the need for extensive training data.
Real-world robotic manipulation demands spatial grounding, task-aware reasoning, and precise control. Learning such capabilities becomes particularly challenging in the low-data regime. Prior methods often trade off scalable task-level reasoning and explicit physical structure: video-based approaches can drift geometrically over long horizons, 3D approaches often require depth sensing, and many flow/trajectory interfaces emphasize motion without an explicit RGB-only geometric representation. We introduce SSI-Policy, a modular framework built around a Structured Scene Interface (SSI) -- a unified, RGB-only intermediate representation that jointly encodes monocular depth features, language-grounded object layouts, and instruction-conditioned 2D motion trajectories. Critically, SSI is robot-agnostic and trainable from action-free video, decoupling perception from control so that the downstream policy can learn from few demonstrations. On the LIBERO benchmark with only 10 demonstrations per task, SSI-Policy improves over the strongest prior method by nearly 15\% and remains competitive with 50-demo methods that leverage large-scale external pretraining. Ablations show that geometric and motion cues provide complementary benefits within the shared interface. We further validate on 13 real-world tasks spanning spatial reasoning, cross-embodiment transfer, and contact-rich manipulation.
