Search papers, labs, and topics across Lattice.
This paper introduces CoStream, a novel framework that composes simple, independent behaviors to achieve high-precision control in complex manipulation tasks. By integrating foundation models with diverse sensing modalities, CoStream orchestrates semantic, predictive, and reactive behaviors that can be dynamically recombined, overcoming the limitations of rigid pipelines and monolithic policies. The approach is validated across eight real-world tasks, demonstrating significant improvements in performance, particularly in contact-rich assembly and object transfer scenarios, while maintaining robustness against manual perturbations.
Complex manipulation capabilities can be achieved by dynamically composing simple behaviors, leading to unprecedented precision and adaptability in real-world tasks.
Long-horizon, contact-rich complex manipulation tasks, such as seating a GPU into a PCIe slot, demand both millimeter high precision and out-of-the-box generalization to new tasks. Existing paradigms struggle to satisfy both: classical pipelines use brittle, task-specific interfaces to achieve high-precision control but require costly pipeline redesigns to adapt to new tasks, whereas monolithic end-to-end policies provide better generalization but lack high precision on complex, out-of-distribution tasks unless retrained with new data. Both paradigms share an implicit assumption: once a manipulation capability is acquired, it must be deployed as a rigid pipeline or monolithic whole, rather than being freely decomposed and recomposed. In this paper, we show that complex manipulation capabilities can emerge naturally from the composition of simple, independent behaviors. Rather than deploying a monolithic policy or a rigid pipeline, we propose \ourshort, a framework orchestrating foundation models and diverse sensing modalities into multiple composable core behaviors: a semantic behavior extracting spatial constraints via foundation models; a predictive behavior forecasting trajectories by tracking keypoints in imagined videos; and a reactive behavior providing high-frequency tactile and force corrections. On a shared $SE(3)$ interface, these outputs compose by right-multiplication into a single pose command at each control step, executed by a compliant controller. We demonstrate \ourshort on 8 real-world tasks spanning everyday manipulation and precision assembly, with the strongest gains in contact-rich assembly and object transfer, and show robust recovery from manual perturbations during execution. {Website:} https://costream-simple.github.io
MIT CSAIL
Stanford HAI