Search papers, labs, and topics across Lattice.
This paper introduces GarmentPile++, a garment retrieval pipeline designed to retrieve single garments from cluttered piles using vision-language reasoning for home-assistant robotics. The pipeline integrates a Segment Anything Model (SAM2) for garment segmentation, a VLM for high-level reasoning based on visual cues, and a dual-arm cooperation framework for handling large or sagging garments. Experiments in both real-world and simulated environments demonstrate the pipeline's effectiveness in executing safe and clean garment retrieval based on language instructions.
Finally, a robot can reliably pick out that specific shirt from a messy pile, thanks to a new vision-language pipeline that reasons about garment affordances.
Garment manipulation has attracted increasing attention due to its critical role in home-assistant robotics. However, the majority of existing garment manipulation works assume an initial state consisting of only one garment, while piled garments are far more common in real-world settings. To bridge this gap, we propose a novel garment retrieval pipeline that can not only follow language instruction to execute safe and clean retrieval but also guarantee exactly one garment is retrieved per attempt, establishing a robust foundation for the execution of downstream tasks (e.g., folding, hanging, wearing). Our pipeline seamlessly integrates vision-language reasoning with visual affordance perception, fully leveraging the high-level reasoning and planning capabilities of VLMs alongside the generalization power of visual affordance for low-level actions. To enhance the VLM's comprehensive awareness of each garment's state within a garment pile, we employ visual segmentation model (SAM2) to execute object segmentation on the garment pile for aiding VLM-based reasoning with sufficient visual cues. A mask fine-tuning mechanism is further integrated to address scenarios where the initial segmentation results are suboptimal. In addition, a dual-arm cooperation framework is deployed to address cases involving large or long garments, as well as excessive garment sagging caused by incorrect grasping point determination, both of which are strenuous for a single arm to handle. The effectiveness of our pipeline are consistently demonstrated across diverse tasks and varying scenarios in both real-world and simulation environments. Project page: https://garmentpile2.github.io/.
