Search papers, labs, and topics across Lattice.
This paper introduces AmodalCG, a framework that leverages Multimodal Large Language Models (MLLMs) to improve amodal completion, the task of reconstructing occluded parts of objects in images. The framework selectively invokes MLLM guidance based on the degree of occlusion, using the MLLM to reason about the extent and content of missing regions. A visual generative model then refines the MLLM's guidance, leading to significant improvements over existing amodal completion methods on real-world images.
MLLMs can now guide visual generative models to imagine what's hidden behind objects, significantly boosting amodal completion performance.
With the widespread adoption of autonomous vehicles and robotics, amodal completion, which reconstructs the occluded parts of people and objects in an image, has become increasingly crucial. Just as humans infer hidden regions based on prior experience and common sense, this task inherently requires physical knowledge about real-world entities. However, existing approaches either depend solely on the image generation ability of visual generative models, which lack such knowledge, or leverage it only during the segmentation stage, preventing it from explicitly guiding the completion process. To address this, we propose AmodalCG, a novel framework that harnesses the real-world knowledge of Multimodal Large Language Models (MLLMs) to guide amodal completion. Our framework first assesses the extent of occlusion to selectively invoke MLLM guidance only when the target object is heavily occluded. If guidance is required, the framework further incorporates MLLMs to reason about both the (1) extent and (2) content of the missing regions. Finally, a visual generative model integrates these guidance and iteratively refines imperfect completions that may arise from inaccurate MLLM guidance. Experimental results on various real-world images show impressive improvements compared to all existing works, suggesting MLLMs as a promising direction for addressing challenging amodal completion.
