Search papers, labs, and topics across Lattice.
This paper introduces CAD-Prompted SAM3, a novel instance segmentation framework that leverages CAD models as prompts to overcome limitations of language and appearance-based prompting, particularly for industrial objects. The method uses multi-view renderings of CAD models to provide geometry-based conditioning to SAM3, enabling segmentation independent of surface appearance. Experiments using synthetic data demonstrate the effectiveness of CAD-prompted segmentation for objects with varying materials and finishes.
Forget language and appearance: CAD models can now directly prompt accurate instance segmentation of industrial objects, even with diverse surface properties.
Verbal-prompted segmentation is inherently limited by the expressiveness of natural language and struggles with uncommon, instance-specific, or difficult-to-describe objects: scenarios frequently encountered in manufacturing and 3D printing environments. While image exemplars provide an alternative, they primarily encode appearance cues such as color and texture, which are often unrelated to a part's geometric identity. In industrial settings, a single component may be produced in different materials, finishes, or colors, making appearance-based prompting unreliable. In contrast, such objects are typically defined by precise CAD models that capture their canonical geometry. We propose a CAD-prompted segmentation framework built on SAM3 that uses canonical multi-view renderings of a CAD model as prompt input. The rendered views provide geometry-based conditioning independent of surface appearance. The model is trained using synthetic data generated from mesh renderings in simulation under diverse viewpoints and scene contexts. Our approach enables single-stage, CAD-prompted mask prediction, extending promptable segmentation to objects that cannot be robustly described by language or appearance alone.
CMU ML