Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
CIM-Embed learns instruction-conditioned multimodal embeddings enabling flexible clustering of objects based on varying criteria. By employing supervised contrastive learning with heterogeneous structured batches, the framework directly maps multimodal inputs and instructions to embeddings. This approach achieves a mean V-measure of 95.01 on the ProductMIC benchmark, significantly outperforming existing general-purpose embedding baselines for instruction-conditioned clustering.
Multimodal clustering can now adapt to arbitrary instructions, achieving near-perfect V-measures by directly embedding images, text, and instructions.
Multimodal objects can require different partitions under different instructions, as when products in an e-commerce catalog are grouped by color, material, or both. Retrieval-oriented embeddings may not recover these partitions, while generative clustering often relies on intermediate descriptions or iterative decisions. This paper presents CIM-Embed, a framework for learning instruction-conditioned multimodal embeddings for clustering. Supervised contrastive learning uses same-group positives and negatives from other groups, optionally including other tasks; heterogeneous structured batches accommodate tasks with different group counts. At inference, the model maps each object's image, text, and instruction directly to an embedding; HDBSCAN then forms a partition without the ground-truth cluster count. We construct ProductMIC from product metadata in Amazon Reviews 2023, comprising training data and an evaluation benchmark for instruction-conditioned clustering. The benchmark contains 5,373 products and 449 single-attribute and compositional grouping tasks spanning 50 leaf categories excluded from our training. With full-dimensional embeddings and HDBSCAN, direct within-task SupCon training achieves a mean per-task V-measure of 95.01, compared with 53.06 for WeMM 9B, the strongest general-purpose embedding baseline evaluated. Frozen product-trained encoders also improve out-of-domain conditional grouping, while controlled transfer experiments distinguish partition transfer from instruction response. We will release the code and dataset upon acceptance.