Search papers, labs, and topics across Lattice.
This paper introduces a hybrid pipeline that combines text-to-image (T2I) generation with context-aware image-to-image (I2I) editing to enhance long-tailed instance segmentation. By leveraging a teacher-student scheme for label reliability and the novel VRAIN editor for rare-class augmentation, the approach effectively addresses the limitations of existing data synthesis methods. The results demonstrate significant improvements on the LVIS benchmark, with overall average precision (AP) increasing by up to 4.0 points and rare-class AP by up to 9.5 points, showcasing the method's effectiveness in generating high-quality, contextually relevant data.
A novel hybrid approach boosts rare-class instance segmentation performance by up to 9.5 AP points by intelligently combining T2I generation with context-aware I2I editing.
Large-vocabulary instance segmentation is constrained by long-tailed category distributions and fine-grained inter-class ambiguity. While data synthesis offers a promising alternative, current paradigms have complementary limitations: text-to-image (T2I) methods inherit noisy pseudo-labels and struggle on rare classes, whereas copy-paste methods compromise contextual realism. To address these issues, we propose a hybrid pipeline coupling T2I generation with context-aware image-to-image (I2I) editing. The T2I branch provides broad category and scene diversity, while a teacher-student scheme ensures label reliability by selectively retaining only prompt-specified categories. To strengthen supervision for rare classes, we introduce VRAIN (Verified Rare-class Augmentation via INstructed editing), a novel I2I editor. VRAIN inserts high-confidence instances at semantically appropriate locations within in-the-wild scenes, yielding semantically coherent and visually natural edits that reduce domain gaps and enable targeted augmentation. On the LVIS benchmark, our method surpasses existing baselines, improving overall AP by up to +4.0 points and rare-class AP by up to +9.5 points, while scaling effectively with backbone capacity. Our project page is available at https://seokhunchoi.github.io/TMI
