Search papers, labs, and topics across Lattice.
The paper introduces SIGMA, a method for automatically generating pixel-level masks for image manipulation localization (IML) by leveraging existing (original, edited) image pairs. SIGMA uses semantic-feature differencing in a vision foundation backbone, refined with an instruction-derived spatial prior, to localize intended edits while accounting for diffusion-induced perturbations and unintended side-effects. Trained in two stages with inpainting supervision and diffusion-domain adaptation, SIGMA significantly outperforms existing mask generators and enables the creation of a large IML training dataset, improving the performance of diverse detectors.
Unlock a treasure trove of free training data: SIGMA turns millions of unannotated image edits into high-quality pixel masks, boosting image manipulation detection by 18%.
Text-driven image editing has advanced rapidly, but reliably localizing these manipulations requires image manipulation localization (IML) models trained on large pixel-annotated datasets, and there is still no low-cost way to obtain such training data at scale. We observe that these data already exist in disguise: public editing datasets contain millions of structurally identical (original, edited) pairs to IML training samples, lacking only pixel-level masks. Recovering these masks automatically is non-trivial: pixel differencing is overwhelmed by diffusion-induced perturbations across all pixels, and instruction-only grounding localizes only what the prompt describes, missing unintended editor side-effects. We propose SIGMA (Semantic-difference Instruction-Grounding Mask Annotator), which performs semantic-feature differencing in a vision foundation backbone and injects an instruction-derived spatial prior into this visual stream via bidirectional cross-modal refinement, amplifying the difference signal at intended-edit regions when the editor faithfully realizes user intent. SIGMA is trained in two complementary stages: Stage I supervises on inpainting masks; Stage II closes the diffusion-domain shift via VAE-roundtrip noise calibration, EMA self-training, and an edit-noise disentanglement loss. SIGMA outperforms existing automatic mask generators on five benchmarks (+12.20% F1, +11.16% IoU). When applied to public editing corpora, it produces a ~1.1M IML training set that improves six diverse detectors by +18.34% F1 across five datasets, turning previously unused editing data into a model-agnostic supervisory resource for IML. We'll release the full codebase as soon as the paper is accepted.
DAMO