Search papers, labs, and topics across Lattice.
This paper introduces InstructMixup, a novel data augmentation technique that enhances the robustness of deep visual models by creating label-consistent training samples from a single visual input. By leveraging a lightweight saliency detector to extract multi-scale salient patches and refining them with an instruction-guided generative model, InstructMixup maintains semantic integrity while minimizing computational overhead. Empirical evaluations across various benchmarks demonstrate that InstructMixup consistently outperforms nine existing augmentation methods, achieving superior performance in both classification and robustness tasks.
InstructMixup achieves superior performance in visual model training by creating robust, label-consistent samples from a single image, outperforming nine competing methods.
In image and video technologies, data augmentation is widely used to improve the generalization of deep visual models, and mixup-based strategies that interpolate between samples have become the dominant approach. However, computing informative mixing regions adds substantial overhead, and blending content across different images frequently disrupts the semantic integrity of the resulting sample. We propose \our{}, a data augmentation method that constructs challenging yet label-consistent training samples entirely within a single visual sample. \our{} first extracts multi-scale salient patches from the sample using a lightweight saliency detector, refines each patch with an instruction-guided generative model, and blends the edited patch back into the non-salient regions of the same sample; because the generative edits are computed once and cached offline, this step adds negligible training cost. To further diversify the learned representation, \our{} injects self-similar fractal structure into the same salient regions at an adaptive ratio, so each training sample carries both fractal and non-fractal structure. We derive a second-order approximation of the resulting vicinal risk, showing that the method simultaneously enforces invariance to the generative edit and suppresses curvature along the perturbed salient directions, and we verify both predictions empirically. We evaluate on small to large backbones for instance Convolutional Neural Networks (CNNs), Vision Transformers (ViTs) and Vision-Language Foundational Models (VLMs) across seven benchmarks covering coarse- and fine-grained classification, robustness to corruption and occlusion, calibration, and transfer and self-supervised learning, InstructMixup outperforms nine competing augmentation methods, surpassing the strongest baseline across all benchmarks.
