Search papers, labs, and topics across Lattice.
This paper introduces VISA, a novel agentic framework for multimodal instruction synthesis that operates as a self-evolving loop, enhancing the quality and diversity of training data. By integrating feedback from failed samples and leveraging a persistent memory system, VISA adapts its constraint space to improve instruction generation and verification iteratively. Experiments demonstrate that VISA significantly outperforms existing synthesis methods on MM-IFEval while maintaining general multimodal capabilities across multiple benchmarks.
VISA's self-evolving framework not only enhances multimodal instruction synthesis but also adapts in real-time to improve training data quality and model performance.
Multimodal instruction-following models require training data that is accurate, diverse, verifiable, and challenging. Existing synthesis pipelines typically follow a one-pass generate-and-filter paradigm, discarding feedback from failed samples, verifier outcomes, and target-model errors. We present VISA (Visual Instruction Synthesis Agent), an agentic framework that reformulates multimodal instruction synthesis as a self-evolving loop. At each round, VISA analyzes an image to filter incompatible constraints and discover new verifiable ones, samples diversity- and difficulty-aware constraint sets from persistent memory, generates candidate instructions, and verifies the resulting samples with executable tools and structured large language model judges. Failed samples trigger diagnostic-guided recovery, while accepted samples are probed against the target model to estimate difficulty. The resulting verifier signals and target-model failure profiles are written back to memory, allowing subsequent rounds to adaptively expand the constraint space, reduce template repetition, and focus on unresolved model weaknesses. The same verifier contracts further provide reward signals for reinforcement learning without a separately trained reward model. Experiments on MM-IFEval show that VISA consistently improves multimodal instruction following over strong baselines, while preserving general multimodal capability across seven public benchmarks.
