Search papers, labs, and topics across Lattice.
The paper introduces Gen-n-Val, an agentic data generation framework using Layer Diffusion (LD), LLMs, and VLLMs to address data scarcity and label noise in computer vision tasks. Gen-n-Val employs an LLM-based LD prompt agent to generate high-quality, single-object instance images and masks, and a VLLM-based data validation agent to filter out low-quality samples, with both agents' prompts refined via TextGrad. Experiments demonstrate that Gen-n-Val reduces invalid synthetic data and improves performance on COCO instance segmentation and open-vocabulary object detection benchmarks compared to existing synthetic data approaches.
LLMs and VLLMs can team up to generate synthetic image data so good, it beats state-of-the-art methods and boosts performance on rare classes and open-vocabulary object detection.
Recently, Large Language Models (LLMs) and Vision Large Language Models (VLLMs) have demonstrated impressive performance as agents across various tasks while data scarcity and label noise remain significant challenges in computer vision tasks, such as object detection and instance segmentation. A common solution for resolving these issues is to generate synthetic data. However, current synthetic data generation methods struggle with issues, such as multiple objects per mask, inaccurate segmentation, and incorrect category labels, limiting their effectiveness. To address these issues, we introduce Gen-n-Val, a novel agentic data generation framework that leverages Layer Diffusion (LD), LLMs, and VLLMs to produce high-quality, single-object masks and diverse backgrounds. Gen-n-Val consists of two agents: (1) The LD prompt agent, an LLM, optimizes prompts for LD to generate high-quality foreground instance images and segmentation masks. These optimized prompts ensure the generation of single-object synthetic data with precise instance masks and clean backgrounds. (2) The data validation agent, a VLLM, which filters out low-quality synthetic instance images. The system prompts for both agents are refined through TextGrad. Additionally, we use image harmonization to combine multiple instances within scenes. Compared to state-of-the-art synthetic data approaches like MosaicFusion, our approach reduces invalid synthetic data from 50% to 7% and improves performance by 1% mAP on rare classes in COCO instance segmentation with YOLOv9c and YOLO11m. Furthermore, Gen-n-Val shows significant improvements (7. 1% mAP) over YOLO-Worldv2-M in open-vocabulary object detection benchmarks with YOLO11m. Moreover, Gen-n-Val improves the performance of YOLOv9 and YOLO11 families in instance segmentation and object detection.
