Search papers, labs, and topics across Lattice.
This paper introduces DeepBias, an adaptive framework designed to probe social biases in Large Vision-Language Models (LVLMs) through a dynamic "generation-evolution-probing" loop. By utilizing a generative ProposerAgent and a skill-driven DiggerAgent, the framework iteratively synthesizes and rewrites test data based on model responses, revealing deeper biases that static datasets fail to capture. The effectiveness of DeepBias is validated through comprehensive experiments, establishing a new benchmark, DeepBiasBench, for evaluating LVLM vulnerabilities across various architectures.
Static bias evaluations miss the mark; DeepBias reveals hidden vulnerabilities in LVLMs through an adaptive probing process that evolves with the model's responses.
While Large Vision-Language Models (LVLMs) demonstrate remarkable capabilities, they remain highly susceptible to embedded social biases. Existing bias evaluation protocols predominantly rely on static datasets, which provide only a superficial assessment, as their fixed test cases cannot adaptively evolve to measure the true depth and limits of model vulnerabilities. We introduce DeepBias, an adaptive framework for the in-depth probing of social biases in LVLMs with carefully designed agents. Our approach operates through a dynamic''generation-evolution-probing''loop. First, a generative ProposerAgent synthesizes test data and is iteratively updated via Direct Preference Optimization (DPO) based on the target LVLM's responses, exploring model-specific failure modes. Second, an autonomous skill-driven DiggerAgent rewrites each test data across multiple probing turns, adaptively selecting from a curated skill library of deepening and rewriting strategies. At each turn, this process is conditioned on the model's previous response, enabling progressively deeper biases to be exposed. Furthermore, we build a benchmark named DeepBiasBench using our framework. By employing an ensemble of five diverse state-of-the-art LVLMs as anchors, the benchmark captures vulnerabilities shared across architectures. Comprehensive experiments demonstrate the effectiveness of our framework and show that DeepBias provides a challenging benchmark for in-depth bias evaluation, establishing an evolutionary paradigm for LVLM safety assessment.
