Search papers, labs, and topics across Lattice.
The paper introduces Stable-GFN (S-GFN), a novel approach to LLM red-teaming that addresses the instability and mode collapse issues of standard Generative Flow Networks (GFNs) by eliminating partition function estimation. S-GFN uses pairwise comparisons and robust masking to handle noisy rewards, alongside a fluency stabilizer to avoid gibberish outputs. Experiments across various settings demonstrate that S-GFN achieves superior attack performance and diversity compared to existing methods.
Red-teaming LLMs just got more robust: Stable-GFN sidesteps GFN's notorious instability, unlocking more diverse and effective attacks.
Large Language Model (LLM) Red-Teaming, which proactively identifies vulnerabilities of LLMs, is an essential process for ensuring safety. Finding effective and diverse attacks in red-teaming is important, but achieving both is challenging. Generative Flow Networks (GFNs) that perform distribution matching are a promising methods, but they are notorious for training instability and mode collapse. In particular, unstable rewards in red-teaming accelerate mode collapse. We propose Stable-GFN (S-GFN), which eliminates partition function $Z$ estimation in GFN and reduces training instability. S-GFN avoids Z-estimation through pairwise comparisons and employs a robust masking methodology against noisy rewards. Additionally, we propose a fluency stabilizer to prevent the model from getting stuck in local optima that produce gibberish. S-GFN provides more stable training while maintaining the optimal policy of GFN. We demonstrate the overwhelming attack performance and diversity of S-GFN across various settings.
