Search papers, labs, and topics across Lattice.
The paper introduces BLOOM-WILT, an innovative auditing pipeline designed to elicit rare behaviors from language models more efficiently than traditional methods. By optimizing the sampling process through logit tilting and adaptive conversational strategies, BLOOM-WILT significantly enhances the detection of problematic behaviors during automated audits. Evaluations show that it outperforms baseline auditors across multiple models and behaviors, achieving a remarkable increase in behavior presence from 51% to 100% for self-harm encouragement in Qwen3.5-4B.
Rare behaviors in language models can be elicited with 100% presence using BLOOM-WILT, revolutionizing automated auditing efficiency.
Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions. On the output side, WILT adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt, so that behaviour-relevant generations are sampled ahead of others it finds equally probable when unprompted. We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings. WILT raises average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method we port into the same pipeline at matched compute, without pushing output probability below the baseline's.
