Search papers, labs, and topics across Lattice.
This paper introduces a novel approach called "overthinking," which amplifies reasoning weights in language models to enhance the extraction of hidden information during black box audits. By defining an overthinking model that combines parameters from a non-reasoning instruct model and a reasoning-distilled model, the authors demonstrate that this amplification can reveal hidden secrets or unintended behaviors in models up to ten times more frequently than traditional reasoning methods. The findings highlight the importance of targeted reasoning amplification and the varying effectiveness of different perturbation strategies based on the type of hidden information being sought.
Amplifying reasoning weights can uncover hidden model secrets up to 10 times more effectively than standard approaches.
Black box auditing of language models is an essential pre-deployment tool, but it may miss subtle forms of misalignment and hidden information. To better elicit hidden information during an auditing process, we introduce \emph{overthinking}: the process of using reasoning task vectors to amplify the propensity to think out loud of reasoning models. Given the parameters of a non-reasoning instruct model $M$ and reasoning-distilled model $R$, we define the \emph{overthinking model} as $\boldsymbol{\theta}_{\mathcal{O}_\alpha} = \boldsymbol{\theta}_{\mathcal{M}} + \alpha(\boldsymbol{\theta}_{\mathcal{R}} - \boldsymbol{\theta}_{\mathcal{M}})$, where $\alpha>1$ amplifies reasoning beyond the pure reasoning model $R$. Additionally, we introduce new layer-wise attenuation strategies that selectively amplify reasoning without losing quality and coherence of model outputs. We demonstrate that overthinking models are more likely to reveal hidden information across four experimental settings, across 2B-32B models. Our findings suggest that reasoning amplification may surface secrets or unintended behaviors acquired during training up to $10\times$ more frequently than the original reasoning model. How secrets surface depends on the secret type: some require perturbation along the reasoning direction, while others yield to any sufficiently large weight perturbation.
