Search papers, labs, and topics across Lattice.
This paper investigates the vulnerability of open-weight large language models (LLMs) to prefill attacks, where attackers manipulate initial response tokens to elicit harmful behavior. Through an extensive empirical evaluation of over 20 prefill strategies across various model families, the authors demonstrate the consistent effectiveness of these attacks in bypassing internal safeguards. The study reveals a significant and previously underexplored vulnerability in open-weight LLMs, highlighting the need for targeted defenses.
Open-weight LLMs are systematically vulnerable to prefill attacks, a largely unexplored attack vector that bypasses internal safeguards even in state-of-the-art reasoning models.
As the capabilities of large language models continue to advance, so does their potential for misuse. While closed-source models typically rely on external defenses, open-weight models must primarily depend on internal safeguards to mitigate harmful behavior. Prior red-teaming research has largely focused on input-based jailbreaking and parameter-level manipulations. However, open-weight models also natively support prefilling, which allows an attacker to predefine initial response tokens before generation begins. Despite its potential, this attack vector has received little systematic attention. We present the largest empirical study to date of prefill attacks, evaluating over 20 existing and novel strategies across multiple model families and state-of-the-art open-weight models. Our results show that prefill attacks are consistently effective against all major contemporary open-weight models, revealing a critical and previously underexplored vulnerability with significant implications for deployment. While certain large reasoning models exhibit some robustness against generic prefilling, they remain vulnerable to tailored, model-specific strategies. Our findings underscore the urgent need for model developers to prioritize defenses against prefill attacks in open-weight LLMs.
