Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of "pigeonholing," where Large Language Models (LLMs) experience performance degradation due to bad contextual prompts, leading to mode collapse. Through experiments across ten tasks and various models, the authors demonstrate that incorrect contexts can cause significant drops in performance (up to 40%) and narrow the range of generated responses, particularly as conversation turns increase. To address this issue, they introduce a method called RLVR with synthetic errors, which enhances model performance by 43-60% in adverse contexts compared to standard reinforcement learning techniques.
Bad prompts can lead to a staggering 40% drop in LLM performance, revealing a critical vulnerability in in-context learning.
While in-context learning is generally shown to be effective in Large Language Models (LLMs), bad contexts can cause performance degradation and mode collapse, a phenomenon we call "pigeonholing." **Unintentionally bad** contexts can happen without malicious jailbreaking intents: For example, a user asks the model to justify an incorrect math theorem or fails to correct the model's buggy code. Specifically, we investigate ``pigeonholing" in two scenarios: (1) when the user suggests a solution, and (2) when the conversation context includes the assistant's previous (incorrect) responses. Our experiments across 10 verifiable and open-ended tasks with 10 different models show that pigeonholing manifests in several ways: (1) repeating the incorrect answers from context (leading to 38-40% performance drop), (2) converging on a narrow set of answers in coding and text generation without exploring alternatives, and (3) flipping stance on controversial topics to align with the user or the assistant's previous claims. We find that pigeonholing worsens almost monotonically with the number of conversation turns (performance drops by additional 14+% as repeated mistakes increase from 1 to 5), and pigeonholing-induced mode collapse can happen even when the provided example is correct. As a step toward mitigation, we propose RLVR with synthetic errors which improves models by 43-60% under bad contexts compared to vanilla RLVR baselines.
Stanford HAI
UW