Search papers, labs, and topics across Lattice.
This study investigates the stability of persona-driven generations (PDGs) in multiple-choice question answering (MCQA) tasks, addressing a notable gap in existing research. By developing three distinct metrics to evaluate performance, outcome, and question correctness stability, the authors reveal that instability is significantly influenced by model family, size, and question domain, particularly in math and commonsense queries. The findings highlight that task prompt format contributes more to prediction instability than other hyperparameters, emphasizing the critical need to assess hyperparameter effects on PDGs for reliable performance.
Instability in persona-driven generations can vary dramatically across model families and question types, with math and commonsense tasks proving particularly volatile.
Persona-driven generations (PDGs) have seen prolific use in research and industry applications, where a large language model (LLM) takes on a 'persona' while completing some task. While persona expressed through free-form text (like dialogue) has substantial work investigating stability or consistency, relatively, persona expressed in non-text-heavy outputs (like in multiple-choice question answering, or MCQA) is often overlooked. We work to address this gap, seeking to understand the instability of LLM PDGs in MCQA tasks. We develop three metrics investigating the performance, outcome, and question correctness stability, evaluating three distinct dimensions. Using these metrics, we find that instability varies consistently between model families and model size, and across question domains, with math/commonsense questions leading to greater instability. We also find task prompt format introduces more prediction instability than other hyperparameters, like temperature. Finally, we find that instability is related to task accuracy, and using our instability metrics, find different experimental settings that result in different best and worst personas for tasks, despite their similarity. This reveals the importance of checking hyperparameter instability in PDGs.
