Search papers, labs, and topics across Lattice.
This paper investigates the ability of LLMs to perform stochastic sampling, a crucial requirement for their use as agents. Through extensive experiments across various models, sizes, and prompting techniques, the authors demonstrate that LLMs struggle to accurately map internal probability estimates to stochastic outputs when directly sampling from distributions. However, they find that LLMs can convert provided random seeds into target distributions, highlighting a disconnect between direct sampling and seed-based generation.
LLMs can fake stochasticity with random seeds, but their direct sampling from distributions is fundamentally broken.
In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents. Agentic systems are frequently required to sample from distributions, often inferred from observed data, a process which needs to be emulated by the LLM. This leads to a distinct failure point: while standard RL agents rely on external sampling mechanisms, LLMs fail to map their internal probability estimates to their stochastic outputs. Through rigorous empirical analysis across multiple model families, model sizes, prompting styles, and distributions, we demonstrate the extent of this failure. Crucially, we show that while powerful frontier models can convert provided random seeds to target distributions, their ability to sample directly from specific distributions is fundamentally flawed.
