Search papers, labs, and topics across Lattice.
This study introduces the concept of semantic bandits, extending the multi-armed bandit framework to account for the influence of semantic priors in large language models (LLMs) during exploration-exploitation tasks. The researchers demonstrate that semantically informative action labels can significantly bias LLMs towards exploitation, enhancing performance when aligned with reward structures but leading to degradation when misaligned. Additionally, they reveal that negative rewards induce greater exploration compared to positive rewards, highlighting the impact of pre-training conventions on decision-making behavior in LLMs.
Semantically informative labels can skew LLM decision-making, leading to either enhanced performance or catastrophic failures depending on alignment with reward structures.
Large language models (LLMs) are increasingly deployed as decision-making agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors --- inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings.
