Search papers, labs, and topics across Lattice.
This study investigates how LLM agents adapt their tool choices when the reliability of tools changes unexpectedly during a session, utilizing a set-shifting framework from cognitive psychology. By implementing a benchmark that includes redundancies in tool-skill libraries, the researchers evaluate agent performance in adapting to these hidden shifts through a branched schedule. The findings reveal that agents tend to default to a narrow set of routines and exhibit distinct failure modes, influenced by how tools are framed as competing or complementary.
Agents default to a narrow routine, struggling to adapt to hidden shifts in tool reliability, revealing critical insights into their decision-making processes.
What happens to an LLM agent's tool choice when the reliable tool silently changes within an ongoing session? We borrow set-shifting from cognitive psychology to study how well agents adapt to hidden reliability shifts. Our benchmark mounts tool-skill libraries with redundancies, where many tools solve the same task but differ in hidden reliability. In our evaluation framework, a branched schedule shifts the reliable tool group at hidden boundaries and pairs every shift with a no-shift control. We find that agents, by default, settle on a small recurring routine within a few turns of each boundary, with call shares concentrating on a few discrete values after each reliability shift. We score the set-shifting accuracy for each agent trajectory: the joint probability of routing to the target tool group in every post-shift window. We test open-weight LLMs in an open-source agentic harness and find qualitatively distinct failure modes across the same set of routines. We also find that set framing, how the toolset presents the alternatives as competing or complementary, shifts the routing dynamics.
