Search papers, labs, and topics across Lattice.
The paper introduces HERO'S JOURNEY, a benchmark designed to evaluate rule induction capabilities in agents performing goal-directed episodic tasks by inferring hidden rules from demonstrations. Through testing state-of-the-art large language models (LLMs) across eight diverse tasks, the study reveals that while these models exhibit some ability to induce rules, their performance is inconsistent and hindered by execution bottlenecks. Notably, while induction-specific steering methods enhance performance on attribute tasks, they fail to provide significant improvements for procedural tasks, highlighting a critical gap in procedural rule induction that warrants further exploration.
LLMs struggle with procedural rule induction, revealing a significant performance gap that challenges current AI capabilities.
We introduce HERO'S JOURNEY, a benchmark for rule induction in goal-directed episodic tasks, where agents must infer hidden rules from demonstrations and act on them through multi-step execution. HERO'S JOURNEY covers eight tasks across attribute and procedural induction families, each with four structural rule forms, controllable lexical grounding, and identifiability conditions. Evaluating state-of-the-art LLMs, we find that models show evidence of rule induction, but the ability is limited and uneven across tasks. Meanwhile, process execution adds an execution bottleneck for models, whereas surface semantics has minimal effect. Induction-specific steering methods improve performance on attribute tasks but show no reliable gains on procedural tasks, suggesting the gap in procedural induction remains an open challenge.
