Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
RuleTrek is a new benchmark designed to evaluate LLM agents' ability to identify and apply domain-specific rules in complex, multi-file scenarios, addressing limitations of existing benchmarks that use simplified factual representations. The benchmark assesses Rule Identification, Rule Deduction, and End-to-End Rule Reasoning within interactive workspaces. Experiments reveal that even state-of-the-art LLMs struggle significantly, achieving at most 62.47% on Rule Identification and 60.43% on Rule Deduction, with End-to-End scores below 55.29%, highlighting a critical gap in agentic rule reasoning capabilities.
Frontier LLMs fail to reliably identify and apply domain-specific rules in complex, multi-file scenarios, with top models scoring below 63% on core reasoning tasks.
Large language models (LLMs) increasingly tackle complex real-world tasks in specialized domains. However, identifying and correctly applying domain-specific rules remains challenging when scenario information is distributed across multiple files. Existing rule-reasoning benchmarks typically represent scenarios using only a few factual statements, simplifying the identification and application of relevant rules. To address this gap, this paper introduces RuleTrek, a multi-domain benchmark for agentic rule reasoning in complex scenarios. Its rule-grounded construction pipeline extracts structured rule graphs, generates and incrementally composes rule-specific sub-scenarios while ensuring consistency and solvability. To systematically evaluate agentic rule reasoning in complex scenarios, RuleTrek assesses three capabilities in interactive workspaces: Rule Identification (RI), Rule Deduction (RD), and End-to-End Rule Reasoning (E2E). Experiments on 11 frontier LLMs show that the highest RI and RD scores are only 62.47\% and 60.43\%, respectively, while E2E scores range from 26.97\% to 55.29\%. These results demonstrate that current LLM agents face substantial challenges in rule reasoning over complex scenarios. We make our code and dataset publicly available at https://github.com/yuan247/RuleTrek{RuleTrek}.