Search papers, labs, and topics across Lattice.
This paper introduces RuleWeaver, a novel benchmark framework designed to evaluate rule-centered scenario reasoning in large language models (LLMs). By constructing complex rule-based question-answering instances from corpus-derived IF-THEN meta rules, the framework allows for both final-answer correctness and process-level evaluation metrics such as rule recall and precision. Experiments reveal that even the top-performing LLMs struggle with this reasoning task, achieving only about 50% of the maximum rubric score, highlighting significant limitations in current models' capabilities.
Current LLMs falter in complex rule-centered reasoning, with top models only reaching half of the potential performance on a new benchmark.
Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios. However, existing benchmarks only partially evaluate this capability, as they either focus on output-level instruction constraints or overlook the distinct roles that rules play in scenario reasoning. To address these gaps, this paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning. RuleWeaver starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances. Beyond final-answer correctness, RuleWeaver further supports process-level evaluation through rubric-based answer quality, rule recall, and rule precision. Experiments on 11 representative LLMs show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/RuleWeaver.
