Search papers, labs, and topics across Lattice.
This paper introduces RPA-Check, a multi-stage automated framework for evaluating the performance of LLM-based role-playing agents (RPAs) by assessing role adherence, logical consistency, and narrative stability. The framework uses a four-step pipeline: dimension definition, augmentation into boolean checklist indicators, semantic filtering for objectivity, and LLM-as-a-Judge evaluation using chain-of-thought verification. Applied to LLM Court, RPA-Check reveals that smaller, instruction-tuned models (8-9B) can outperform larger models in procedural consistency due to user-alignment bias in larger models.
Smaller, instruction-tuned LLMs can actually beat their larger counterparts in complex role-playing scenarios, thanks to the surprising discovery that bigger models often suffer from user-alignment bias and reduced consistency.
The rapid adoption of Large Language Models (LLMs) in interactive systems has enabled the creation of dynamic, open-ended Role-Playing Agents (RPAs). However, evaluating these agents remains a significant challenge, as standard NLP metrics fail to capture the nuances of role adherence, logical consistency, and long-term narrative stability. This paper introduces RPA-Check, a multi-stage automated evaluation framework designed to objectively assess the performance of LLM-based RPAs in complex, constraints-heavy environments. Our methodology is based on a four-step pipeline: (1) Dimension Definition, establishing high-level qualitative behavioral criteria; (2) Augmentation, where these requirements are expanded into granular boolean checklist indicators; (3) Semantic Filtering, to ensure indicator objectivity, no redundancy and agent isolation; and (4) LLM-as-a-Judge Evaluation, which employs chain-of-thought verification to score agent fidelity. We validate this framework by applying it to LLM Court, a serious game for forensic training involving several quantized local models. Experimental results across five distinct legal scenarios demonstrate the framework's ability to identify subtle trade-offs between model size, reasoning depth, and operational stability. Notably, the findings reveal an inverse relationship between parametric scale and procedural consistency, showing that smaller, adequately instruction-tuned models (8-9B) can outperform larger architectures prone to user-alignment bias or sycophancy. RPA-Check thus provides a standardized and reproducible metric for future research in generative agent evaluation within specialized domains.
