Search papers, labs, and topics across Lattice.
This paper introduces MCP-Persona, a novel benchmark designed to evaluate large language model (LLM) agents on personalized applications that utilize the Model Context Protocol (MCP). By focusing on real-world social and enterprise tools, such as Reddit and Slack, the benchmark reveals significant performance limitations of state-of-the-art agents in handling personalized interactions. The findings underscore the necessity of tailored evaluation metrics for LLMs in personal contexts, paving the way for improved agent performance in real-world applications.
LLM agents struggle significantly with personalized tool use, revealing critical gaps in their capabilities that existing benchmarks overlook.
The Model Context Protocol (MCP) has emerged as a transformative standard for connecting large language models (LLMs) with external data sources and tools, and has been rapidly adopted across personal applications and development platforms. However, existing benchmarks predominantly focus on generic information-seeking tools and fail to capture the practical challenges posed by personal social applications, where tools interact with individual accounts or local databases. To bridge this critical gap, we introduce MCP-Persona, the first benchmark specifically designed for evaluating agent performance on real-world, personalized MCP tools. MCP-Persona encompasses a diverse set of widely-used applications, ranging from social media platforms like Reddit and Xiaohongshu (Rednote) to enterprise collaboration suites such as Lark (Feishu) and Slack. Our extensive experiments on various state-of-the-art (SOTA) agents demonstrate their significant struggles with personalized tool use, thereby highlighting the benchmark's crucial role in identifying and addressing these limitations. MCP-Persona is publicly available at https://github.com/wwh0411/MCP-Persona}{https://github.com/wwh0411/MCP-Persona.
