Search papers, labs, and topics across Lattice.
This paper introduces SafeMCP, a server-side defense plugin designed to mitigate the risks associated with the expanded action spaces of Large Language Model (LLM) agents. By employing an internal world model for look-ahead reasoning, SafeMCP proactively filters tool acquisition and implements immediate interventions to prevent hazardous power-seeking behaviors. Experiments demonstrate that SafeMCP successfully balances safety and utility, achieving a safe equilibrium in various testing environments, including PowerSeeking Bench and ToolEmu.
SafeMCP effectively mitigates the risks of power-seeking behaviors in LLM agents while maintaining their operational utility.
As Large Language Model (LLM) agents increasingly leverage the Model Context Protocol (MCP) to operate in complex environments, the expansion of their action spaces offers agents unsafe capabilities and underscores the risk of power-seeking. While broad action space and greater environment influence are essential for task fulfillment, they create a fragile risk surface where minor errors or hallucinations are magnified into catastrophic failures. In response, we propose SafeMCP, a {server-side} defense plugin that constrains tool acquisition via predictive reasoning regarding future safety risks. SafeMCP utilizes an internal world model for look-ahead reasoning to implement a two-tier defense: proactive tool filtering to constrain hazardous power expansion and immediate intervention as a fail-safe. To train SafeMCP, we introduce a three-stage pipeline comprising environmental dynamic grounding, safe policy initialization, and reinforcement learning (RL) with dual verifiable rewards. Experiments on PowerSeeking Bench, ToolEmu, and AgentHarm show that SafeMCP achieves a safe equilibrium, effectively mitigating risks while preserving agent utility.
