Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of the Model Context Protocol (MCP) in optimizing resource utilization by Large Language Models (LLMs) when accessing embedded reference data. Through a comprehensive 54,000-trial analysis across 24 LLMs, the authors found that while most models can reliably access embedded data when search tools are disabled, the presence of these tools significantly hampers performance for many models. The findings suggest that MCP implementations should prioritize server instructions over tool selection to enhance LLM efficiency and reliability in real-world applications.
A staggering 23 out of 24 LLMs excel at accessing embedded data when search tools are removed, revealing a critical flaw in current server designs.
The Model Context Protocol (MCP) standardizes how servers expose data and tools to Large Language Models (LLMs). A common server design embeds frequently used reference data, such as identifier lookup tables, directly in the server instructions: the system-prompt text a server hands to the host application. When a query concerns an entry of the embedded table, the model can act on it immediately instead of re-discovering the same information through a search tool. We test whether client LLMs actually consume such instruction-embedded data, reporting a 54,000-trial study across 24 LLMs (9 Claude, 6 Gemini, 9 GPT) on a production legal-information MCP server. A diagnostic condition that removes the competing search tool shows that failures are dominated by behavioral preference rather than missing capability. With search unavailable, 23 of 24 models read the embedded data reliably (hit ratio at least 98%); with a search tool merely present, 9 models drop below 15%. A 2^3 factorial analysis of three instruction-level interventions reveals strong interaction effects: combining all three restores at least 86% for 20 of 24 models, but individual interventions can backfire for specific model families. Per-server prompt engineering is therefore a workaround rather than a fix; we argue that MCP host applications should provide an explicit mechanism that places server instructions ahead of tool selection in the client LLM's deliberation.
