Search papers, labs, and topics across Lattice.
The paper introduces the Decentralized Evaluation Protocol (DEP), a framework designed to address the lack of standardization and potential data leakage issues in current LLM evaluation benchmarks. DEP decouples users, LLMs, and benchmarks through a matching server, enabling modular, plug-and-play evaluation where benchmark files and evaluation logic remain server-side. The authors implemented DEP Toolkit, which supports features like breakpoint resume and concurrent requests, and demonstrated its effectiveness by evaluating multiple LLMs across a variety of benchmarks.
Say goodbye to benchmark leakage and inconsistent LLM evals: a new decentralized protocol keeps your data safe while streamlining the evaluation process.
With the rapid development of Large Language Models (LLMs), a large number of benchmarks have been proposed. However, most benchmarks lack unified evaluation standard and require the manual implementation of custom scripts, making results hard to ensure consistency and reproducibility. Furthermore, mainstream evaluation frameworks are centralized, with datasets and answers, which increases the risk of benchmark leakage. To address these issues, we propose a Decentralized Evaluation Protocol (DEP), a decentralized yet unified and standardized evaluation framework through a matching server without constraining benchmarks. The server can be mounted locally or deployed remotely, and once adapted, it can be reused over the long term. By decoupling users, LLMs, and benchmarks, DEP enables modular, plug-and-play evaluation: benchmark files and evaluation logic stay exclusively on the server side. In remote setting, users cannot access the ground truth, thereby achieving data isolation and leak-proof evaluation. To facilitate practical adoption, we develop DEP Toolkit, a protocol-compatible toolkit that supports features such as breakpoint resume, concurrent requests, and congestion control. We also provide detailed documentation for adapting new benchmarks to DEP. Using DEP toolkit, we evaluate multiple LLMs across benchmarks. Experimental results verify the effectiveness of DEP and show that it reduces the cost of deploying benchmark evaluations. As of February 2026, we have adapted over 60 benchmarks and continue to promote community co-construction to support unified evaluation across various tasks and domains.
