Search papers, labs, and topics across Lattice.
The paper introduces Proxy State-Based Evaluation, a novel LLM-driven simulation framework for evaluating multi-turn tool-calling LLM agents that replaces costly deterministic backends with an LLM state tracker to infer structured proxy states from interaction traces. This approach enables scalable and reliable agent benchmarking by using LLM judges to verify goal completion and detect hallucinations against scenario constraints defined by user goals, facts, expected states, and behaviors. Empirical results demonstrate stable model rankings, effective on-/off-policy supervision transfer, near-zero simulator hallucination rates with careful scenario specification, and high human-LLM judge agreement, highlighting the framework's practicality for industrial LLM agent evaluation.
Ditch the deterministic databases: this LLM-driven simulation framework evaluates tool-calling agents with surprisingly reliable proxy states, offering a scalable alternative to costly benchmarks.
Interactive large language model (LLM) agents operating via multi-turn dialogue and multi-step tool calling are increasingly used in production. Benchmarks for these agents must both reliably compare models and yield on-policy training data. Prior agentic benchmarks (e.g., tau-bench, tau2-bench, AppWorld) rely on fully deterministic backends, which are costly to build and iterate. We propose Proxy State-Based Evaluation, an LLM-driven simulation framework that preserves final state-based evaluation without a deterministic database. Specifically, a scenario specifies the user goal, user/system facts, expected final state, and expected agent behavior, and an LLM state tracker infers a structured proxy state from the full interaction trace. LLM judges then verify goal completion and detect tool/user hallucinations against scenario constraints. Empirically, our benchmark produces stable, model-differentiating rankings across families and inference-time reasoning efforts, and its on-/off-policy rollouts provide supervision that transfers to unseen scenarios. Careful scenario specification yields near-zero simulator hallucination rates as supported by ablation studies. The framework also supports sensitivity analyses over user personas. Human-LLM judge agreement exceeds 90%, indicating reliable automated evaluation. Overall, proxy state-based evaluation offers a practical, scalable alternative to deterministic agentic benchmarks for industrial LLM agents.
