Search papers, labs, and topics across Lattice.
This paper introduces ATLAS, a dual-horizon diagnostic evaluation framework designed to enhance the reliability of industrial tool-use agents by providing insights into both request-specific and user-interaction contexts. By analyzing execution deficiencies and sustained service behavior through structured diagnostic signals, ATLAS enables more effective identification of capability gaps and informs policy optimization. Evaluations on Meituan Xiaotuan's production traffic demonstrate that ATLAS not only improves diagnostic fidelity but also leads to significant gains in user engagement and business outcomes through optimized policy feedback.
ATLAS reveals that understanding both immediate request trajectories and long-term user interactions can dramatically enhance the performance of industrial tool-use agents.
Large language model (LLM) agents are increasingly deployed in user-facing services that require iterative tool use under dynamic business conditions. Reliable evaluation is essential for sustained improvement: it must reveal capability deficiencies, inform priorities, and assess interventions. Yet industrial agent service unfolds both through the iterative trajectory of a current request and through continued user interaction. Final-outcome assessment can therefore obscure where deficiencies arise and whether later service remains aligned with context from earlier exchanges. We propose ATLAS, a dual-horizon diagnostic evaluation framework for industrial tool-use agents. At the request horizon, trajectory-wise diagnostic signals relate deficiencies to execution locations and capability concerns. At the interaction horizon, user-wise signals assess whether service remains responsive across continued interaction. Together, these views provide structured diagnostic evidence for analyzing execution deficiencies and sustained service behavior. ATLAS instantiates them as executable signals with explicit evidence scopes and decision boundaries. LLM judge interfaces are calibrated against high-confidence references from real business logs; when needed, their decision behavior is distilled into efficient diagnostic models for lower-latency, lower-cost evaluation. The resulting feedback supports policy optimization. We evaluate ATLAS on Meituan Xiaotuan production traffic. Offline experiments assess diagnostic-signal fidelity and replay-based policy improvement, while online A/B experiments show concurrent gains in user engagement, downstream business outcomes, and sampled human-audit quality.
