Search papers, labs, and topics across Lattice.
This paper introduces ClawTrack, a dual-assessment benchmark designed to evaluate LLM-based agents by measuring both their outcomes (Task Score) and the reasoning processes behind those outcomes (Process Score). By analyzing 21 models across 16,000+ trials, the authors demonstrate that process scores can effectively pinpoint the reasons for success or failure in complex tasks, revealing that result verification is a critical bottleneck in the reasoning process. The findings suggest that using process-based evaluations can lead to significant improvements in model performance across different scales.
Process evaluations reveal hidden failures in LLM reasoning, showing that lucky successes can mask critical deficiencies in agent performance.
As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks. In this work, we present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves (Task Score) and how it achieves it (Process Score). ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services. A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification), anchored by 12,541 task-specific rubric items. Evaluating 21 models over 16,000+ trials, we find that: (1) process scores effectively attribute success and failure to specific reasoning dimensions, filtering lucky passes invisible to outcome-only evaluation; (2) the four dimensions are complementary, with result verification as the systematic bottleneck; (3) the framework is robust to evaluator choice across different judge LLMs; and (4) process-based trajectory filtering yields consistent post-training improvements across model scales.
