Search papers, labs, and topics across Lattice.
This paper introduces Copy-on-Write (CoW) Scoring, a novel framework for evaluating LLM-based agents within application-specific environments, addressing the limitations of existing evaluation methods that lack construct validity and are costly. By leveraging a PostgreSQL-level Copy-on-Write mechanism, CoW Scoring isolates agent write operations to produce detailed session- and operation-level performance metrics. The application of this framework on the Plane project-management platform revealed critical issues in the tool surface, leading to targeted fixes that significantly enhanced model performance.
CoW Scoring reveals precise failure points in agent operations, enabling targeted improvements that can dramatically enhance performance in real-world applications.
Trustworthy deployment of LLM-based agents in software systems requires evaluating how they perform on application-specific workflows, with enough granularity to localize where they succeed and fail. Yet existing agent evaluation mechanisms are limited: benchmarks have low construct validity for application-specific workflows and environments, and replica evaluation environments are expensive and prone to drift. We propose Copy-on-Write (CoW) Scoring, a framework that evaluates agent operations directly within application environments using a PostgreSQL-level Copy-on-Write mechanism to isolate agent writes. CoW Scoring produces session- and operation-level scores that highlight where agents'database write operations succeed and fail in a given application environment, enabling inexpensive evaluation and iteration on agent harnesses and tool surfaces. We demonstrate the framework on Plane, an open-source project-management platform, where analysis surfaced specific issues in the tool surface, and corresponding fixes produced measurable improvements on affected models. Python library: https://github.com/trail-ml/agent-cow-python
