Search papers, labs, and topics across Lattice.
AgentAssay is introduced as a token-efficient regression testing framework for non-deterministic AI agent workflows, addressing the lack of principled verification methods after agent modifications. It employs stochastic three-valued verdicts based on hypothesis testing, agent coverage metrics, mutation testing, metamorphic relations, and behavioral fingerprinting to detect regressions. Experiments across multiple models and scenarios demonstrate significant cost reduction (78-100%) and improved detection power (86%) compared to binary testing.
Stop wasting tokens on flaky agent tests: AgentAssay slashes testing costs by up to 100% while boosting regression detection by leveraging behavioral fingerprinting and statistical decision procedures.
Autonomous AI agents are deployed at unprecedented scale, yet no principled methodology exists for verifying that an agent has not regressed after changes to its prompts, tools, models, or orchestration logic. We present AgentAssay, the first token-efficient framework for regression testing non-deterministic AI agent workflows, achieving 78-100% cost reduction while maintaining rigorous statistical guarantees. Our contributions include: (1) stochastic three-valued verdicts (PASS/FAIL/INCONCLUSIVE) grounded in hypothesis testing; (2) five-dimensional agent coverage metrics; (3) agent-specific mutation testing operators; (4) metamorphic relations for agent workflows; (5) CI/CD deployment gates as statistical decision procedures; (6) behavioral fingerprinting that maps execution traces to compact vectors, enabling multivariate regression detection; (7) adaptive budget optimization calibrating trial counts to behavioral variance; and (8) trace-first offline analysis enabling zero-cost testing on production traces. Experiments across 5 models (GPT-5.2, Claude Sonnet 4.6, Mistral-Large-3, Llama-4-Maverick, Phi-4), 3 scenarios, and 7,605 trials demonstrate that behavioral fingerprinting achieves 86% detection power where binary testing has 0%, SPRT reduces trials by 78%, and the full pipeline achieves 100% cost savings through trace-first analysis. Implementation: 20,000+ lines of Python, 751 tests, 10 framework adapters.
