Search papers, labs, and topics across Lattice.
This paper introduces an automated acceptance testing framework for LLM-based software (LBS) that addresses the limitations of traditional testing methods by employing Requirements-Augmented Generation (REAG) to interpret user intentions through adaptive retrieval of software requirements and personas. The framework incorporates a confidence-calibrated cascade judgment system that enhances verdict reliability by simulating expert agreement, allowing for high-confidence verdicts while managing ambiguity effectively. An industrial case study demonstrates that REAG not only improves oracle quality from 3.91 to 4.30 but also achieves a 98.8% accuracy rate and a 31.7% cost-efficiency improvement over conventional single-judge methods.
REAG transforms acceptance testing for LLM-based software by achieving a 3.91 to 4.30 improvement in oracle quality while ensuring 98.8% accuracy in verdict reliability.
LLM-based software (LBS) integrates large language models as core components to deliver flexible, personalised responses. Unlike traditional software with deterministic outputs, LBSs exhibit context-dependent, stochastic behaviour that renders classical acceptance testing and test oracles insufficient: the same query may require fundamentally different responses depending on user personas and software context. This gap creates an urgent need for automated acceptance testing frameworks that autonomously interpret user instructions, while reliably inferring user intentions in a changing environment. In this paper, we present an automated acceptance testing framework for LBS with calibrated verdict reliability via two technical contributions. First, we introduce Requirements-Augmented Generation (REAG), which interprets user intentions by retrieving relevant software requirements, domain knowledge, and personas via adaptive RAG and self-reasoning to generate context-aware test oracles. Second, recognising that oracle generation may retrieve irrelevant constraints, misinterpret intent, or hallucinate requirements, we introduce a confidence-calibrated cascade judgment. This method quantifies verdict reliability via simulated expert agreement -- accepting high-confidence verdicts, escalating ambiguous cases, or abstaining when uncertain -- with empirical reliability guarantees backed by conformal risk control. An industrial case study on a production nutrition advisory application demonstrates that REAG achieves a 3.91/5 oracle quality score, reaching qualified or marginal oracle quality in 82% of cases. The confidence-calibrated cascade achieves 98.8% accuracy, improves oracle quality from 3.91 to 4.30 by filtering unqualified outputs, and delivers a 31.7% cost-efficiency improvement over single-judge baselines, validating industrial viability
