Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This systematic literature review analyzes the sources of authority in test oracles generated by large language models (LLMs), addressing a significant gap in understanding how these oracles can be trusted. By filtering 2,436 records and employing dual human screening, the study identifies 54 relevant studies and categorizes them based on the source of authority, form, and adjudication mechanism. The findings reveal that while specification-derived authority is prevalent, a substantial number of oracles operate without any specification, highlighting critical areas for future research and evaluation of LLM-generated verdicts.
Over half of the test oracles from LLMs lack any specification, raising crucial questions about their reliability and trustworthiness.
Large language models (LLMs) are increasingly used to produce test oracles, the part of a test that decides whether observed behavior is correct. Yet a clear account of where these oracles draw their authority is missing. Prior secondary studies organize the area by oracle form or by LLM technique. None organizes it by the source of the verdict's authority, the property that governs how far a verdict can be trusted. This article presents a systematic literature review, conducted and reported under the PRISMA 2020 guidelines. From 2,436 records, an LLM pre-filter followed by independent dual human screening (reviewer agreement, a Cohen's kappa of 0.79) and full-text assessment yielded 54 included studies. We analyze these along three axes: the source of an oracle's authority, the form it takes, and the mechanism that adjudicates it. We characterize the landscape of domains, languages, models, and adaptation strategies. Specification-derived authority, though the most common single source, covers about half of the studies (28 of 54). The remaining 26 reach a verdict with no specification at all. The source of authority and the adjudication mechanism cross-cut: the same source is checked by several mechanisms and one mechanism serves several sources, so a label such as LLM-as-a-judge names a mechanism rather than a basis for trust. We further report how these oracles are evaluated and how they fail, and read the sparse and empty regions of the taxonomy as a research agenda. The protocol, search query, and per-study coding sheet are released as supplementary material.