Search papers, labs, and topics across Lattice.
This paper introduces an automated method for assessing the quality of defeaters generated by large language models (LLMs) in the context of assurance cases for high-integrity systems. By leveraging natural language processing techniques, including structural analysis of directed acyclic graphs and vector-based semantic embeddings, the authors demonstrate that their approach significantly reduces subjective variance in quality assessment. The method achieves an average F1-score of 0.84 and improves inter-rater agreement, indicating a more reliable evaluation process for LLM-generated content in safety-critical domains.
Automating the quality assessment of LLM-generated defeaters could revolutionize how we validate safety claims in high-integrity systems.
High-integrity systems, such as autonomous vehicle fleets or large-scale energy infrastructures, rely on structured assurance cases, represented as rooted directed acyclic graphs (DAGs), to justify safety claims. To remain valid in the face of change, these cases must be robust against potential challenges, known as ‘defeaters’. While large language models (LLMs) have recently enabled scalable generation of such defeaters, validating their quality remains a predominantly manual and subjective process. This paper presents an automated method for assessing LLM-generated defeaters using natural language processing (NLP) techniques. Our approach combines structural analysis of assurance case graphs with vector-based semantic embeddings and meta-classifiers trained on expert-assessed consensus defeaters. We evaluate our method through two case studies in the automotive and energy domains, quantifying human reviewer dissensus using Cohen’s kappa ($\kappa \lt 0.442$), which indicates low inter-rater agreement. Our automated approach achieves greater consistency with individual raters, improving ($\kappa \approx 40 \%$). The method delivers an average F1-score of 0.84 across validation, reducing subjective variance through scalable, objective assessment. Our approach aims to advance the tool support for automation of assurance case synthesis.
