Search papers, labs, and topics across Lattice.
This paper investigates the potential of LLMs to automate security-specific thematic analysis of free-text justifications from human experiments. Four top-performing LLMs were prompted to detect nine security-relevant codes in comments from human subjects analyzing vulnerable code snippets, and their performance was compared to human annotators using Cohen's Kappa. The results show that while detailed code descriptions improve performance, LLMs still cannot reliably replace human annotators for this task.
LLMs can't reliably automate security-specific qualitative analysis of human subject comments, even with detailed codebooks and annotation best practices.
[Background:] Thematic analysis of free-text justifications in human experiments provides significant qualitative insights. Yet, it is costly because reliable annotations require multiple domain experts. Large language models (LLMs) seem ideal candidates to replace human annotators. [Problem:] Coding security-specific aspects (code identifiers mentioned, lines-of-code mentioned, security keywords mentioned) may require deeper contextual understanding than sentiment classification. [Objective:] Explore whether LLMs can act as automated annotators for technical security comments by human subjects. [Method:] We prompt four top-performing LLMs on LiveBench to detect nine security-relevant codes in free-text comments by human subjects analyzing vulnerable code snippets. Outputs are compared to human annotators using Cohen's Kappa (chance-corrected accuracy). We test different prompts mimicking annotation best practices, including emerging codes, detailed codebooks with examples, and conflicting examples. [Negative Results:] We observed marked improvements only when using detailed code descriptions; however, these improvements are not uniform across codes and are insufficient to reliably replace a human annotator. [Limitations:] Additional studies with more LLMs and annotation tasks are needed.
