Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study empirically evaluates the performance of open-source GenAI Large Language Model agents in comparison to a traditional Static Application Security Testing (SAST) tool, Bandit, focusing on metrics such as precision, recall, and false positives. The assessment involved three different Ollama-hosted models and revealed that these GenAI agents do not meet the effectiveness required for SAST tasks under realistic conditions. The findings challenge the assumption that modern open-source LLMs can serve as viable replacements for established security testing tools, highlighting significant limitations in their current capabilities.
Open-source GenAI LLM agents fall short of replacing traditional SAST tools, revealing critical gaps in their effectiveness for cybersecurity tasks.
This paper explores the value of agentic AI tools for cybersecurity purposes. We evaluate the efficacy of a general-purpose GenAI Large Language Model- (GenAI-) based agent when powered by three different Ollama-hosted general-purpose open source models. We assess each agent's performance using precision, recall, false positive count, and a calculated composite score based upon the interplay of the captured metrics, against the baseline performance of an existing, vetted Static Application Security Testing (SAST) tool, Bandit. Our findings refute the notion that a modern open-source GenAI LLM-based agent is currently suitable for the specialized task of SAST scanning under realistic conditions.