Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper benchmarks the performance of Deep Seek Coder and Meta-llama-3-70b-instruct in detecting SQL injection vulnerabilities using a labeled dataset of malicious and legitimate SQL queries. The evaluation focuses on Boolean-based attacks and measures precision, recall, F1-score, and accuracy. Meta-llama-3-70b-instruct achieved superior recall and overall accuracy (74.00%) compared to Deep Seek Coder (60.00%), suggesting it is better at detecting a wider range of malicious queries, though both models require further refinement for standalone security analysis.
Meta-Llama-3-70B-Instruct edges out DeepSeek Coder in SQL injection detection accuracy, but neither model is ready to replace traditional security analysis.
With the ongoing incorporation of large language models (LLMs) into cybersecurity operations, it is important to examine how they perform in real-life attacks. This paper presents a comparative evaluation of two advanced large language models, Deep Seek Coder and meta-llama-3-70b-instruct, in detecting SQL injection vulnerabilities in source code. This article places greater emphasis on Boolean-based attacks. To test these models, we use a labelled dataset, which contains malicious and legitimate SQL queries. The most important parameters of classification, such as precision, recall, F1-score, and accuracy, are used in the measurement of performance. The results of the evaluation indicate that the meta-llama-3-70b-instruct model performed better in important parameters of accuracy with scores of 68.75% precision, 88.00% recall, 72.42% F1 score, and 74.00% overall accuracy. Comparatively, Deep Seek Coder attained a precision of 71.50%, a recall rate of 44.00%, an F1 score of 57.17%, and an accuracy rate of 60.00%. This comparison shows that Deep Seek Coder is particularly strong in precision, which hints at the ability to find common vulnerabilities, but also very weak in recall and accuracy. Meta-LLaMA-3-70B-Instruct has a better recall and overall accuracy and is best able to detect more varieties of malignant queries. So, it should have significant issues with detecting more context-sensitive or obfuscated security concerns. The results make it clear that these forms of models can be an efficient helper in a secure coding process, but they cannot be used as stand-alone security analysis tools at the current stage. Suggestions are being made to develop a combination of LLMs and classic static analysis and fine-tuning with domain-specific security datasets.