Search papers, labs, and topics across Lattice.
This paper introduces a standardized vulnerability assessment framework to evaluate the security of five widely-deployed LLM families (GPT-4, GPT-3.5 Turbo, Claude-3 Haiku, LLaMA-2-70B, and Gemini-2.5-pro) against 10,000 adversarial prompts across six attack categories. The study reveals significant security disparities, showing vulnerability rates ranging from 11.9% to 29.8% and a lack of correlation between LLM capability and security. To address these vulnerabilities, the authors develop a defensive framework that achieves 83% average detection accuracy with a 5% false positive rate, demonstrating the effectiveness of systematic security assessment and external defenses.
LLM capability doesn't equal security: vulnerability rates vary by over 15% across top models, proving that bigger isn't always better when it comes to adversarial attacks.
Large Language Models increasingly power critical infrastructure from healthcare to finance, yet their vulnerability to adversarial manipulation threatens system integrity and user safety. Despite growing deployment, no comprehensive comparative security assessment exists across major LLM architectures, leaving organizations unable to quantify risk or select appropriately secure LLMs for sensitive applications. This research addresses this gap by establishing a standardized vulnerability assessment framework and developing a multi-layered defensive system to protect against identified threats. We systematically evaluate five widely-deployed LLM families GPT-4, GPT-3.5 Turbo, Claude-3 Haiku, LLaMA-2-70B, and Gemini-2.5-pro against 10,000 adversarial prompts spanning six attack categories. Our assessment reveals critical security disparities, with vulnerability rates ranging from 11.9\% to 29.8\%, demonstrating that LLM capability does not correlate with security robustness. To mitigate these risks, we develop a production-ready defensive framework achieving 83\% average detection accuracy with only 5\% false positives. These results demonstrate that systematic security assessment combined with external defensive measures provides a viable path toward safer LLM deployment in production environments.
