Search papers, labs, and topics across Lattice.
This paper introduces a goal-driven risk assessment approach for LLM-powered systems, focusing on healthcare applications, to address the limitations of traditional threat modeling in capturing the nuances of LLM-specific attacks. The approach uses attack trees to contextualize threats with detailed attack vectors, preconditions, and attack paths, harmonizing LLM-specific and conventional cyberattacks. A case study demonstrates the approach on an LLM agent-based healthcare system, providing a structured risk assessment to improve secure-by-design practices.
LLM-powered healthcare systems are vulnerable to complex attack paths combining prompt injection and conventional cyberattacks, demanding a new goal-driven risk assessment approach.
While incorporating LLMs into systems offers significant benefits in critical application areas such as healthcare, new security challenges emerge due to the potential cyber kill chain cycles that combine adversarial model, prompt injection and conventional cyber attacks. Threat modeling methods enable the system designers to identify potential cyber threats and the relevant mitigations during the early stages of development. Although the cyber security community has extensive experience in applying these methods to software-based systems, the elicited threats are usually abstract and vague, limiting their effectiveness for conducting proper likelihood and impact assessments for risk prioritization, especially in complex systems with novel attacks surfaces, such as those involving LLMs. In this study, we propose a structured, goal driven risk assessment approach that contextualizes the threats with detailed attack vectors, preconditions, and attack paths through the use of attack trees. We demonstrate the proposed approach on a case study with an LLM agent-based healthcare system. This study harmonizes the state-of-the-art attacks to LLMs with conventional ones and presents possible attack paths applicable to similar systems. By providing a structured risk assessment, this study makes a significant contribution to the literature and advances the secure-by-design practices in LLM-based systems.
