Search papers, labs, and topics across Lattice.
This paper introduces Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots that addresses limitations of prior evaluation methods by grounding honeypots in 16 backend applications and using AI hacking agents with verifiable exploit goals. Experiments using Honeyval demonstrate that LLM-powered honeypots lead to longer attacker interactions and are less frequently detected compared to rule-based baselines, while maintaining a cost advantage. The framework also reveals trade-offs in counter-offensive honeypot configurations, such as increased interaction length at the cost of higher detection rates.
LLM-powered honeypots can trick even frontier models into longer interactions than rule-based systems, all while costing less to run.
Honeypots are decoy systems mimicking real system components designed to defend against cyber attacks. Recently, LLMs increasingly serve as simulation backbones for honeypots. They enable defenders to construct high-interaction honeypots with low system security risks. However, LLM-powered honeypot development lacks a unified evaluation framework. Most evaluations consist of measuring response similarity on fixed commands, manual testing, or real-world deployment. These methods are often not scalable for development, reproducible across evaluations, representative of practical attacks, or adaptable to various attacker and honeypot configurations. In this work, we bridge this gap and propose Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots. We address the limitations of prior evaluations by grounding the honeypots in 16 backend applications, using AI hacking agents as attackers, employing two control tasks to monitor agent and honeypot capabilities across customizations, and defining clear and verifiable exploit goals for the attacker. Using Honeyval, we conduct an extensive evaluation of recent cost-efficient LLMs as HTTP honeypots. Our experiments highlight the promise of LLM-powered honeypots; they lead to substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while, on average, preserving a running cost advantage against agentic attackers. Further, we experiment with different counter-offensive honeypots configurations, and observe unique trade-offs, such as longer interactions at the cost of increased detection.