Search papers, labs, and topics across Lattice.
This paper investigates human-AI collaboration in Capture-the-Flag (CTF) cybersecurity competitions by studying how human players perceive and interact with AI assistance, and comparing their performance to autonomous AI agents. The study involved 41 participants collaborating with an AI agent in a live CTF, alongside benchmarking four autonomous AI agents on the same challenges. The key finding is that human-AI team performance is often limited by ineffective prompting and context specification from the human side, while autonomous agents that self-direct their prompting and tool use outperform most human teams.
Human-AI cybersecurity teams are held back by poor prompting, as autonomous AI agents that self-direct their tool use outperformed most human teams in a live CTF competition.
Capture-the-Flag (CTF) competitions are increasingly becoming a testbed for evaluating AI capabilities at solving security tasks, due to the controlled environments and objective success criteria. Existing evaluations have focused on how successful AI is at solving CTF challenges in isolation from human CTF players. As AI usage increases in both academic and industrial settings, it is equally likely that human players may collaborate with AI agents to solve challenges. This possibility exposes a key knowledge gap: how do humans perceive AI CTF assistance; when assistance is provided, how do they collaborate and is it effective with respect to human performance; how do humans assisted by AI compare to the performance of fully autonomous AI agents on the same challenges. We address this gap with the first empirical study of AI assistance in a live, onsite CTF. In a study with 41 participants, we qualitatively study (i) how participants'perception, trust, and expectations shift before versus after hands-on AI use, and (ii) how participants collaborate with an instrumented AI agent. Moreover, we also (iii) benchmark four autonomous AI agents on the same fresh challenge set to compare outcomes with human teams and analyze agent trajectories. We find that, as the competition progresses, teams increasingly delegate larger subtasks to the AI, giving it more agency. Interestingly, CTF challenges solving rates are often constrained not by model's reasoning capabilities, but rather by the human players: ineffective prompting and poor context specification become the primary bottleneck. Remarkably, autonomous agents that self-direct their prompting and tool use bypass this bottleneck and outperform most human teams, coming in second overall in the competition. We conclude with implications for the future design of CTF challenges and for building effective human-in-the-loop AI systems for security.
