Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Reveree, a diagnostic framework designed to evaluate large language model (LLM) agents' performance in reverse engineering (RE) tasks by scoring their trajectories across multiple criteria, including solve rate and milestone progress. The evaluation of nine advanced models on CTF challenges reveals that the base model consistently outperforms others, while larger or more expensive models do not guarantee superior results. Notably, the study uncovers that comprehension stages are the primary bottleneck in the RE process, indicating a fundamental competence limit rather than a mere resource constraint in LLM performance.
LLMs struggle significantly at comprehension stages in reverse engineering, revealing a competence limit that challenges assumptions about model size and cost.
Reverse engineering (RE) is critical to security tasks such as malware analysis and vulnerability discovery, and large language model (LLM) agents are increasingly able to perform it autonomously. Capture-the-flag (CTF) RE challenges have become the standard proxy for measuring this capability, but evaluation rests on a single criterion: whether the agent captures the flag. This solve rate reveals neither where in the RE process an agent fails nor whether a success reflects analysis of the binary or recall of a public solution. In this paper, we propose Reveree, a diagnostic framework that scores an LLM RE agent's trajectory at three tiers: solve rate, milestone progress through an eight-stage RE schema, and a behavioral profile of its actions. Comprehension stages are scored by an outcome-blinded LLM judge validated against a human expert; all other stages are verified deterministically. Using Reveree, we evaluate nine frontier models and four prompting strategies on 88 picoCTF and NYU-CTF challenges. We find that the base model dominates performance, whereas prompting strategy is a secondary, model-dependent effect. Surprisingly, larger, newer, or costlier models are not reliably stronger. We also find that failures concentrate at the comprehension stages of the RE process, and that extra budget, persistence, or reasoning effort rescues few of them, pointing to a competence limit rather than a resource limit. Regarding memorization, while models reproduce picoCTF flags from challenge descriptions alone, NYU-CTF shows minimal measurable recall, and most solves survive surface perturbation, indicating that genuine analysis coexists with memorization. We release Reveree to the community.