Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critically examines benchmarking practices in microarchitectural side-channel attacks by analyzing 83 attack papers from leading security and architecture conferences between 2014 and 2024. It identifies 19 recurrent flaws in these benchmarks that compromise evaluation completeness, relevance, soundness, and reproducibility, with an average of 5.5 flaws per paper. The findings underscore the urgent need for improved benchmarking standards to enhance the reliability and relevance of research in this rapidly evolving field.
Flawed benchmarking practices in microarchitectural side-channel research are more common than expected, with each paper averaging 5.5 significant issues that undermine reproducibility and relevance.
Microarchitectural side-channel research has grown at an exceptional pace in recent years, increasing the need for rigorous and meaningful benchmarking. Early attack papers typically relied on indirect proxies, such as covert-channel bandwidth or key-recovery on naive AES and RSA implementations, setting de facto standards that many subsequent works continued to replicate, sometimes by directly comparing against raw numbers from prior work. While these practices offer convenient points of comparison, current benchmarks may not be the most relevant to assess specific properties of new primitives. Even more problematic, microarchitectural attacks are notoriously sensitive to experimental conditions: minimal changes in the target system can significantly alter outcomes and performance. As a result, inadequate evaluation practices undermine reproducibility and cast doubt on the relevance of comparisons, even in top-tier venues where such issues should be identified. This paper tackles the core problem of proper benchmarking for microarchitectural side-channel attacks and examines its broader impact on research quality in the field. We survey 83 attack papers published in top-ranked security and architecture conferences from 2014 to 2024. From this corpus, we identify and define 19 recurrent benchmarking flaws that affect evaluation completeness, relevance, soundness, and reproducibility. These flaws include unfair or absent comparisons, missing code or materials, and the failure to evaluate the key attack properties. On average, each paper exhibits 5.5 such flaws, highlighting how widespread the issue is, even in highly selective venues. Based on our findings, we identify and suggest key properties that are relevant to properly evaluate new attacks. We also highlight trends over time and different practices between security and architecture conferences.