Search papers, labs, and topics across Lattice.
This paper introduces EXE-Bench, a novel benchmark designed to systematically evaluate AI-based Windows malware detectors by assessing their performance, temporal robustness, adversarial resilience, and computational overhead. The study reveals that traditional evaluations fail to capture the long-term effectiveness and security vulnerabilities of these models, emphasizing the importance of domain knowledge in feature engineering. Key findings indicate that models leveraging domain expertise outperform deep networks in resisting time decay and adversarial attacks, underscoring the need for comprehensive pre-deployment assessments.
AI-based malware detectors that rely on domain knowledge significantly outperform deep learning models in long-term resilience and security against adversarial attacks.
Due to the lack of systematic evaluations, we are not yet able to determine which AI-based Windows malware detector to deploy in production, since existing evaluations (i) differ in terms of data used for both training and testing; (ii) do not consider temporal analysis to showcase whether models withstand the passage of time; (iii) avoid security evaluations with adversarial attacks that could highlight their brittleness against content-injection attacks; and (iv) neglect the computational requirements for deployment, risking slow inference on endpoints. For these reasons, we develop EXE-Bench, a comprehensive benchmark of AI-based Windows malware detectors. EXE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison. Through EXE-Bench, we highlight how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance. In particular, through our analysis, we remark how much domain knowledge instilled through feature engineering is still extremely useful in this domain, resisting both time and adversarial attacks, in stark contrast with most of the deep networks that only excel right after deployment.
