Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critically evaluates the efficacy of Machine Learning-based Intrusion Detection Systems (IDS) by testing their generalization capabilities against unseen attacks. The authors reveal that despite improvements in detection rates and reduced false alarms, many models fail to effectively identify novel threats, exposing significant vulnerabilities in current evaluation practices. By conducting experiments that assess the impact of dimensionality on classifier performance, the study identifies structural weaknesses and proposes seven evaluation criteria to enhance future research in this area.
Machine Learning-based Intrusion Detection Systems may appear effective, but they often fail to detect unseen attacks, revealing critical vulnerabilities in their design and evaluation.
Intrusion Detection has been revolutionized due to the integration of Machine Learning. Improved detection rates, reduced false alarms, and optimized algorithms contribute to the perception of improved systems with optimal accuracy and near-perfect performance, the illusion of model efficacy. However, the value of this effectiveness diminishes when confronted with unseen attacks. In this paper, we go beyond solely algorithmic enhancements and metric adjustments in ML-based Network Intrusion Detection Systems. We design an experiment to test the generalization capabilities of certain classifiers on unseen attacks. Our approach examines the dimensionality parameter's impact through two experimental methodologies, which are applied in two distinct settings. The experimental findings reveal how effectively the models could identify even a fraction of unseen attacks and underscore structural weaknesses in ML-based IDS research and evaluation techniques. Finally, seven evaluation criteria are outlined to address these challenges.