Search papers, labs, and topics across Lattice.
This paper introduces Multi-Level Distributional Entropy (MDE), a novel framework for extracting interpretable entropy features from aggregated flow statistics in network intrusion detection systems (IDS), without requiring raw packet data. MDE operates at three levels鈥擥aussian differential entropy, Jensen-Shannon divergence, and Shannon entropy of TCP flags鈥攜ielding competitive performance on multiple benchmarks while revealing critical detection rate issues masked by traditional aggregate metrics. The findings highlight that while MDE achieves high weighted F1 scores, it can also expose significant vulnerabilities in detection rates, emphasizing the need for careful evaluation of IDS performance metrics.
Entropy-only features can match conventional IDS performance while exposing hidden vulnerabilities in detection rates that traditional metrics overlook.
Machine learning network intrusion detection systems (IDS) rely on aggregate flow statistics that discard distributional structure, while established entropy measures require raw packet sequences unavailable in pre-aggregated flow datasets. We propose Multi-Level Distributional Entropy (MDE), an analytical framework that derives interpretable entropy features directly from flow-level summary statistics at three levels: within-flow Gaussian differential entropy, cross-directional Jensen-Shannon divergence (JSD), and Transmission Control Protocol (TCP) flag-pattern Shannon entropy, without raw packet access or training data. Across four benchmarks (NSL-KDD, CICIDS-2017, CICIDS-2018, UNSW-NB15) under a leakage-free fold-local pipeline, entropy-only features achieve weighted F1 of 0.708-0.989, matching conventional features without degrading performance. Full operational metric reporting then exposes failure modes that aggregate F1 conceals. On CICIDS-2018, F1=0.74 hides a detection rate (DR) of 0.48, and on held-out attack families F1 exceeds 0.998 while DR falls to zero. Under temporal shift, a pseudo-live replay of 703K flows reveals a threshold-ranking divergence in which score ranking is preserved (AUC=0.87) but fixed thresholds collapse (DR=0.082) and recalibration offers no recovery. SHapley Additive exPlanations (SHAP) fold-stability analysis (Spearman rho=0.80-0.95) confirms that entropy attributions are reproducible and domain-coherent across heterogeneous environments.
