Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critiques the Edge-IIoTset benchmark for machine-learning intrusion detection in the industrial Internet of Things, revealing that high reported accuracy is largely due to a serialization artifact rather than actual intrusion detection capabilities. By demonstrating that four categorical columns can perfectly distinguish between attack and normal traffic based solely on the formatting of protocol fields, the authors expose a significant flaw in the dataset's integrity. In response, they introduce AgriEdge, a new benchmark that mitigates these issues by providing a more robust dataset with comprehensive device attribution and minimal class separation, leading to more realistic evaluation metrics for intrusion detection models.
High accuracy in Edge-IIoTset is misleading, as it stems from a serialization artifact rather than genuine intrusion detection performance.
Edge-IIoTset is the reference benchmark for machine-learning intrusion detection in the industrial Internet of Things, and results reported on it cluster above 99%. We show that much of that performance is not intrusion detection. The preprocessing recipe distributed with the dataset instructs researchers to one-hot encode seven categorical columns. Four of them separate attack from normal traffic with an accuracy of 1.0000 on their own, through the spelling of the placeholder written for an absent protocol field: the string "0" in the normal-traffic branch of the dataset build against "0.0" in the attack branch. The label is recoverable from a serialisation artifact encoding file provenance, with no network behaviour modelled, and separates every row of both curated subsets. Under 5-fold x 3-repeat cross-validation, five of six standard classifiers attain exactly 1.0000 +/- 0.0000 accuracy and the sixth attains 0.99998. Under a corrected protocol, naive Bayes falls by 0.3005 macro-F1 and the strongest model settles at 0.9503 +/- 0.0011. Label, ordinal and frequency encoding leak identically. Because the curated subsets also lack Modbus and per-device identity, we rebuild the benchmark from the raw captures under uniform parsing, producing AgriEdge: 1,276,122 rows, five devices with full attribution, and no column separating the classes above 0.0288. A leave-one-device-out sweep locates the generalisation boundary at the perception/actuation layer, where random forest falls from 0.9988 to 0.5083 balanced accuracy. Non-IID federated partitioning costs at most 0.0037 macro-F1, but a 20-round LoRaWAN training run costs 4.6 hours of uplink.