Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper benchmarks seven stream learning classifiers across 13 real and synthetic streams to evaluate their memory consumption and adaptability to concept drift under constrained resource budgets. The study reveals two primary failure modes: adaptive ensembles often exceed small budgets quickly due to their initial size, while incremental trees grow significantly over time, with HoeffdingTrees and Extremely Fast Decision Trees increasing in size by factors of 7.37 and 5.87, respectively. The findings highlight that many current stream learning methods are inadequate for embedded systems, emphasizing the need for resource usage to be prioritized alongside drift adaptation in future designs.
Many state-of-the-art stream learning methods are ill-suited for embedded systems, with some exceeding memory budgets almost immediately or growing unsustainably over time.
Stream learning is commonly evaluated through predictive performance and adaptation to concept drift. However, sustained operation of a stream learner also requires predictable and bounded resource usage even on long streams. This requirement becomes even more critical when learning moves from servers to near-sensor embedded systems where memory and processing are scarce resources. In state-of-the-art stream learning, however, we perceive a strong focus on concept drift adaptation, whereas resource usage is often an evaluation byproduct. To close this gap, we benchmark seven representative stream classifiers on 13 real and synthetic streams under model-size budgets from 128\,KiB to approximately 8\,MiB. Our benchmark comprises a total of 6,463 experiments. We measure failure-aware accuracy, peak model size, time to budget exhaustion, and prediction-plus-update latency. The results reveal two distinct resource failure modes. Adaptive ensembles can exceed small budgets almost immediately because of their initial footprint, even when their size remains stable thereafter. Incremental trees can fit initially but grow throughout a long stream, with HoeffdingTrees (HT) and Extremely Fast Decision Trees (EFDT) increasing by median factors of 7.37 and 5.87. Explicitly compact methods remain the only viable option under the smallest budgets, but are usually overtaken as larger budgets make adaptive ensembles competitive. Hence, many state-of-the-art methods are only partially applicable in embedded systems or for long-running systems. We therefore call on the stream-learning community to make bounded resource usage a first-class design objective alongside drift adaptation, and propose concrete steps toward this goal, including an API through which stream learners can explicitly expose and respect resource budgets.