[Paper Review] An Ensemble Classification Algorithm Based on Information Entropy for Data Streams

Junhong Wang, Shuliang Xu+3 more|arXiv (Cornell University)|Aug 11, 2017

Data Stream Mining Techniques25 references3 citations

TL;DR

This paper proposes an ensemble classification algorithm (ECBE) for data streams that uses information entropy to detect concept drift and dynamically reweight classifiers. By measuring uncertainty via entropy, ECBE identifies performance degradation, discards low-weight classifiers, and maintains high accuracy and efficiency, outperforming OS-ELM in both accuracy and time on multiple benchmark datasets, especially under concept drift conditions.

ABSTRACT

Data stream mining problem has caused widely concerns in the area of machine learning and data mining. In some recent studies, ensemble classification has been widely used in concept drift detection, however, most of them regard classification accuracy as a criterion for judging whether concept drift happening or not. Information entropy is an important and effective method for measuring uncertainty. Based on the information entropy theory, a new algorithm using information entropy to evaluate a classification result is developed. It uses ensemble classification techniques, and the weight of each classifier is decided through the entropy of the result produced by an ensemble classifiers system. When the concept in data streams changing, the classifiers' weight below a threshold value will be abandoned to adapt to a new concept in one time. In the experimental analysis section, six databases and four proposed algorithms are executed. The results show that the proposed method can not only handle concept drift effectively, but also have a better classification accuracy and time performance than the contrastive algorithms.

Motivation & Objective

• Address the challenge of concept drift in data stream classification, where data distributions shift over time.
• Overcome the limitation of existing ensemble methods that rely solely on classification accuracy for drift detection, which fails to capture information uncertainty.
• Develop a method that uses information entropy to quantify uncertainty in classification results and guide dynamic classifier reweighting.
• Improve classification performance and computational efficiency in dynamic data stream environments compared to existing state-of-the-art algorithms.
• Ensure robustness across different types of concept drift, including abrupt and gradual changes.

Proposed method

• Propose an ensemble classification framework (ECBE) that maintains multiple base classifiers and assigns dynamic weights based on the information entropy of their collective output.
• Use the entropy of the ensemble's prediction distribution as a measure of uncertainty: higher entropy indicates greater uncertainty and potential concept drift.
• Define a classifier weight update rule that reduces the weight of classifiers contributing to high-entropy outputs, with weights below a threshold being discarded.
• Introduce a threshold-based mechanism to detect concept drift: when entropy increases significantly, the system triggers reweighting and classifier replacement.
• Apply the algorithm with various activation functions (radbas, sigmoid, sine, hardlim) and tune hyperparameters (α=0.05, β=2, γ=0.1) for optimal performance.
• Use a sliding window mechanism (with window size tuned per dataset) to maintain a fixed historical data window for entropy calculation and drift detection.

Experimental results

Research questions

• RQ1 Can information entropy serve as a reliable indicator for detecting concept drift in data streams, beyond traditional accuracy-based methods?
• RQ2 How does the proposed ECBE algorithm maintain high classification accuracy under varying data stream conditions, including concept drift?
• RQ3 To what extent does ECBE outperform existing ensemble methods like OS-ELM in terms of accuracy and time efficiency?
• RQ4 Can ECBE effectively detect both abrupt and gradual concept drift, as observed in synthetic and real-world datasets?
• RQ5 How does the dynamic classifier reweighting mechanism based on entropy improve long-term model stability and performance?

Key findings

• ECBE achieved higher classification accuracy than OS-ELM on four out of six benchmark datasets (waveform, Hyperplane, LED, sensor_reading_24), with statistically significant improvements on sensor_reading_24 and shuttle.
• On the waveform and Hyperplane datasets, OS-ELM slightly outperformed ECBE by less than 0.02 in accuracy, but ECBE showed superior time efficiency on four datasets.
• ECBE demonstrated strong time efficiency, with execution times as low as 0.2187 seconds on page-blocks and 0.3486 seconds on sensor_reading_24, outperforming OS-ELM on most datasets.
• ECBE successfully detected 15 concept drift events across four datasets (Hyperplane, SEA, waveform, RBF), identifying all 6 abrupt drifts in Hyperplane and waveform, and detecting gradual drifts in RBF.
• Despite minor false alarms or omissions (e.g., one false alert on Hyperplane, one missed drift on SEA), ECBE's entropy-based detection mechanism proved effective overall.
• The algorithm maintained high accuracy post-detection by discarding low-performing classifiers and restoring performance, confirming the effectiveness of the entropy-driven reweighting strategy.