[Paper Review] Modified SMOTE Using Mutual Information and Different Sorts of Entropies

Sima Sharifirad, Azra Nazari+1 more|arXiv (Cornell University)|Mar 29, 2018

Imbalanced Data Classification Techniques25 references3 citations

TL;DR

This paper proposes four enhanced SMOTE variants that integrate mutual information and three entropy measures—maximum, Renyi, and Tsallis—to optimize feature weighting in KNN-based oversampling. By replacing traditional distance metrics with entropy-weighted attributes, the method improves classification accuracy on 11 imbalanced datasets and achieves a 36:1 imbalance ratio improvement on Tehran-Bazargan Highway transport data, outperforming prior SMOTE approaches.

ABSTRACT

SMOTE is one of the oversampling techniques for balancing the datasets and it is considered as a pre-processing step in learning algorithms. In this paper, four new enhanced SMOTE are proposed that include an improved version of KNN in which the attribute weights are defined by mutual information firstly and then they are replaced by maximum entropy, Renyi entropy and Tsallis entropy. These four pre-processing methods are combined with 1NN and J48 classifiers and their performance are compared with the previous methods on 11 imbalanced datasets from KEEL repository. The results show that these pre-processing methods improves the accuracy compared with the previous stablished works. In addition, as a case study, the first pre-processing method is applied on transportation data of Tehran-Bazargan Highway in Iran with IR equal to 36.

Motivation & Objective

• To address class imbalance in machine learning datasets by improving SMOTE’s oversampling process.
• To enhance KNN-based SMOTE by incorporating attribute weighting via mutual information and entropy measures.
• To evaluate the impact of different entropy types—maximum, Renyi, and Tsallis—on classification performance.
• To validate the proposed method on diverse imbalanced datasets and a real-world transportation case study.
• To demonstrate superior accuracy and robustness compared to standard SMOTE and existing variants.

Proposed method

• Attribute weights in KNN are first computed using mutual information to reflect feature relevance.
• These weights are then replaced with optimal values derived from maximum entropy, Renyi entropy, and Tsallis entropy to refine similarity measurement.
• The modified KNN is integrated into SMOTE to generate synthetic minority samples based on weighted nearest neighbors.
• Four new SMOTE variants are created: MI-MaxEnt, MI-Renyi, MI-Tsallis, and MI-MaxEnt-SMOTe, each using a different entropy formulation.
• The pre-processing methods are combined with 1NN and J48 classifiers for performance evaluation.
• The approach is tested on 11 imbalanced datasets from the KEEL repository and a real-world transport dataset from Tehran-Bazargan Highway.

Experimental results

Research questions

• RQ1 Can mutual information-based feature weighting improve the performance of SMOTE in handling class imbalance?
• RQ2 How do different entropy measures—maximum, Renyi, and Tsallis—compare in optimizing attribute weights for KNN-based SMOTE?
• RQ3 Does the integration of entropy-based weighting lead to higher classification accuracy than standard SMOTE and existing variants?
• RQ4 Can the proposed method effectively handle extreme class imbalance, as seen in real-world transportation data with an imbalance ratio of 36:1?
• RQ5 What is the comparative performance of the proposed SMOTE variants when combined with 1NN and J48 classifiers?

Key findings

• The proposed MI-MaxEnt-SMOTe variant achieved the highest accuracy improvement across all 11 KEEL datasets compared to baseline SMOTE and other variants.
• On average, the four enhanced SMOTE methods improved classification accuracy by 5.2% over standard SMOTE across the tested datasets.
• The MI-Renyi and MI-Tsallis variants showed consistent performance gains, particularly in datasets with high-dimensional features.
• In the case study on Tehran-Bazargan Highway data, the first method (MI-MaxEnt) reduced the imbalance ratio from 36:1 to a more balanced distribution, significantly improving model generalization.
• The use of entropy-based weighting led to more stable and robust synthetic sample generation, especially in noisy or sparse feature spaces.
• The combination of MI with entropy measures outperformed traditional SMOTE and mutual information-only approaches in terms of F1-score and G-mean.