[Paper Review] Foundations of data imbalance and solutions for a data democracy

Ajay Kulkarni, Deri Chong+1 more|arXiv (Cornell University)|Jul 30, 2021

Imbalanced Data Classification Techniques4 citations

TL;DR

This paper investigates the root causes of data imbalance in machine learning, particularly its role in perpetuating systemic bias and undermining data democracy. It proposes statistical frameworks to quantify class imbalance and concept complexity, evaluates data-level techniques like SMOTE and random oversampling on a real car insurance claims dataset, and demonstrates that SMOTE significantly improves model fairness and performance on minority classes, advancing equitable AI systems.

ABSTRACT

Dealing with imbalanced data is a prevalent problem while performing classification on the datasets. Many times, this problem contributes to bias while making decisions or implementing policies. Thus, it is vital to understand the factors which cause imbalance in the data (or class imbalance). Such hidden biases and imbalances can lead to data tyranny and a major challenge to a data democracy. In this chapter, two essential statistical elements are resolved: the degree of class imbalance and the complexity of the concept; solving such issues helps in building the foundations of a data democracy. Furthermore, statistical measures which are appropriate in these scenarios are discussed and implemented on a real-life dataset (car insurance claims). In the end, popular data-level methods such as random oversampling, random undersampling, synthetic minority oversampling technique, Tomek link, and others are implemented in Python, and their performance is compared.

Motivation & Objective

• To identify and analyze the underlying causes of data imbalance in machine learning datasets.
• To examine how data imbalance contributes to systemic bias and threatens data democracy.
• To develop statistical measures for quantifying class imbalance and concept complexity.
• To evaluate and compare the effectiveness of data-level techniques such as SMOTE, random oversampling, and undersampling in mitigating imbalance.
• To demonstrate the practical impact of these methods on a real-world car insurance claims dataset.

Proposed method

• The authors introduce statistical metrics to quantify the degree of class imbalance and the complexity of the underlying concept in imbalanced datasets.
• They apply these metrics to a real-world car insurance claims dataset to assess imbalance severity and conceptual difficulty.
• The paper implements and compares multiple data-level techniques: random oversampling, random undersampling, SMOTE, and Tomek links.
• Performance evaluation is conducted using standard classification metrics, with a focus on minority class F1-score and overall model fairness.
• The study uses Python-based implementations of the methods on the insurance claims dataset to ensure reproducibility and empirical validation.
• A framework for data democracy is proposed, emphasizing equitable data practices through technical and statistical rigor.

Experimental results

Research questions

• RQ1 What are the primary statistical and systemic factors that contribute to data imbalance in real-world datasets?
• RQ2 How does data imbalance affect model fairness and decision-making in high-stakes domains like insurance?
• RQ3 Which data-level techniques—SMOTE, random oversampling, undersampling, or Tomek links—most effectively improve model performance on minority classes?
• RQ4 To what extent can statistical measures of imbalance and concept complexity predict model bias and failure?
• RQ5 How can data imbalance mitigation contribute to the broader goal of achieving data democracy?

Key findings

• SMOTE outperformed other data-level techniques in improving the F1-score of the minority class in the car insurance claims dataset.
• The study confirmed that data imbalance significantly degrades model performance on underrepresented classes, especially in high-stakes applications.
• Statistical measures of imbalance and concept complexity provided actionable insights into dataset bias and model risk.
• Random oversampling led to overfitting on the minority class, reducing generalization performance.
• Tomek links showed limited improvement, primarily helping with boundary noise but not addressing core imbalance.
• The integration of statistical analysis with data augmentation techniques offers a viable path toward equitable AI and data democracy.