[Paper Review] Deep Reinforcement Learning for Imbalanced Classification

Enlu Lin, Qiong Chen+1 more|arXiv (Cornell University)|Jan 5, 2019

Imbalanced Data Classification Techniques36 references19 citations

TL;DR

This paper proposes DQNimb, a deep reinforcement learning framework that formulates imbalanced classification as a sequential decision-making process using deep Q-learning. By assigning higher rewards to minority class samples and using a tailored reward function, the agent learns an optimal classification policy that significantly improves detection of rare instances, outperforming traditional methods—especially in extreme class imbalance scenarios like text datasets.

ABSTRACT

Data in real-world application often exhibit skewed class distribution which poses an intense challenge for machine learning. Conventional classification algorithms are not effective in the case of imbalanced data distribution, and may fail when the data distribution is highly imbalanced. To address this issue, we propose a general imbalanced classification model based on deep reinforcement learning. We formulate the classification problem as a sequential decision-making process and solve it by deep Q-learning network. The agent performs a classification action on one sample at each time step, and the environment evaluates the classification action and returns a reward to the agent. The reward from minority class sample is larger so the agent is more sensitive to the minority class. The agent finally finds an optimal classification policy in imbalanced data under the guidance of specific reward function and beneficial learning environment. Experiments show that our proposed model outperforms the other imbalanced classification algorithms, and it can identify more minority samples and has great classification performance.

Motivation & Objective

• Address the challenge of poor minority class detection in highly imbalanced real-world datasets such as disease diagnosis and anomaly detection.
• Overcome limitations of conventional machine learning models that favor majority class performance due to skewed data distributions.
• Develop a novel deep reinforcement learning framework that dynamically adjusts learning focus toward minority samples through reward shaping.
• Improve classification performance on extreme imbalanced datasets by combining experience replay (data-level) and reward-based learning (algorithmic-level).

Proposed method

• Formulate the imbalanced classification problem as an Interpretable Classification Markov Decision Process (ICMDP), treating each sample as a sequential decision step.
• Use a deep Q-network (DQN) to learn the optimal classification policy through interaction with an environment that provides rewards based on prediction correctness.
• Design a reward function where minority class samples receive a reward of 1 and majority class samples receive a reward of λ, with λ set to the imbalance ratio ρ to balance learning impact.
• Implement experience replay to store and replay past interactions, reducing data skew in memory and improving generalization.
• Introduce episode termination upon misclassifying a minority sample, reinforcing learning focus on rare instances.
• The Q-network is trained to minimize the loss function that balances contributions from both majority and minority samples via adjusted reward scaling.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning effectively address class imbalance by learning a policy that prioritizes minority class samples?
• RQ2 How does the choice of reward function—particularly λ for majority class—impact the model's ability to generalize and detect rare instances?
• RQ3 Does combining data-level (experience replay) and algorithmic-level (reward shaping) strategies in a DQN framework improve performance on extreme imbalanced datasets?
• RQ4 How does the DQNimb model compare to deep neural networks and traditional imbalanced learning methods in terms of G-mean and F-measure under varying imbalance ratios?
• RQ5 What is the optimal value of λ in the reward function to balance the influence of majority and minority samples during training?

Key findings

• On the IMDB text dataset, DQNimb achieved a G-mean score 7.7% higher than the second-best method (CSM) and 12.3% higher when the imbalance ratio was 2%.
• The F-measure of DQNimb declined less sharply than other models as imbalance increased, showing superior robustness to extreme class skew.
• The DNN model suffered the most severe performance drop in F-measure under high imbalance, while DQNimb maintained strong performance due to targeted learning on minority samples.
• The optimal reward value for majority class samples was found to be λ = ρ, where ρ is the imbalance ratio; deviating from this value degraded performance.
• Experience replay in DQNimb helped mitigate data distribution skew in memory, contributing to better generalization and reduced overfitting.
• Theoretical analysis confirmed that adjusting λ balances the loss function contributions from both classes, enabling stable training and improved minority class detection.