[Paper Review] Deep Imbalanced Learning for Face Recognition and Attribute Prediction

Chen Huang, Yining Li+2 more|arXiv (Cornell University)|Jun 1, 2018

Face recognition and analysis72 references30 citations

TL;DR

This paper proposes Cluster-based Large Margin Local Embedding (CLMLE), a deep learning method that enforces angular margins between clusters on a hypersphere to mitigate class imbalance in face recognition and attribute prediction. By maintaining inter-cluster margins within and across classes, CLMLE enables more balanced local decision boundaries, achieving state-of-the-art accuracy on benchmark datasets with imbalanced data.

ABSTRACT

Data for face analysis often exhibit highly-skewed class distribution, i.e., most data belong to a few majority classes, while the minority classes only contain a scarce amount of instances. To mitigate this issue, contemporary deep learning methods typically follow classic strategies such as class re-sampling or cost-sensitive training. In this paper, we conduct extensive and systematic experiments to validate the effectiveness of these classic schemes for representation learning on class-imbalanced data. We further demonstrate that more discriminative deep representation can be learned by enforcing a deep network to maintain inter-cluster margins both within and between classes. This tight constraint effectively reduces the class imbalance inherent in the local data neighborhood, thus carving much more balanced class boundaries locally. We show that it is easy to deploy angular margins between the cluster distributions on a hypersphere manifold. Such learned Cluster-based Large Margin Local Embedding (CLMLE), when combined with a simple k-nearest cluster algorithm, shows significant improvements in accuracy over existing methods on both face recognition and face attribute prediction tasks that exhibit imbalanced class distribution.

Motivation & Objective

• To address the challenge of class imbalance in deep representation learning for face recognition and attribute prediction.
• To investigate whether classic re-sampling and cost-sensitive learning strategies are sufficient for deep imbalanced learning.
• To develop a method that enforces discriminative clustering with inter-cluster margins to improve local decision boundary balance.
• To demonstrate the effectiveness of CLMLE on real-world imbalanced face datasets and simulated benchmarks.
• To show that simple k-neighborhood classification with CLMLE outperforms complex baselines.

Proposed method

• CLMLE learns deep representations by enforcing angular margins between clusters on a hyperspherical manifold, improving inter-class and intra-class separation.
• The method uses a cluster-based sampling strategy that prioritizes clusters with higher observed loss during training.
• A cost-sensitive scheme re-balances class contributions at the score level, especially effective in multi-label attribute settings.
• Feature embeddings are optimized using a margin-based loss that constrains the angular distance between cluster centers.
• The final classifier uses k-nearest cluster (k-NC) instead of instance-wise k-NN, improving speed and performance.
• The approach is trained end-to-end with standard backpropagation, integrating margin learning with standard cross-entropy loss.

Experimental results

Research questions

• RQ1 Can classic re-sampling and cost-sensitive learning strategies effectively mitigate deep imbalanced learning in face analysis?
• RQ2 Does enforcing inter-cluster margins on a hypersphere lead to better local decision boundaries in imbalanced data?
• RQ3 How does CLMLE compare to state-of-the-art methods on real-world imbalanced face recognition and attribute prediction benchmarks?
• RQ4 What is the contribution of cluster-level sampling and cost-sensitive weighting in CLMLE’s performance?
• RQ5 Can CLMLE generalize to generic imbalanced image classification tasks beyond face analysis?

Key findings

• CLMLE achieves 99.62% accuracy on the LFW face verification benchmark, surpassing previous SOTA methods.
• On the CelebA attribute prediction dataset, CLMLE achieves 88.78% average balanced accuracy, outperforming prior methods.
• Ablation studies show that non-uniform cluster sampling and cost-sensitive learning significantly improve performance, especially in multi-label settings.
• On CIFAR-100 with high imbalance (γ=0.5), CLMLE achieves 39.7% mean class-balanced accuracy, outperforming CE+CRL (37.4%) and LMLE (38.1%).
• Replacing k-NC with instance-wise k-NN reduces attribute prediction accuracy to 88.59%, confirming the benefit of cluster-level classification.
• CLMLE maintains strong performance even under open-set protocols, demonstrating its robustness to unseen classes due to discriminative feature learning.