[Paper Review] Multi-model Ensemble Learning Method for Human Expression Recognition

Jun Yu, Zhongpeng Cai+3 more|arXiv (Cornell University)|Mar 28, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a multi-model ensemble learning method for human expression recognition in wild video data, combining ResNet, EfficientNet, and InceptionNet backbones with focal loss to address class imbalance. The approach uses both intra-fold and cross-fold ensemble strategies on the AffWild2 dataset, achieving improved F1-score through model fusion and data distribution diversity, demonstrating strong performance in real-world expression recognition scenarios.

ABSTRACT

Analysis of human affect plays a vital role in human-computer interaction (HCI) systems. Due to the difficulty in capturing large amounts of real-life data, most of the current methods have mainly focused on controlled environments, which limit their application scenarios. To tackle this problem, we propose our solution based on the ensemble learning method. Specifically, we formulate the problem as a classification task, and then train several expression classification models with different types of backbones--ResNet, EfficientNet and InceptionNet. After that, the outputs of several models are fused via model ensemble method to predict the final results. Moreover, we introduce the multi-fold ensemble method to train and ensemble several models with the same architecture but different data distributions to enhance the performance of our solution. We conduct many experiments on the AffWild2 dataset of the ABAW2022 Challenge, and the results demonstrate the effectiveness of our solution.

Motivation & Objective

• To address the challenge of recognizing human facial expressions in unconstrained, real-world (in-the-wild) video environments where data is scarce and complex.
• To improve recognition performance by leveraging diverse deep learning architectures (ResNet, EfficientNet, InceptionNet) for feature extraction.
• To mitigate class imbalance in the AffWild2 dataset using focal loss during model training.
• To enhance model generalization and robustness by applying a multi-fold ensemble strategy with different data splits.
• To achieve state-of-the-art performance in the ABAW2022 Expression Classification Challenge using ensemble-based fusion of multiple models.

Proposed method

• Trained three separate expression classification models using ResNet50, EfficientNet-b0, and InceptionNet-v1 as backbones, each initialized with pre-trained weights from VGGFace2 or ImageNet for better feature learning.
• Applied data augmentation techniques including RandomHorizontalFlipping and ColorJitter to increase training data diversity and improve model generalization.
• Employed a two-stage ensemble strategy: first, fused outputs from different backbone models using learnable fusion weights; second, applied five-fold cross-validation to train and ensemble models with the same architecture but different data splits.
• Used focal loss as the loss function to address class imbalance in the AffWild2 dataset, defined as $\mathbf{F}\mathbf{L}(p)=\sum_{i=1}^{m}y_{i}\alpha_{i}(1-p_{i})^{\gamma_{i}}\log(p_{i})$, with class-specific weights $\alpha_i$ and focusing parameter $\gamma$.
• Optimized the training process using the Adam optimizer with a cosine annealing learning rate scheduler and a batch size of 256 on NVIDIA Tesla A100 GPUs.
• Fused final predictions from all models using weighted averaging to maximize performance, with fusion weights tuned based on validation performance.

Experimental results

Research questions

• RQ1 Can combining multiple deep learning architectures with different structural designs improve expression recognition performance in wild video data?
• RQ2 How effective is the use of focal loss in mitigating class imbalance in the AffWild2 dataset for expression classification?
• RQ3 Does training and ensembling models across multiple data folds with varying data distributions enhance model robustness and generalization?
• RQ4 To what extent does model ensemble learning outperform single-model baselines in the in-the-wild expression recognition setting?
• RQ5 What is the optimal fusion strategy for combining predictions from diverse models trained on different backbones and data splits?

Key findings

• The multi-model ensemble method achieved a final average F1-score of 0.421 on the AffWild2 test set, significantly outperforming individual models.
• ResNet50 achieved the highest F1-score of 0.317 in Fold 1, while EfficientNet-b0 and InceptionNet-v1 showed competitive performance across folds, indicating complementary feature learning.
• The cross-fold ensemble strategy improved model performance by reducing variance and enhancing generalization, especially when combining models trained on different data splits.
• The use of focal loss led to better convergence and improved F1-scores on minority classes, effectively addressing the long-tail distribution in the AffWild2 dataset.
• The final ensemble model, fusing results from multiple fusion configurations, achieved the best overall performance, confirming the effectiveness of the proposed multi-model and multi-fold ensemble approach.
• The method demonstrated strong robustness and generalization in unconstrained environments, validating its suitability for real-world human-computer interaction applications.