[Paper Review] Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training

Darshan Gera, S. Balasubramanian|arXiv (Cornell University)|Jul 8, 2021

Emotion and Mood Recognition34 references4 citations

TL;DR

This paper proposes Consensual Collaborative Training (CCT), a robust deep learning framework for facial expression recognition in the wild that mitigates noisy annotations by co-training three networks using a dynamic blend of supervision and consistency losses. The method outperforms multiple competing approaches on the Aff-Wild2 dataset, achieving a 0.4814 overall score on the validation set with a 3-network setup and pretraining on AffectNet, demonstrating superior generalization under noisy real-world conditions without relying on audio or landmark features.

ABSTRACT

Facial expression recognition (FER) in the wild is crucial for building reliable human-computer interactive systems. However, annotations of large scale datasets in FER has been a key challenge as these datasets suffer from noise due to various factors like crowd sourcing, subjectivity of annotators, poor quality of images, automatic labelling based on key word search etc. Such noisy annotations impede the performance of FER due to the memorization ability of deep networks. During early learning stage, deep networks fit on clean data. Then, eventually, they start overfitting on noisy labels due to their memorization ability, which limits FER performance. This report presents Consensual Collaborative Training (CCT) framework used in our submission to expression recognition track of the Affective Behaviour Analysis in-the-wild (ABAW) 2021 competition. CCT co-trains three networks jointly using a convex combination of supervision loss and consistency loss, without making any assumption about the noise distribution. A dynamic transition mechanism is used to move from supervision loss in early learning to consistency loss for consensus of predictions among networks in the later stage. Co-training reduces overall error, and consistency loss prevents overfitting to noisy samples. The performance of the model is validated on challenging Aff-Wild2 dataset for categorical expression classification. Our code is made publicly available at https://github.com/1980x/ABAW2021DMACS.

Motivation & Objective

• To address the challenge of noisy annotations in large-scale in-the-wild facial expression datasets like Aff-Wild2, which degrade model performance due to deep network memorization.
• To develop a noise-robust training framework that does not require prior knowledge of noise distribution or architectural constraints.
• To improve generalization in facial expression recognition by leveraging consensus among multiple networks during training.
• To validate the effectiveness of the proposed method on the challenging Aff-Wild2 dataset in the ABAW 2021 competition.

Proposed method

• CCT co-trains three deep neural networks with identical architecture but different random initializations to promote diverse learning paths and reduce individual error accumulation.
• The training loss is a convex combination of cross-entropy supervision loss and consistency loss, dynamically balanced via a Gaussian-like ramp-up function that shifts from supervision to consensus-building over time.
• Consistency loss enforces agreement among the three networks' predictions by aligning their posterior distributions, preventing overfitting to noisy labels.
• The dynamic weighting factor λ transitions from emphasizing supervision loss early in training to prioritizing consistency loss later, enabling robust learning on clean data first and consensus on hard samples later.
• The framework is architecture-agnostic and does not assume any specific noise distribution, making it broadly applicable to noisy FER datasets.
• Pretraining on large-scale datasets like AffectNet improves performance, and ablation studies confirm the effectiveness of the three-network setup and consistency loss.

Experimental results

Research questions

• RQ1 Can co-training three deep networks with a dynamic loss transition reduce the impact of noisy annotations in facial expression recognition?
• RQ2 How does the dynamic balance between supervision and consistency loss affect model generalization on noisy in-the-wild datasets?
• RQ3 What is the optimal number of networks for collaborative training in the presence of label noise?
• RQ4 Does pretraining on large-scale datasets like AffectNet improve performance in the CCT framework?
• RQ5 How does the inclusion of consistency loss compare to training with supervision loss alone?

Key findings

• The three-network CCT setup achieved the highest performance on the Aff-Wild2 validation set with an overall score of 0.4814, outperforming models with one, two, or four networks.
• Using a dynamic balancing factor β = 4.0 yielded the best performance, with an overall score of 0.4814, indicating optimal transition from supervision to consistency loss.
• Pretraining on AffectNet significantly improved performance (F1: 0.4040, Accuracy: 0.6378) compared to RAFDB or MS-Celeb-1M, confirming its value for large-scale FER.
• Oversampling did not improve performance and was therefore excluded from final training, with the best model trained without oversampling.
• The inclusion of consistency loss improved the overall score from 0.445 (using only CE loss) to 0.454, demonstrating its effectiveness in reducing overfitting to noisy labels.
• The model achieved a test set F1 score of 0.4646, placing it among the top submissions in the ABAW 2021 expression classification track despite not using audio or video features.