[Paper Review] Using Self-Supervised Co-Training to Improve Facial Representation.

Mahdi Pourmirzaei, Farzaneh Esmaili+1 more|arXiv (Cornell University)|May 13, 2021

Face recognition and analysisComputer Science45 references9 citations

TL;DR

This paper proposes Hybrid Learning (HL), a multi-task framework that combines self-supervised co-training with supervised learning for facial representation learning. By jointly training on supervised facial expression recognition and self-supervised pre-tasks like jigsaw puzzling and in-painting, HL improves performance on AffectNet without additional data, reduces error by up to 9% on head pose estimation, and mitigates overfitting under low-data and strong-augmentation settings.

ABSTRACT

In this paper, at first, the impact of ImageNet pre-training on Facial Expression Recognition (FER) was tested under different augmentation levels. It could be seen from the results that training from scratch could reach better performance compared to ImageNet fine-tuning at stronger augmentation levels. After that, a framework was proposed for standard Supervised Learning (SL), called Hybrid Learning (HL) which used Self-Supervised co-training with SL in Multi-Task Learning (MTL) manner. Leveraging Self-Supervised Learning (SSL) could gain additional information from input data like spatial information from faces which helped the main SL task. It is been investigated how this method could be used for FER problems with self-supervised pre-tasks such as Jigsaw puzzling and in-painting. The supervised head (SH) was helped by these two methods to lower the error rate under different augmentations and low data regime in the same training settings. The state-of-the-art was reached on AffectNet via two completely different HL methods, without utilizing additional datasets. Moreover, HL's effect was shown on two different facial-related problem, head poses estimation and gender recognition, which concluded to reduce in error rate by up to 9% and 1% respectively. Also, we saw that the HL methods prevented the model from reaching overfitting.

Motivation & Objective

• To investigate whether self-supervised pre-training improves facial representation learning in low-data and high-augmentation regimes.
• To develop a multi-task learning framework that integrates self-supervised co-training with supervised learning for facial expression recognition.
• To evaluate the effectiveness of self-supervised pre-tasks—jigsaw puzzling and in-painting—on improving generalization and reducing overfitting.
• To demonstrate the transferability of the proposed method to other facial analysis tasks beyond expression recognition.
• To achieve state-of-the-art performance on AffectNet without using external datasets.

Proposed method

• The proposed Hybrid Learning (HL) framework combines supervised learning with self-supervised co-training in a multi-task learning setup.
• Two self-supervised pre-tasks—jigsaw puzzling and in-painting—are used to extract spatial and structural information from facial images.
• The model jointly optimizes the supervised head for facial expression recognition and the self-supervised heads for pre-tasks during training.
• The framework is trained end-to-end with shared feature extractors, allowing knowledge distillation from self-supervised tasks to the main supervised task.
• The method is evaluated under varying augmentation levels and data regimes to assess robustness and generalization.
• The approach is extended to other facial analysis tasks, including head pose estimation and gender recognition, to validate transferability.

Experimental results

Research questions

• RQ1 Does self-supervised co-training improve performance on facial expression recognition under strong data augmentation and low-data regimes?
• RQ2 Can self-supervised pre-tasks such as jigsaw puzzling and in-painting enhance the representation capacity of a supervised facial recognition model?
• RQ3 Does the proposed Hybrid Learning framework reduce overfitting compared to standard supervised training or ImageNet fine-tuning?
• RQ4 To what extent does the HL framework generalize to other facial analysis tasks beyond expression recognition?
• RQ5 Can state-of-the-art performance on AffectNet be achieved without using additional datasets?

Key findings

• Training from scratch outperformed ImageNet fine-tuning under strong data augmentation, indicating that pre-training may not always benefit FER in high-augmentation settings.
• The Hybrid Learning framework reduced error by up to 9% on head pose estimation and by 1% on gender recognition compared to baseline supervised learning.
• The HL framework achieved state-of-the-art performance on AffectNet without relying on external datasets.
• Self-supervised pre-tasks provided complementary information, particularly spatial structure, which improved the main supervised task's generalization.
• The method effectively prevented overfitting, especially under low-data and high-augmentation conditions.
• The improvement was consistent across different augmentation levels, demonstrating robustness of the proposed co-training strategy.