[Paper Review] Learning to Amend Facial Expression Representation via De-albino and Affinity

Jiawei Shi, Songhao Zhu+1 more|arXiv (Cornell University)|Mar 18, 2021

Emotion and Mood Recognition39 references62 citations

TL;DR

Introduces the Amend Representation Module (ARM) to replace pooling in CNNs, addressing padding erosion (albino features) and leveraging affinity among expressions to boost FER accuracy; achieves state-of-the-art results on RAF-DB, AffectNet, and SFEW.

ABSTRACT

Facial Expression Recognition (FER) is a classification task that points to face variants. Hence, there are certain affinity features between facial expressions, receiving little attention in the FER literature. Convolution padding, despite helping capture the edge information, causes erosion of the feature map simultaneously. After multi-layer filling convolution, the output feature map named albino feature definitely weakens the representation of the expression. To tackle these challenges, we propose a novel architecture named Amending Representation Module (ARM). ARM is a substitute for the pooling layer. Theoretically, it can be embedded in the back end of any network to deal with the Padding Erosion. ARM efficiently enhances facial expression representation from two different directions: 1) reducing the weight of eroded features to offset the side effect of padding, and 2) decomposing facial features to simplify representation learning. Experiments on public benchmarks prove that our ARM boosts the performance of FER remarkably. The validation accuracies are respectively 90.42% on RAF-DB, 65.2% on Affect-Net, and 58.71% on SFEW, exceeding current state-of-the-art methods. Our implementation and trained models are available at https://github.com/JiaweiShiCV/Amend-Representation-Module.

Motivation & Objective

• Identify and mitigate padding-induced albino feature erosion in CNN-based FER.
• Propose ARM as a pooling substitute to preserve expression-relevant information.
• Exploit affinity among facial expressions to simplify representation learning.
• Demonstrate gains on standard FER datasets and provide ablations to validate components.

Proposed method

• Replace pooling with De-albino (DA) block to suppress edge-eroded features.
• Use Feature Arrangement (FA) block to rearrange feature maps without padding.
• Incorporate Sharing Affinity (SA) block to model generic vs. unique expression components via EMA-based affinity.
• Embed ARM in ResNet-18 backbones and train with Adam optimizer and standard FER protocols.
• Optionally apply minimal random resampling (MRR) to balance AffectNet data during fine-tuning.

Experimental results

Research questions

• RQ1 Can padding-induced albino features degrade FER representations and can this be mitigated without padding?
• RQ2 Does a module that rearranges and reweights feature maps improve FER performance across wild datasets?
• RQ3 Do affinity-based representations between expressions provide measurable gains in FER tasks?

Key findings

• ARM improves ResNet-18 performance from 86.4 to 90.42 WA on RAF-DB (state-of-the-art).
• ARM achieves 65.2% accuracy on AffectNet (7-class) and 58.71% on SFEW (7-class/8-class variants referenced).
• Ablation shows DA and SA blocks provide clear gains, with FA assisting DA for further improvement.
• ARM outperforms several baselines and state-of-the-art FER methods on the evaluated datasets.