[Paper Review] POSTER++: A simpler and stronger facial expression recognition network

Jiawei Mao, Rui Xu+4 more|arXiv (Cornell University)|Jan 28, 2023

Emotion and Mood Recognition14 citations

TL;DR

POSTER++ improves POSTER by removing the image-to-landmark branch, applying window-based cross-attention, and direct multi-scale feature extraction with a small ViT, achieving state-of-the-art FER performance with lower computation (8.4G FLOPs, 43.7M params).

ABSTRACT

Facial expression recognition (FER) plays an important role in a variety of real-world applications such as human-computer interaction. POSTER achieves the state-of-the-art (SOTA) performance in FER by effectively combining facial landmark and image features through two-stream pyramid cross-fusion design. However, the architecture of POSTER is undoubtedly complex. It causes expensive computational costs. In order to relieve the computational pressure of POSTER, in this paper, we propose POSTER++. It improves POSTER in three directions: cross-fusion, two-stream, and multi-scale feature extraction. In cross-fusion, we use window-based cross-attention mechanism replacing vanilla cross-attention mechanism. We remove the image-to-landmark branch in the two-stream design. For multi-scale feature extraction, POSTER++ combines images with landmark's multi-scale features to replace POSTER's pyramid design. Extensive experiments on several standard datasets show that our POSTER++ achieves the SOTA FER performance with the minimum computational cost. For example, POSTER++ reached 92.21% on RAF-DB, 67.49% on AffectNet (7 cls) and 63.77% on AffectNet (8 cls), respectively, using only 8.4G floating point operations (FLOPs) and 43.7M parameters (Param). This demonstrates the effectiveness of our improvements.

Motivation & Objective

• Address the computational inefficiency of POSTER while preserving or improving FER accuracy.
• Eliminate redundant branches to reduce FLOPs and parameters.
• Introduce efficient cross-fusion and multi-scale feature integration.
• Demonstrate state-of-the-art performance on standard FER benchmarks while being resource-efficient.

Proposed method

• Retain facial landmark detector and image backbone from POSTER.
• Remove the image-to-landmark branch from the two-stream design.
• Replace vanilla cross-attention with window-based cross-attention (W-MCSA) for linear complexity.
• Eliminate the pyramid multi-scale feature extraction; fuse multi-scale features directly from backbone and landmark detector using a two-layer Vision Transformer (ViT).
• Introduce a small ViT to integrate multi-scale features.
• Use window-based cross-fusion to enhance local modeling and reduce computation.

Experimental results

Research questions

• RQ1 Can POSTER++ maintain or improve FER accuracy while substantially reducing parameters and FLOPs compared to POSTER?
• RQ2 What is the contribution of each architectural change (two-stream removal, cross-fusion via windowed attention, and direct multi-scale extraction) to overall performance?
• RQ3 Is window-based cross-attention sufficient to replace vanilla cross-attention in two-stream FER architectures?
• RQ4 Do multi-scale features integrated by a lightweight ViT provide comparable or better results than POSTER's pyramid design?

Key findings

• POSTER++ achieves SOTA FER performance on RAF-DB (92.21%), AffectNet (7 cls: 67.49%), and AffectNet (8 cls: 63.77%).
• On CAER-S, POSTER++ reaches 93.00%, outperforming POSTER (92.73%).
• POSTER++ uses 8.4G FLOPs and 43.7M parameters, reducing both FLOPs and parameters compared to POSTER (8.4G vs 15.7G; 43.7M vs 71.8M).
• Ablation studies show landmark-to-image cross-fusion is critical, while removing the image-to-landmark branch reduces cost with limited accuracy loss.
• Window-based cross-attention (W-MCSA) provides linear computational complexity and improves FER accuracy over vanilla cross-attention.
• Direct multi-scale feature extraction with a two-layer ViT suffices for effective integration of multi-scale information.