[Paper Review] NR-DFERNet: Noise-Robust Network for Dynamic Facial Expression Recognition

Hanting Li, Mingzhe Sui+2 more|arXiv (Cornell University)|Jun 10, 2022

Emotion and Mood Recognition20 citations

TL;DR

NR-DFERNet introduces a noise-robust architecture for DFER by fusing dynamic and static spatial features, using a dynamic class token for temporal framing, and a snippet-based filter at decision time to mitigate noisy frames in wild video data. It achieves state-of-the-art results on DFEW and competitive results on AFEW.

ABSTRACT

Dynamic facial expression recognition (DFER) in the wild is an extremely challenging task, due to a large number of noisy frames in the video sequences. Previous works focus on extracting more discriminative features, but ignore distinguishing the key frames from the noisy frames. To tackle this problem, we propose a noise-robust dynamic facial expression recognition network (NR-DFERNet), which can effectively reduce the interference of noisy frames on the DFER task. Specifically, at the spatial stage, we devise a dynamic-static fusion module (DSF) that introduces dynamic features to static features for learning more discriminative spatial features. To suppress the impact of target irrelevant frames, we introduce a novel dynamic class token (DCT) for the transformer at the temporal stage. Moreover, we design a snippet-based filter (SF) at the decision stage to reduce the effect of too many neutral frames on non-neutral sequence classification. Extensive experimental results demonstrate that our NR-DFERNet outperforms the state-of-the-art methods on both the DFEW and AFEW benchmarks.

Motivation & Objective

• Motivate robust DFER under real-world noise by distinguishing key frames from noisy frames.
• Develop spatial, temporal, and decision-stage components that suppress noise while preserving expression signals.
• Demonstrate that dynamic-static fusion improves spatial discriminability and robustness.
• Show that a dynamic class token helps the transformer focus on target-related frames.
• Prove the effectiveness of a snippet-based filter at the decision stage for handling excessive neutral frames.

Proposed method

• Dynamic-static fusion module that combines dynamic frame differences with downsampled static features.
• A transformer-based temporal stage using a dynamic class token to attenuate target-irrelevant frames.
• Snippet-based filter at the decision stage to down-weight neutral-dominated snippets and favor non-neutral judgments.
• Dynamic sampling of clips during training/testing to create length-16 sequences.
• Ablation studies validating DSF, DCT, and SF contributions to UAR and WAR.
• Experiments on DFEW and AFEW benchmarks with cross-validation and pre-training strategy.

Experimental results

Research questions

• RQ1 Can NR-DFERNet suppress the influence of noisy frames (N1 and N2) in wild DFER sequences?
• RQ2 Does integrating dynamic and static spatial features improve discriminability for dynamic expressions?
• RQ3 Does a dynamic class token improve temporal frame selection during attention-based fusion?
• RQ4 Is a snippet-based filter at the decision stage effective in reducing misclassification caused by excess neutral frames?
• RQ5 How does NR-DFERNet perform against state-of-the-art methods on DFEW and AFEW datasets?

Key findings

• NR-DFERNet with DSF, DCT, and SF achieves state-of-the-art UAR/WAR on DFEW (e.g., 88.47/53.54 with standard setup).
• Dynamic-static fusion consistently improves both WAR and UAR over baselines.
• Dynamic class token provides additional gains by reducing interference from target-irrelevant frames; e.g., UAR/WAR improvements of ~1.15/1.95 percentage points in ablations.
• Snippet-based filter yields gains that grow as overall model accuracy improves, validating its role in handling neutral-frame dominance.
• On AFEW, NR-DFERNet outperforms several baselines and approaches Former-DFER when pre-trained on DFEW and fine-tuned on AFEW.
• Across DFEW with oversampling, NR-DFERNet surpasses Former-DFER by 2.08% UAR and 2.31% WAR under class-imbalance mitigation.