[Paper Review] CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognition

Hanting Li, Hongjing Niu+2 more|arXiv (Cornell University)|Mar 1, 2023

Emotion and Mood Recognition4 citations

TL;DR

CLIPER proposes a unified vision-language framework for in-the-wild facial expression recognition (FER) by leveraging CLIP's contrastive pre-training and introducing Multiple Expression Text Descriptors (METD) to learn fine-grained, interpretable text prompts for each expression. It achieves state-of-the-art performance on five in-the-wild FER benchmarks by automatically generating diverse, semantically rich text descriptors that capture variability in real-world expressions.

ABSTRACT

Facial expression recognition (FER) is an essential task for understanding human behaviors. As one of the most informative behaviors of humans, facial expressions are often compound and variable, which is manifested by the fact that different people may express the same expression in very different ways. However, most FER methods still use one-hot or soft labels as the supervision, which lack sufficient semantic descriptions of facial expressions and are less interpretable. Recently, contrastive vision-language pre-training (VLP) models (e.g., CLIP) use text as supervision and have injected new vitality into various computer vision tasks, benefiting from the rich semantics in text. Therefore, in this work, we propose CLIPER, a unified framework for both static and dynamic facial Expression Recognition based on CLIP. Besides, we introduce multiple expression text descriptors (METD) to learn fine-grained expression representations that make CLIPER more interpretable. We conduct extensive experiments on several popular FER benchmarks and achieve state-of-the-art performance, which demonstrates the effectiveness of CLIPER.

Motivation & Objective

• To address the lack of semantic interpretability in existing FER methods that rely on one-hot or soft labels.
• To overcome the limitations of standard CLIP in FER by adapting vision-language pre-training to abstract, compound facial expression categories.
• To develop a unified framework applicable to both static and dynamic facial expression recognition in unconstrained environments.
• To automatically learn diverse, fine-grained text descriptors for each expression class, avoiding manual prompt engineering.
• To improve model interpretability by learning text embeddings that reflect subtle variations in expressions (e.g., 'a little happy' vs. 'very happy').

Proposed method

• Adapts CLIP's contrastive vision-language pre-training to facial expression recognition by using text as supervision instead of one-hot labels.
• Introduces Multiple Expression Text Descriptors (METD), where each expression class is associated with K distinct text embeddings (K=5) learned via a two-stage training process.
• Each METD token is a learnable continuous text embedding that is optimized to align with image features of the same expression, using contrastive loss.
• Uses temporal average pooling to aggregate frame-level features for dynamic FER, avoiding complex temporal modeling.
• Employs a two-stage training paradigm: first, fine-tunes the image encoder using CLIP’s pre-trained vision encoder; second, trains the METD embeddings using contrastive loss with image-text alignment.
• Leverages CLIP’s pre-trained text encoder to initialize and guide the learning of METD, ensuring semantic coherence with natural language.

Experimental results

Research questions

• RQ1 Can contrastive vision-language pre-training be effectively adapted to in-the-wild facial expression recognition, where expression categories are abstract and variable?
• RQ2 How can we automatically generate semantically rich and diverse text descriptors for facial expressions without relying on manual prompt engineering?
• RQ3 Can multiple text descriptors per expression class improve model performance and interpretability compared to single-class prompts?
• RQ4 To what extent does METD capture fine-grained expression variations such as intensity or expression form (e.g., smile vs. laugh)?
• RQ5 Does CLIPER outperform existing SOTA methods on both static and dynamic FER benchmarks in unconstrained settings?

Key findings

• CLIPER achieves state-of-the-art performance on five in-the-wild FER benchmarks, including RAF-DB, AffectNet, and FER2013, outperforming previous SOTA methods.
• The model significantly improves zero-shot generalization on FER datasets by learning more robust and semantically diverse text descriptors through METD.
• Visualization shows that METD learns not only direct expression-related words (e.g., 'saddest' for sad) but also semantically related concepts (e.g., 'zombies' for surprise), indicating interpretability.
• The method successfully learns fine-grained subclasses for expressions: e.g., HA₁ and HA₃ in RAF-DB correspond to subtle intensity differences in 'happy' (smile vs. laugh).
• Despite using only temporal average pooling (no complex temporal modeling), CLIPER achieves strong DFER performance, indicating its strong feature representation capability.
• The full list of nearest words to learned METD tokens confirms that the model captures diverse semantic forms of expressions, such as 'roaring' for anger and 'kindness' for happiness.