[Paper Review] Graph-based Facial Affect Analysis: A Review of Methods, Applications and Challenges.

Yang Liu, Xingming Zhang+4 more|arXiv (Cornell University)|Mar 29, 2021

Emotion and Mood RecognitionPsychology179 references4 citations

TL;DR

This paper presents the first comprehensive survey of graph-based facial affect analysis (FAA), proposing a framework that models semantic relationships among facial components using graph structures to improve affect recognition. It reviews graph construction methods, relational reasoning via traditional and deep learning models—especially graph neural networks—and benchmarks state-of-the-art methods, highlighting performance gains and open challenges in the field.

ABSTRACT

Facial affect analysis (FAA) using visual signals is important in human-computer interaction. Early methods focus on extracting appearance and geometry features associated with human affects, while ignoring the latent semantic information among individual facial changes, leading to limited performance and generalization. Recent work attempts to establish a graph-based representation to model these semantic relationships and develop frameworks to leverage them for various FAA tasks. In this paper, we provide a comprehensive review of graph-based FAA, including the evolution of algorithms and their applications. First, the FAA background knowledge is introduced, especially on the role of the graph. We then discuss approaches that are widely used for graph-based affective representation in literature and show a trend towards graph construction. For the relational reasoning in graph-based FAA, existing studies are categorized according to their usage of traditional methods or deep models, with a special emphasis on the latest graph neural networks. Performance comparisons of the state-of-the-art graph-based FAA methods are also summarized. Finally, we discuss the challenges and potential directions. As far as we know, this is the first survey of graph-based FAA methods. Our findings can serve as a reference for future research in this field.

Motivation & Objective

• To provide a systematic review of graph-based approaches in facial affect analysis (FAA), addressing limitations in traditional feature extraction methods.
• To analyze the evolution of graph construction techniques for modeling semantic relationships among facial components.
• To categorize and evaluate relational reasoning methods in FAA, focusing on traditional models and deep learning, particularly graph neural networks.
• To summarize performance comparisons of state-of-the-art graph-based FAA methods and identify key trends.
• To outline open challenges and future research directions in graph-based FAA for improved generalization and performance.

Proposed method

• The paper employs a structured literature review methodology to analyze graph-based FAA methods across algorithmic development, graph construction, and relational reasoning.
• It classifies graph construction approaches based on facial keypoint locations, semantic facial regions, or learned representations to model facial dynamics.
• Relational reasoning is examined through two categories: traditional methods (e.g., graph matching, kernel methods) and deep learning models, especially graph neural networks (GNNs).
• The review evaluates state-of-the-art methods using standardized benchmarks, comparing performance across datasets and task types.
• Core components include graph convolutional networks (GCNs), graph attention networks (GATs), and message-passing mechanisms for feature propagation in facial graphs.
• The framework emphasizes modeling latent semantic relationships among facial changes to enhance affect recognition beyond appearance and geometry alone.

Experimental results

Research questions

• RQ1 How have graph-based representations evolved in facial affect analysis to address limitations of traditional appearance and geometry-based methods?
• RQ2 What are the dominant approaches for constructing graphs from facial data, and how do they influence affect recognition performance?
• RQ3 How do traditional versus deep learning-based relational reasoning methods compare in graph-based FAA frameworks?
• RQ4 What are the key performance gains achieved by state-of-the-art graph-based FAA models over non-graph baselines?
• RQ5 What major challenges remain in graph-based FAA, and what future research directions are most promising?

Key findings

• Graph-based FAA methods significantly outperform traditional appearance and geometry-based approaches by modeling latent semantic relationships among facial components.
• A clear trend toward learned graph construction is observed, where attention mechanisms and learnable embeddings improve graph representation quality.
• Graph neural networks (GNNs), particularly GCNs and GATs, have become the dominant architecture for relational reasoning in FAA, enabling effective feature propagation and aggregation.
• Performance comparisons show consistent improvements in affect recognition accuracy across multiple benchmark datasets when using graph-based frameworks.
• Despite progress, challenges remain in generalization across diverse datasets, interpretability of learned graphs, and robustness to variations in lighting, pose, and occlusion.