[Paper Review] From Facial Expression Recognition to Interpersonal Relation Prediction

Zhanpeng Zhang, Ping Luo+2 more|arXiv (Cornell University)|Sep 21, 2016

Emotion and Mood Recognition75 references16 citations

TL;DR

This paper proposes a novel Siamese deep learning framework that predicts fine-grained interpersonal relation traits—such as friendliness, dominance, and warmth—from face images by leveraging a multitask convolutional network trained on heterogeneous datasets with missing labels. The method uses attribute propagation via Markov Random Fields to unify disparate data sources, achieving state-of-the-art performance on facial expression recognition and enabling accurate, context-aware interpersonal relation prediction on real-world images.

ABSTRACT

Interpersonal relation defines the association, e.g., warm, friendliness, and dominance, between two or more people. Motivated by psychological studies, we investigate if such fine-grained and high-level relation traits can be characterized and quantified from face images in the wild. We address this challenging problem by first studying a deep network architecture for robust recognition of facial expressions. Unlike existing models that typically learn from facial expression labels alone, we devise an effective multitask network that is capable of learning from rich auxiliary attributes such as gender, age, and head pose, beyond just facial expression data. While conventional supervised training requires datasets with complete labels (e.g., all samples must be labeled with gender, age, and expression), we show that this requirement can be relaxed via a novel attribute propagation method. The approach further allows us to leverage the inherent correspondences between heterogeneous attribute sources despite the disparate distributions of different datasets. With the network we demonstrate state-of-the-art results on existing facial expression recognition benchmarks. To predict inter-personal relation, we use the expression recognition network as branches for a Siamese model. Extensive experiments show that our model is capable of mining mutual context of faces for accurate fine-grained interpersonal prediction.

Motivation & Objective

• To investigate whether fine-grained interpersonal relation traits like friendliness, dominance, and warmth can be predicted from face images in the wild.
• To address the challenge of learning from heterogeneous face datasets with incomplete or missing attribute annotations (e.g., gender, age, pose, expression).
• To develop a deep learning framework that jointly models pairwise facial interactions for high-level relational reasoning beyond single-person expression recognition.
• To enable end-to-end learning of interpersonal relation prediction using a Siamese architecture initialized with a multitask facial representation network.

Proposed method

• A multitask deep convolutional network is trained to jointly recognize facial expressions, gender, age, and head pose using a combination of existing datasets and a newly collected large-scale ExpW dataset with over 90,000 web images.
• A novel attribute propagation method based on Markov Random Fields (MRFs) is introduced to infer missing attribute labels during training, enabling effective learning across datasets with disparate distributions and incomplete annotations.
• The facial representation network is used as shared trunk in a Siamese-like architecture, where two face inputs are processed in parallel with shared weights and their features are fused for joint relation reasoning.
• The Siamese network is trained end-to-end to predict eight interpersonal relation traits (e.g., friendly, competitive, dominant) from pairwise face images.
• Spatial context cues (e.g., relative face positions) are incorporated into the final model to improve relational reasoning, enhancing performance on complex social interactions.
• The framework is evaluated on a movie frame subset with objective annotations, demonstrating robustness and reliability in real-world settings.

Experimental results

Research questions

• RQ1 Can interpersonal relation traits such as friendliness, dominance, and warmth be accurately predicted from face images without explicit textual or contextual cues?
• RQ2 How can deep neural networks be effectively trained on heterogeneous face datasets with missing or inconsistent attribute annotations?
• RQ3 To what extent does joint modeling of two faces in a pair improve the prediction of high-level relational traits compared to single-face analysis?
• RQ4 Can attribute propagation via MRFs effectively bridge the domain gap between datasets with different statistical distributions and labeling completeness?

Key findings

• The proposed S-DCN model achieves a balanced accuracy of 70.20% on the movie testing subset for interpersonal relation prediction, outperforming all ablated variants.
• The full S-DCN model with spatial cues achieves the highest performance, indicating that spatial context significantly improves relational reasoning.
• The model with DCN pre-trained on expression labels yields the best performance (65.36%) among baseline variants, highlighting the importance of strong facial representation learning.
• Qualitative results show the model successfully captures dynamic social interactions, such as friendly conversations and conflicts, as seen in Iron Man scenes with high probability alignment to expected traits.
• False positives are often due to lack of contextual information (e.g., misinterpreting a serious book-reading scene as competitive), indicating a need for multimodal context integration.
• The model predicts high dominance between Obama and his daughter and high competitiveness between Merkel and Obama, aligning with real-world social perceptions.