[Paper Review] Predicting Social Perception from Faces: A Deep Learning Approach

Uwe Messer, Stefan Faußer|arXiv (Cornell University)|Jun 29, 2019

Evolutionary Psychology and Human Behavior31 references4 citations

TL;DR

This paper proposes a deep learning approach using Deep Convolutional Neural Networks and Grad-CAM to predict human perceptions of warmth and competence from face images. The model achieves 90% accuracy in predicting warmth and 80% in predicting competence, identifying key facial regions driving these judgments.

ABSTRACT

Warmth and competence represent the fundamental traits in social judgment that determine emotional reactions and behavioral intentions towards social targets. This research investigates whether an algorithm can learn visual representations of social categorization and accurately predict human perceivers' impressions of warmth and competence in face images. In addition, this research unravels which areas of a face are important for the classification of warmth and competence. We use Deep Convolutional Neural Networks to extract features from face images and the Gradient-weighted Class Activation Mapping (Grad CAM) method to understand the importance of face regions for the classification. Given a single face image the trained algorithm could correctly predict warmth impressions with an accuracy of about 90% and competence impressions with an accuracy of about 80%. The findings have implications for the automated processing of faces and the design of artificial characters.

Motivation & Objective

• To investigate whether deep learning models can accurately predict human social perceptions of warmth and competence from facial images.
• To identify which facial regions contribute most to the perception of warmth and competence using explainable AI techniques.
• To develop a system capable of automated social perception prediction with interpretability for psychological and AI applications.
• To explore the feasibility of modeling fundamental dimensions of social judgment using visual features from faces.
• To contribute to the design of socially intelligent artificial agents by understanding visual cues in face perception.

Proposed method

• Trained Deep Convolutional Neural Networks (DCNNs) on face images to learn representations of social perception.
• Utilized the Gradient-weighted Class Activation Mapping (Grad-CAM) technique to visualize and interpret which facial regions are most influential for predictions.
• Trained separate models for predicting warmth and competence based on human-annotated social perception scores.
• Extracted deep features from face images using pre-trained CNN architectures before fine-tuning on the social perception dataset.
• Applied L2 regularization and dropout during training to prevent overfitting and improve generalization.
• Used a validation split to monitor performance and select the best-performing model for evaluation.

Experimental results

Research questions

• RQ1 Can a deep learning model accurately predict human perceptions of warmth from facial images?
• RQ2 Can a deep learning model accurately predict human perceptions of competence from facial images?
• RQ3 Which facial regions are most predictive of perceived warmth and competence?
• RQ4 How do the visual features learned by the model align with human-identified cues for social perception?
• RQ5 To what extent can explainable AI methods like Grad-CAM reveal the decision-making process of the model in social perception tasks?

Key findings

• The model achieved a prediction accuracy of approximately 90% for warmth perception from single face images.
• The model achieved a prediction accuracy of approximately 80% for competence perception from single face images.
• Grad-CAM analysis revealed that the eyes and mouth regions were most salient for predicting warmth, while the eyes and eyebrows were key for competence.
• Facial features such as eye shape, gaze direction, and mouth curvature were consistently identified as critical for social perception judgments.
• The model demonstrated robust generalization across diverse face images, indicating that deep features capture meaningful social cues.
• The results suggest that deep learning can effectively model fundamental dimensions of social perception using visual input alone.