[Paper Review] Being the center of attention: A Person-Context CNN framework for Personality Recognition

Dario Dotti, Mirela Popa+1 more|arXiv (Cornell University)|Oct 15, 2019

Personality Traits and Psychology4 citations

TL;DR

This paper proposes a novel Person-Context CNN framework that jointly models individual behaviors and contextual scene dynamics using spatio-temporal motion and proxemics descriptors for robust, multi-scenario personality recognition. By fusing person-level, group-level, and proxemic features in a shared embedding space, the model outperforms state-of-the-art methods on two public datasets and reveals personality-linked behavioral patterns via class activation maps.

ABSTRACT

This paper proposes a novel study on personality recognition using video data from different scenarios. Our goal is to jointly model nonverbal behavioral cues with contextual information for a robust, multi-scenario, personality recognition system. Therefore, we build a novel multi-stream Convolutional Neural Network framework (CNN), which considers multiple sources of information. From a given scenario, we extract spatio-temporal motion descriptors from every individual in the scene, spatio-temporal motion descriptors encoding social group dynamics, and proxemics descriptors to encode the interaction with the surrounding context. All the proposed descriptors are mapped to the same feature space facilitating the overall learning effort. Experiments on two public datasets demonstrate the effectiveness of jointly modeling the mutual Person-Context information, outperforming the state-of-the art-results for personality recognition in two different scenarios. Lastly, we present CNN class activation maps for each personality trait, shedding light on behavioral patterns linked with personality attributes.

Motivation & Objective

• To develop a generalizable, multi-scenario personality recognition system that captures both individual and contextual behavioral cues.
• To address the challenge of inconsistent behavioral patterns across diverse real-world scenarios in automatic personality assessment.
• To integrate nonverbal behavioral cues with proxemic (spatial interaction) and social group dynamics information for improved personality inference.
• To enable passive, non-intrusive personality recognition through video analysis, enhancing user experience in smart systems.
• To provide interpretability by linking learned CNN features to psychological personality traits via class activation maps.

Proposed method

• The framework employs a multi-stream CNN architecture with three input streams: individual motion descriptors (D_pers), social group motion descriptors (D_group), and proxemics descriptors (D_prox) encoding spatial relationships with the environment.
• Spatio-temporal motion descriptors are extracted from video clips using 3D CNNs to capture dynamic human behavior at the individual and group levels.
• Proxemics descriptors are computed based on inter-personal distances and orientations, modeling how individuals relate to their physical and social surroundings.
• All descriptors are mapped into a shared feature space through shared embedding layers, enabling joint learning of person-context interactions.
• The final classification is performed using a softmax layer on fused features, with class activation maps used to interpret the contribution of each descriptor type per personality trait.
• The model is trained end-to-end on two public datasets: Salsa and MPII-IAD, using cross-entropy loss and standard data augmentation.

Experimental results

Research questions

• RQ1 How does jointly modeling person-level behavior and contextual scene dynamics improve personality recognition performance across diverse scenarios?
• RQ2 To what extent do proxemic features—representing spatial interaction with the environment—contribute to personality recognition, even in nonsocial settings?
• RQ3 Which behavioral patterns, as revealed by class activation maps, are most predictive of specific Big-Five personality traits?
• RQ4 Do the learned representations align with psychological theories of personality, such as the tendency of extroverts to dominate social interactions?
• RQ5 Can the model generalize across different real-world settings without task-specific fine-tuning?

Key findings

• The proposed Person-Context CNN framework achieves state-of-the-art performance on both the Salsa and MPII-IAD datasets, demonstrating superior accuracy in multi-scenario personality recognition.
• Joint modeling of person and context features leads to significant performance gains, confirming the importance of contextual information in personality inference.
• Class activation maps reveal that high Extroversion is associated with strong activation in person-level features and weak activation in context features, indicating dominance in social settings.
• High Agreeableness and Conscientiousness are linked to balanced activation across both person and context streams, reflecting cooperative and engaged social behavior.
• Low Neuroticism and Openness to Experience are associated with medium to high activation in both streams, suggesting that individuals with these traits are more socially engaged and less reactive.
• The model successfully encodes proxemic features even in nonsocial scenarios, showing that spatial interaction with objects can reflect personality, supporting anthropomorphism-based interpretation.