[Paper Review] A First Step in Combining Cognitive Event Features and Natural Language Representations to Predict Emotions

Andres Campero, Bjarke Felbo+2 more|arXiv (Cornell University)|Oct 23, 2017

Topic Modeling4 references3 citations

TL;DR

This paper proposes a multitask learning framework that combines cognitive appraisal features from appraisal theory with pre-trained natural language representations (e.g., DeepMoji) to improve fine-grained emotion prediction. By jointly training on emotion, emoji, and appraisal prediction, the model achieves human-level performance (65% accuracy) and produces a representational similarity structure that better matches neural activity in mentalizing brain regions than human-labeled appraisals alone.

ABSTRACT

We explore the representational space of emotions by combining methods from different academic fields. Cognitive science has proposed appraisal theory as a view on human emotion with previous research showing how human-rated abstract event features can predict fine-grained emotions and capture the similarity space of neural patterns in mentalizing brain regions. At the same time, natural language processing (NLP) has demonstrated how transfer and multitask learning can be used to cope with scarcity of annotated data for text modeling. The contribution of this work is to show that appraisal theory can be combined with NLP for mutual benefit. First, fine-grained emotion prediction can be improved to human-level performance by using NLP representations in addition to appraisal features. Second, using the appraisal features as auxiliary targets during training can improve predictions even when only text is available as input. Third, we obtain a representation with a similarity matrix that better correlates with the neural activity across regions. Best results are achieved when the model is trained to simultaneously predict appraisals, emotions and emojis using a shared representation. While these results are preliminary, the integration of cognitive neuroscience and NLP techniques opens up an interesting direction for future research.

Motivation & Objective

• To integrate cognitive science's appraisal theory with modern NLP techniques for improved emotion representation.
• To address data scarcity in emotion prediction by leveraging transfer and multitask learning.
• To evaluate whether joint modeling of appraisals, emotions, and emojis improves representation quality and neural similarity.
• To determine if NLP representations can enhance emotion prediction when combined with cognitive features.
• To assess whether multitask training yields representations that better correlate with neural patterns in mentalizing brain regions.

Proposed method

• Fine-tuned a pre-trained DeepMoji model to extract contextual text representations from 200 emotionally charged stories.
• Combined these NLP representations with 38 manually annotated appraisal features from cognitive science.
• Trained a multitask model to simultaneously predict emotions, emojis, and reconstruct appraisal features using shared representations.
• Used a shared representation layer followed by three separate heads: one for emotion classification, one for emoji prediction, and one for appraisal reconstruction.
• Applied mean squared error loss for appraisal reconstruction and categorical cross-entropy for emotion and emoji classification.
• Evaluated model performance via linear SVM on emotion classification and used representational dissimilarity matrices (RDMs) to compare model representations with fMRI neural data.

Experimental results

Research questions

• RQ1 Can combining NLP representations with cognitive appraisal features improve fine-grained emotion prediction beyond state-of-the-art baselines?
• RQ2 Does using appraisal features as auxiliary targets during training improve emotion prediction when only text is available?
• RQ3 Does multitask learning with emotion, emoji, and appraisal prediction produce representations that better match neural patterns in mentalizing brain regions?
• RQ4 How does the representational similarity of the model’s output compare to that of human-labeled appraisal features and neural data?
• RQ5 Can transfer learning from large-scale emoji prediction improve performance on low-resource emotion classification tasks?

Key findings

• The multitask model achieved 64% accuracy on emotion prediction when combining NLP representations and appraisal features, approaching human-level performance (65%).
• Even when appraisals were only used as auxiliary targets during training, the model achieved 42% accuracy from text alone—significantly outperforming baseline models.
• The representational dissimilarity matrix (RDM) of the multitask model showed a Kendall’s tau of 0.29 with the ToM brain region activity, outperforming both DeepMoji (0.25) and appraisal features alone (0.27).
• The multitask model’s representation correlated more strongly with neural patterns in the RTPJ (0.27) and ToM network (0.29) than the human-labeled appraisal features (0.27).
• The model trained on all three tasks (emotion, emoji, appraisal) produced a representation that better captured the similarity structure of mentalizing brain regions than any single-modality approach.
• The combination of NLP representations and appraisal features led to a 7% improvement in emotion classification over the best baseline (DeepMoji: 53% vs. multitask: 64%).