[Paper Review] Emotion Representation Mapping for Automatic Lexicon Construction (Mostly) Performs on Human Level

Sven Buechel, Udo Hahn|arXiv (Cornell University)|Jun 22, 2018

Sentiment Analysis and Opinion Mining49 references20 citations

TL;DR

This paper proposes a deep neural network model for Emotion Representation Mapping (ERM) that converts emotion ratings between different formats—such as Valence-Arousal-Dominance (VAD) and Basic Emotions—outperforming prior state-of-the-art methods. The key contribution is demonstrating that the model's predictions are nearly as reliable as human annotations, even across languages, enabling the creation of high-quality, automatically constructed emotion lexicons for 13 diverse languages.

ABSTRACT

Emotion Representation Mapping (ERM) has the goal to convert existing emotion ratings from one representation format into another one, e.g., mapping Valence-Arousal-Dominance annotations for words or sentences into Ekman's Basic Emotions and vice versa. ERM can thus not only be considered as an alternative to Word Emotion Induction (WEI) techniques for automatic emotion lexicon construction but may also help mitigate problems that come from the proliferation of emotion representation formats in recent years. We propose a new neural network approach to ERM that not only outperforms the previous state-of-the-art. Equally important, we present a refined evaluation methodology and gather strong evidence that our model yields results which are (almost) as reliable as human annotations, even in cross-lingual settings. Based on these results we generate new emotion ratings for 13 typologically diverse languages and claim that they have near-gold quality, at least.

Motivation & Objective

• To develop a neural network-based approach for Emotion Representation Mapping (ERM) that improves upon existing Word Emotion Induction (WEI) techniques for automatic emotion lexicon construction.
• To address the growing fragmentation in emotion representation formats by enabling cross-format and cross-lingual compatibility of emotion resources.
• To establish a rigorous evaluation framework that compares ERM performance directly against human annotation reliability using normalized split-half reliability scores.
• To generate high-quality, automatically constructed emotion lexicons for 13 typologically diverse languages using the proposed ERM model.
• To publicly release the trained models, code, and newly constructed lexicons to support reproducibility and community use.

Proposed method

• Proposes a feedforward neural network (FFNN) model to map emotion ratings between Valence-Arousal-Dominance (VAD) and Basic Emotions (BE5) formats, using word-level embeddings and multi-head attention mechanisms.
• Trains the model on multilingual data from existing psychological lexicons, including 10 datasets across 7 languages (English, Spanish, German, Polish, Dutch, Italian, Portuguese), with additional data from Greek, French, Swedish, Finnish, and Chinese.
• Employs a refined evaluation methodology based on normalized split-half reliability to compare model predictions against human annotation reliability, enabling direct benchmarking of model performance.
• Applies the model in both monolingual and cross-lingual settings: in monolingual mode, it uses language-specific training data; in cross-lingual mode, it concatenates data from all available languages (excluding Dominance for compatibility).
• Uses a consistent preprocessing pipeline including word normalization, embedding alignment, and loss minimization via mean squared error for VAD and cross-entropy for BE5.
• Validates results using 10-fold cross-validation on monolingual data and direct evaluation on cross-lingual data without cross-validation to ensure real-world applicability.

Experimental results

Research questions

• RQ1 Can a neural network-based ERM model outperform existing state-of-the-art WEI and ERM methods in mapping emotion ratings between VAD and BE5 formats?
• RQ2 To what extent does the proposed ERM model match the reliability of human annotations, especially in cross-lingual settings?
• RQ3 Can the model generate emotion lexicons of near-gold quality for multiple typologically diverse languages?
• RQ4 Does the proposed evaluation methodology based on normalized split-half reliability provide a valid and scalable benchmark for comparing ERM performance against human performance?
• RQ5 How does model performance vary across different languages and emotion dimensions (e.g., Valence, Arousal, Joy, Anger) in cross-lingual versus monolingual setups?

Key findings

• The proposed FFNN model outperforms previous state-of-the-art methods in both monolingual and cross-lingual ERM tasks, with performance improvements across all evaluated emotion dimensions.
• In monolingual settings, the model achieves performance competitive with human raters, with 8 out of 10 cases exceeding human reliability for Valence and Arousal, and 11 out of 25 cases surpassing human performance for BE5.
• In cross-lingual settings, the model maintains strong performance, with only minor drops (less than 1%-point on average) in Valence and Joy, and moderate decreases (up to 5%-points) in other dimensions, indicating robust generalization.
• The model’s predictions are deemed to have near-gold quality, especially in cross-lingual applications, based on the normalized split-half reliability comparison with human annotations.
• The authors successfully constructed emotion lexicons for 13 languages, with the largest containing 12,884 words (English, BE5 format), and all are publicly available via GitHub.
• The study demonstrates that ERM can serve as a viable alternative to WEI for automatic lexicon construction, with the added benefit of enabling interoperability across incompatible emotion representation formats.