[Paper Review] DENS: A Dataset for Multi-class Emotion Analysis

Chen Liu, Muhammad Osama+1 more|arXiv (Cornell University)|Oct 25, 2019

Sentiment Analysis and Opinion Mining19 references4 citations

TL;DR

This paper introduces DENS, a new dataset of long-form narrative passages from classic literature and modern online stories, annotated with nine emotion classes (Joy, Sadness, Anger, Fear, Anticipation, Surprise, Love, Disgust, Neutral) using Amazon Mechanical Turk. The study benchmarks multiple models, finding that fine-tuning a pre-trained BERT model achieves the highest performance with a micro-F1 score of 60.4%, demonstrating the dataset's value for advancing multi-class emotion analysis beyond sentence-level and binary emotion tasks.

ABSTRACT

We introduce a new dataset for multi-class emotion analysis from long-form narratives in English. The Dataset for Emotions of Narrative Sequences (DENS) was collected from both classic literature available on Project Gutenberg and modern online narratives available on Wattpad, annotated using Amazon Mechanical Turk. A number of statistics and baseline benchmarks are provided for the dataset. Of the tested techniques, we find that the fine-tuning of a pre-trained BERT model achieves the best results, with an average micro-F1 score of 60.4%. Our results show that the dataset provides a novel opportunity in emotion analysis that requires moving beyond existing sentence-level techniques.

Motivation & Objective

• To address the lack of large-scale, multi-class emotion datasets for long-form narratives in English.
• To provide a benchmark dataset that captures complex emotional arcs beyond binary or short-text emotion classification.
• To enable research into emotion analysis in narrative contexts where emotions are layered and context-dependent.
• To support the development of models that can understand emotional content in extended, coherent passages.

Proposed method

• The dataset was constructed from passages (40–200 tokens) selected from Project Gutenberg and Wattpad, focusing on emotionally rich content.
• Emotion annotation was based on a modified Plutchik’s wheel, replacing Trust with Love and adding Neutral, resulting in nine emotion categories.
• Crowd workers on Amazon Mechanical Turk labeled each passage with the dominant emotion, requiring majority agreement (Fleiss’s κ > 0.4) for inclusion.
• Passages with low annotator agreement were resolved by expert in-house annotators, ensuring label quality.
• A range of models were evaluated, including TF-IDF with SVM, lexicon-based methods, Doc2Vec, RNNs, self-attention, ELMo, and fine-tuned BERT.
• Fine-tuning of the uncased BERT-Large model was used as the state-of-the-art baseline, with cross-validated micro-F1 scores reported.

Experimental results

Research questions

• RQ1 Can multi-class emotion classification be effectively performed on long-form narrative passages rather than short, isolated texts?
• RQ2 How do pre-trained language models like BERT compare to traditional bag-of-words and lexicon-based methods on narrative emotion classification?
• RQ3 To what extent do attention mechanisms and contextual embeddings improve performance on emotion classification in narrative contexts?
• RQ4 What is the impact of narrative context and passage length on emotion annotation consistency and model performance?

Key findings

• Fine-tuning a pre-trained BERT model achieved the highest performance on the DENS dataset, with an average micro-F1 score of 60.4%.
• Bag-of-words and lexicon-based models (e.g., TF-IDF + SVM, NRC + SVM) performed poorly, with micro-F1 scores below 0.46.
• Neural sequence models such as BiRNN with self-attention and ELMo-based models showed moderate improvement, achieving micro-F1 scores between 0.487 and 0.516.
• The use of contextual embeddings (ELMo and BERT) significantly outperformed static word embeddings and lexicon-based approaches.
• The dataset demonstrates that narrative emotion is often complex and context-dependent, requiring models that capture long-range dependencies and semantic nuance.
• The results indicate that current models still struggle with underrepresented emotions, highlighting a need for few-shot learning and knowledge infusion in future work.