[Paper Review] Fine-grained Emotion and Intent Learning in Movie Dialogues

Anuradha Welivita, Yubo Xie+1 more|arXiv (Cornell University)|Dec 25, 2020

Sentiment Analysis and Opinion Mining27 references4 citations

TL;DR

This paper introduces OSED, a large-scale dataset of 1 million movie dialogues annotated with 32 fine-grained emotions and 9 empathetic response intents using a BERT-based semi-supervised classifier. The method reduces human labeling cost through a hybrid human-computation and self-labeling pipeline, achieving 65% accuracy on emotion classification and producing a high-quality dataset that reflects natural human social interaction patterns.

ABSTRACT

We propose a novel large-scale emotional dialogue dataset, consisting of 1M dialogues retrieved from the OpenSubtitles corpus and annotated with 32 emotions and 9 empathetic response intents using a BERT-based fine-grained dialogue emotion classifier. This work explains the complex pipeline used to preprocess movie subtitles and select good movie dialogues to annotate. We also describe the semi-supervised learning process followed to train a fine-grained emotion classifier to annotate these dialogues. Despite the large set of labels, our dialogue emotion classifier achieved an accuracy of $65\%$ and was used to annotate 1M emotional movie dialogues from OpenSubtitles. This scale of emotional dialogue classification has never been attempted before, both in terms of dataset size and fine-grained emotion and intent categories. Visualization techniques used to analyze the quality of the resultant dataset suggest that it conforms to the patterns of human social interaction.

Motivation & Objective

• To address the lack of large-scale, fine-grained emotional dialogue datasets with diverse response intents for training empathetic conversational agents.
• To reduce the cost and complexity of annotating fine-grained emotions and empathetic response intents in dialogue by combining human computation with semi-supervised learning.
• To create a general-purpose, scalable dataset that reflects realistic emotion-intent exchange patterns in human social conversations.
• To evaluate the quality of automated annotations by visualizing emotion-intent flow patterns against known human interaction dynamics.
• To enable the development of more controllable, interpretable, and emotionally responsive neural chatbots.

Proposed method

• Preprocessing movie subtitles from the OpenSubtitles corpus using automatic turn and dialogue segmentation to extract ~9M initial dialogues, reduced to 4M after cleaning and deduplication.
• Applying a weak labeler, EmoBERT—a BERT-based sentence-level classifier pre-trained on the EmpatheticDialogues dataset—to filter 1M emotionally salient dialogues (OSED initial).
• Using a semi-supervised learning pipeline with self-labeling and sentence similarity techniques to refine EmoBERT into EmoBERT+, improving classification performance.
• Training EmoBERT+ on the OSED initial set to produce the final 1M OSED dataset with 32 fine-grained emotions and 9 empathetic response intents per utterance.
• Validating annotation quality through visual analysis of emotion-intent flow patterns, comparing them to known human social interaction patterns from prior work.
• Releasing the OSED dataset and the trained EmoBERT+ model to support research in empathetic dialogue systems.

Experimental results

Research questions

• RQ1 Can a semi-supervised, BERT-based approach achieve high-quality, fine-grained emotion and intent annotation at scale, reducing reliance on costly human labeling?
• RQ2 Do the emotion-intent patterns in the OSED dataset reflect real-world human social interaction dynamics, particularly in empathetic response strategies?
• RQ3 How does the performance of the EmoBERT+ classifier compare to traditional models like FastText in classifying fine-grained emotions and empathetic intents?
• RQ4 To what extent does the OSED dataset support the development of more controllable and interpretable neural chatbots for social chitchat?
• RQ5 What new empathetic response intents emerge from crowd-sourced labeling that extend beyond neutral or basic emotional mirroring?

Key findings

• The EmoBERT+ classifier achieved a 65% accuracy on fine-grained emotion classification across 32 emotion categories and 9 empathetic response intents, demonstrating strong performance despite the high label complexity.
• The visual analysis of emotion-intent flow patterns in OSED shows that questioning is the most frequent follow-up intent after emotional utterances, aligning with findings from prior studies on human interaction.
• Emotional utterances are frequently followed by responses with similar emotions—e.g., joyful followed by excited or content—indicating emotional contagion, which contrasts with the neutral responses seen in curated datasets like EmpatheticDialogues.
• The OSED dataset contains 1 million dialogues with rich, diverse emotion and intent annotations, making it the largest known dataset of its kind with fine-grained labeling.
• The semi-supervised pipeline significantly reduced human labeling cost while maintaining annotation quality, as validated by pattern consistency with human social behavior.
• The study identified new response intents such as Greeting, Commanding, and Accusing, highlighting the complexity of real-world listener responses beyond simple mirroring or neutral reactions.