[Paper Review] EmoInHindi: A Multi-label Emotion and Intensity Annotated Dataset in Hindi for Emotion Recognition in Dialogues

Gopendra Vikram Singh, Priyanshu Priya+3 more|arXiv (Cornell University)|May 27, 2022

Sentiment Analysis and Opinion Mining6 citations

TL;DR

This paper introduces EmoInHindi, a large-scale multi-label emotion and intensity-annotated dataset of 1,814 Hindi dialogues (44,247 utterances), created via a Wizard-of-Oz setup for mental health and legal counseling. It proposes strong contextual baselines using C-Fourier-Transformer and C-Attention-Transformer models, achieving a new state-of-the-art F1-score of 66.24% on single-label emotion and intensity recognition, significantly advancing low-resource language emotion recognition in dialogues.

ABSTRACT

The long-standing goal of Artificial Intelligence (AI) has been to create human-like conversational systems. Such systems should have the ability to develop an emotional connection with the users, hence emotion recognition in dialogues is an important task. Emotion detection in dialogues is a challenging task because humans usually convey multiple emotions with varying degrees of intensities in a single utterance. Moreover, emotion in an utterance of a dialogue may be dependent on previous utterances making the task more complex. Emotion recognition has always been in great demand. However, most of the existing datasets for multi-label emotion and intensity detection in conversations are in English. To this end, we create a large conversational dataset in Hindi named EmoInHindi for multi-label emotion and intensity recognition in conversations containing 1,814 dialogues with a total of 44,247 utterances. We prepare our dataset in a Wizard-of-Oz manner for mental health and legal counselling of crime victims. Each utterance of the dialogue is annotated with one or more emotion categories from the 16 emotion classes including neutral, and their corresponding intensity values. We further propose strong contextual baselines that can detect emotion(s) and the corresponding intensity of an utterance given the conversational context.

Motivation & Objective

• To address the lack of large-scale, multi-label, intensity-annotated conversational datasets in low-resource Indian languages like Hindi.
• To enable context-aware emotion recognition in dialogues, where emotional states depend on prior utterances.
• To develop robust neural baselines for detecting multiple emotions and their intensities (0–3) in Hindi utterances.
• To support the development of emotionally intelligent conversational agents in Hindi, especially for sensitive domains like mental health and legal counseling.
• To establish a benchmark for multi-label emotion and intensity recognition in low-resource, low-resource language settings.

Proposed method

• The dataset was collected using a Wizard-of-Oz framework, simulating real dialogues between users and counselors in mental health and legal support contexts.
• Each utterance was annotated with one or more of 16 emotion classes (e.g., joy, sad, angry) and intensity values on a 0–3 scale.
• The proposed models use a Transformer encoder with two layers, 300-dimensional embeddings, and Adam optimization with a 0.003 initial learning rate.
• Two main models were evaluated: C-Attention-Transformer and C-Fourier-Transformer, both using negative log-likelihood loss and early stopping after 30 epochs.
• The models process the current utterance along with its conversation history (up to t−1) to predict multi-label emotion and intensity outputs.
• Hyperparameters were tuned via grid search, and batch size was set to 32 for training.

Experimental results

Research questions

• RQ1 Can a large-scale, multi-label, intensity-annotated dataset be effectively constructed for Hindi dialogues using a Wizard-of-Oz methodology?
• RQ2 How do contextual models perform in recognizing multiple emotions and their intensities in Hindi conversational text?
• RQ3 What is the performance gap between strong neural baselines and existing methods in low-resource language emotion recognition?
• RQ4 How does incorporating conversation history improve emotion and intensity prediction in Hindi dialogues?
• RQ5 What are the key failure modes in emotion recognition, particularly due to lexical ambiguity or emotional context shifts?

Key findings

• The C-Fourier-Transformer model achieved the highest F1-score of 66.24% on single-label emotion and intensity recognition, outperforming all baselines.
• The C-Fourier-Transformer model achieved a precision of 65.98%, a recall of 66.52%, and an F1-score of 66.24%, showing a 6.82-point improvement over the bc-LSTM baseline.
• The C-Fourier-Transformer model achieved a Hamming Loss of 0.055% and a Jaccard Index of 0.055% on multi-label emotion prediction, outperforming other models in both metrics.
• The model showed significant improvements over prior methods: +6.19 points in accuracy, +6.82 points in F1-score, and +6.8 points in recall compared to bc-LSTM.
• Error analysis revealed that the model sometimes misclassifies emotions due to emotionally charged words (e.g., 'dear' associated with joy) even when context suggests otherwise.
• The dataset, EmoInHindi, is the largest publicly available Hindi conversational dataset with multi-label emotion and intensity annotations, containing 1,814 dialogues and 44,247 utterances.