[Paper Review] HiGRU: Hierarchical Gated Recurrent Units for Utterance-level Emotion Recognition

Wenxiang Jiao, Haiqin Yang+2 more|arXiv (Cornell University)|Apr 9, 2019

Sentiment Analysis and Opinion Mining29 references70 citations

TL;DR

HiGRU introduces a two-level hierarchical GRU model to capture word-level within-utterance features and utterance-level context for utterance-level emotion recognition in dialogues, with two variants (HiGRU-f and HiGRU-sf) that fuse features and utilize self-attention for long-range context.

ABSTRACT

In this paper, we address three challenges in utterance-level emotion recognition in dialogue systems: (1) the same word can deliver different emotions in different contexts; (2) some emotions are rarely seen in general dialogues; (3) long-range contextual information is hard to be effectively captured. We therefore propose a hierarchical Gated Recurrent Unit (HiGRU) framework with a lower-level GRU to model the word-level inputs and an upper-level GRU to capture the contexts of utterance-level embeddings. Moreover, we promote the framework to two variants, HiGRU with individual features fusion (HiGRU-f) and HiGRU with self-attention and features fusion (HiGRU-sf), so that the word/utterance-level individual inputs and the long-range contextual information can be sufficiently utilized. Experiments on three dialogue emotion datasets, IEMOCAP, Friends, and EmotionPush demonstrate that our proposed HiGRU models attain at least 8.7%, 7.5%, 6.0% improvement over the state-of-the-art methods on each dataset, respectively. Particularly, by utilizing only the textual feature in IEMOCAP, our HiGRU models gain at least 3.8% improvement over the state-of-the-art conversational memory network (CMN) with the trimodal features of text, video, and audio.

Motivation & Objective

• Motivate robust utterance-level emotion recognition in dialogues despite context variability, data imbalance, and long-range dependencies.
• Model both word/utterance-level information and inter-utterance context using a hierarchical GRU structure.
• Develop two variants, HiGRU-f and HiGRU-sf, to effectively fuse features and capture long-range context.
• Demonstrate improvement over state-of-the-art on three dialogue emotion datasets (IEMOCAP, Friends, EmotionPush).

Proposed method

• Two-level bidirectional GRUs: lower level models word sequences within an utterance to produce an utterance embedding; upper level models the sequence of utterances to produce contextual utterance embeddings.
• HiGRU-f fuses individual word/utterance embeddings with GRU hidden states to reinforce contextual representations.
• HiGRU-sf adds a self-attention layer on GRU hidden states to capture long-range global context and fuses attention outputs with embeddings and hidden states.
• Contextual utterance embeddings are fed to a fully-connected layer with softmax to predict emotions per utterance.
• Training uses a weighted categorical cross-entropy loss to address data imbalance, with class weights inversely related to class frequencies (adjusted by alpha).

Experimental results

Research questions

• RQ1 Can a hierarchical GRU effectively learn both fine-grained word-level cues and long-range utterance-level context for utterance-level emotion recognition?
• RQ2 Do feature fusion (HiGRU-f) and self-attention with feature fusion (HiGRU-sf) provide measurable gains over vanilla HiGRU and other baselines on textual dialogue data?
• RQ3 How do HiGRU variants perform under data imbalance across emotions and across datasets (IEMOCAP, Friends, EmotionPush)?

Key findings

• HiGRU variants outperform state-of-the-art methods on all three datasets.
• On IEMOCAP with textual features only, HiGRU variants achieve at least 3.8% improvement over CMN using trimodal features.
• HiGRU-f and HiGRU-sf provide additional gains over vanilla HiGRU in both WA (weighted accuracy) and UWA (unweighted accuracy).
• HiGRU models show strong balance across emotions, with notable improvements on minority emotions like anger and sadness.
• Mixing training sets does not universally improve performance; dataset characteristics affect outcomes.
• Self-attention variant HiGRU-sf achieves the best overall results among the proposed models on several settings.