[Paper Review] Multi-Task Learning with Auxiliary Speaker Identification for Conversational Emotion Recognition

Jingye Li, Meishan Zhang+2 more|arXiv (Cornell University)|Mar 3, 2020

Sentiment Analysis and Opinion Mining25 references22 citations

TL;DR

This paper proposes a multi-task learning framework that uses speaker identification (SI) as an auxiliary task to improve conversational emotion recognition (CER), leveraging shared representations through BERT and hierarchical Bi-GRU networks with attention and gate mechanisms. The method achieves new state-of-the-art results on EmoryNLP (+1.16% F1) and MELD (+0.59% F1) by enhancing speaker-aware contextual representations.

ABSTRACT

Conversational emotion recognition (CER) has attracted increasing interests in the natural language processing (NLP) community. Different from the vanilla emotion recognition, effective speaker-sensitive utterance representation is one major challenge for CER. In this paper, we exploit speaker identification (SI) as an auxiliary task to enhance the utterance representation in conversations. By this method, we can learn better speaker-aware contextual representations from the additional SI corpus. Experiments on two benchmark datasets demonstrate that the proposed architecture is highly effective for CER, obtaining new state-of-the-art results on two datasets.

Motivation & Objective

• To improve conversational emotion recognition (CER) by effectively modeling speaker-sensitive utterance representations.
• To address the limitation of relying solely on emotion-annotated data for learning speaker-aware representations.
• To explore the utility of large-scale, unannotated conversational corpora with speaker identities for representation learning.
• To investigate whether speaker identification (SI) as an auxiliary task can enhance CER performance through multi-task learning.
• To design a joint learning framework that enables mutual interaction between CER and SI tasks at both utterance and conversation levels.

Proposed method

• Uses a multi-task learning (MTL) framework where CER and SI are jointly trained using shared and task-specific components.
• Employs BERT as the base contextual encoder for utterance representations, followed by hierarchical bidirectional GRUs at utterance and conversation levels.
• Introduces an attention network and a gate mechanism to enable full interaction between CER and SI at both utterance and conversation levels.
• For SI, constructs binary classification pairs from raw conversations to predict whether two utterances come from the same speaker.
• Leverages large-scale, unannotated conversational data for SI pre-training to enrich speaker-aware representations.
• Unifies the two tasks via shared encoder and task-specific heads, with cross-attention and gating mechanisms to facilitate feature exchange.

Experimental results

Research questions

• RQ1 Can speaker identification (SI) as an auxiliary task improve the performance of conversational emotion recognition (CER) models?
• RQ2 How effective is multi-task learning with SI in enhancing speaker-aware contextual representations in conversations?
• RQ3 Does the mutual interaction between CER and SI through attention and gating mechanisms lead to better emotion recognition?
• RQ4 How does the performance gain from MTL vary across different pre-trained word representations (e.g., GloVe, ELMo, BERT)?
• RQ5 To what extent do the learned attention patterns reflect meaningful speaker-related and contextually relevant utterances?

Key findings

• The proposed MTL framework with SI as an auxiliary task achieves new state-of-the-art performance on both EmoryNLP and MELD datasets.
• On EmoryNLP, the BERT-based model with MTL achieves 35.92% F1, a 1.16% improvement over the baseline (34.76%).
• On MELD, the MTL-enhanced BERT model reaches 61.90% F1, a 0.59% gain over the baseline (61.31%).
• The improvements from MTL are statistically significant (p < 0.0001) across all settings, including with GloVe, ELMo, and BERT.
• The model learns to attend to speaker-related words (e.g., 'Phoebe', 'Rachel') and connect utterances from the same speaker or target speakers, indicating effective speaker-aware representation learning.
• The performance gain diminishes as the baseline becomes stronger (e.g., BERT vs. GloVe), suggesting that MTL is most beneficial when the base model is already strong.