[Paper Review] HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition

Soumya Dutta, Sriram Ganapathy|arXiv (Cornell University)|Apr 14, 2023

Emotion and Mood Recognition11 citations

TL;DR

HCAM presents a hierarchical multi-modal framework that fuses audio (wav2vec) and text (RoBERTa) via a co-attention module, trained in three stages with supervised contrastive loss to achieve state-of-the-art results on IEMOCAP, MELD, and CMU-MOSI.

ABSTRACT

Emotion recognition in conversations is challenging due to the multi-modal nature of the emotion expression. We propose a hierarchical cross-attention model (HCAM) approach to multi-modal emotion recognition using a combination of recurrent and co-attention neural network models. The input to the model consists of two modalities, i) audio data, processed through a learnable wav2vec approach and, ii) text data represented using a bidirectional encoder representations from transformers (BERT) model. The audio and text representations are processed using a set of bi-directional recurrent neural network layers with self-attention that converts each utterance in a given conversation to a fixed dimensional embedding. In order to incorporate contextual knowledge and the information across the two modalities, the audio and text embeddings are combined using a co-attention layer that attempts to weigh the utterance level embeddings relevant to the task of emotion recognition. The neural network parameters in the audio layers, text layers as well as the multi-modal co-attention layers, are hierarchically trained for the emotion classification task. We perform experiments on three established datasets namely, IEMOCAP, MELD and CMU-MOSI, where we illustrate that the proposed model improves significantly over other benchmarks and helps achieve state-of-art results on all these datasets.

Motivation & Objective

• Address emotion recognition in conversations by leveraging multi-modal signals (audio and text).
• Introduce a hierarchical training pipeline that first learns uni-modal utterance embeddings, then incorporates context, and finally performs cross-modal fusion.
• Explore a supervised contrastive loss in addition to cross-entropy to improve hard example learning.
• Evaluate the approach on IEMOCAP, MELD, and CMU-MOSI and demonstrate improvements over baselines and prior state-of-the-art.
• Assess robustness to ASR transcripts for text modality.

Proposed method

• Process audio with a learnable wav2vec 2.0 based front-end and sum transformer layer outputs across frames, followed by a 1-D CNN and utterance-level pooling to produce audio embeddings.
• Process text with RoBERTa embeddings (last four layers) plus a bi-GRU to form utterance-level text embeddings, with full fine-tuning.
• Stage II uses a contextual-GRU with self-attention to encode inter-utterance context for each modality.
• Stage III fuses modalities via a co-attention network consisting of cross-attention and self-attention blocks to produce a joint multi-modal representation.
• Train stages with a combined loss Ltot = beta·CE + (1−beta)·Lsup-con, where Lsup-con is a supervised contrastive loss across utterances within batches.
• Inference ensembles predictions across stages by weighting stage-wise outputs for audio, text, and co-attention modules.

Figure 1 : Block diagram of the proposed model. Here, $S_{1}$ , $S_{2}$ and $S_{3}$ refer to the speech utterances in a conversation. Similarly, the $T_{1}$ , $T_{2}$ and $T_{3}$ refer to the text transcripts for the corresponding speech signals. $\hat{Y_{1}}$ , $\hat{Y_{2}}$ and $\hat{Y_{3}}$ refer

Experimental results

Research questions

• RQ1 Can a hierarchical, stage-wise training regime improve ERC by progressively incorporating utterance-level, contextual, and cross-modal information?
• RQ2 Does co-attention effectively fuse audio and text modalities for emotion recognition in conversation across diverse datasets?
• RQ3 Does incorporating supervised contrastive loss improve learning and performance on ERC tasks?
• RQ4 How robust is HCAM to ASR-based transcripts during testing?

Key findings

• Stage I (audio) achieves 64.3% (4-way) and 64.4% (4-way with stage I text alignment in table) weighted F1 on IEMOCAP; TODAY values reflect the paper's reported numbers per modality.
• Stage II (contextual audio) achieves 78.7% (IEMOCAP 4-way); 65.7% (IEMOCAP 6-way); improvements over Stage I of +14 to +19 percentage points for audio across datasets.
• Stage I (text) achieves 68.4% (IEMOCAP 4-way); 84.3% (CMU-MOSI) with RoBERTa-based embeddings after fine-tuning; Stage II text yields 81.4% (IEMOCAP 4-way); 85.4% (CMU-MOSI).
• Stage III multi-modal fusion (audio+text) reaches 85.9% (IEMOCAP 4-way); 70.5% (IEMOCAP 6-way); 65.8% (MELD 7-way); 85.8% (CMU-MOSI).
• HCAM achieves state-of-the-art multi-modal fusion results on IEMOCAP, MELD, and CMU-MOSI compared to prior works.
• ASR transcripts at test time yield 84.6% (IEMOCAP 4-way) and 65.8% (CMU-MOSI) relative to original text results, with a modest drop for some settings.

Figure 2 : Block diagram of the contextual GRU with self-attention. Here, $U_{T}$ , $U_{T\pm 1}$ and $U_{T\pm 2}$ refer to the uni-modal embeddings from stage I of the model for each utterance in the conversation.