[Paper Review] A Low-rank Matching Attention based Cross-modal Feature Fusion Method for Conversational Emotion Recognition

Yuntao Shou, Liu, Huan+3 more|arXiv (Cornell University)|Jun 16, 2023

Emotion and Mood Recognition12 citations

TL;DR

LMAM is a low-rank cross-modal fusion module that enhances CER by efficiently fusing text, audio, and video features with significantly fewer parameters than self-attention, and it boosts performance when plugged into state-of-the-art CER models.

ABSTRACT

Conversational emotion recognition (CER) is an important research topic in human-computer interactions. {Although recent advancements in transformer-based cross-modal fusion methods have shown promise in CER tasks, they tend to overlook the crucial intra-modal and inter-modal emotional interaction or suffer from high computational complexity. To address this, we introduce a novel and lightweight cross-modal feature fusion method called Low-Rank Matching Attention Method (LMAM). LMAM effectively captures contextual emotional semantic information in conversations while mitigating the quadratic complexity issue caused by the self-attention mechanism. Specifically, by setting a matching weight and calculating inter-modal features attention scores row by row, LMAM requires only one-third of the parameters of self-attention methods. We also employ the low-rank decomposition method on the weights to further reduce the number of parameters in LMAM. As a result, LMAM offers a lightweight model while avoiding overfitting problems caused by a large number of parameters. Moreover, LMAM is able to fully exploit the intra-modal emotional contextual information within each modality and integrates complementary emotional semantic information across modalities by computing and fusing similarities of intra-modal and inter-modal features simultaneously. Experimental results verify the superiority of LMAM compared with other popular cross-modal fusion methods on the premise of being more lightweight. Also, LMAM can be embedded into any existing state-of-the-art CER methods in a plug-and-play manner, and can be applied to other multi-modal recognition tasks, e.g., session recommendation and humour detection, demonstrating its remarkable generalization ability.

Motivation & Objective

• Motivate improved cross-modal fusion for CER by addressing intra-modal and inter-modal interactions and high model complexity.
• Propose a low-rank matching attention mechanism (LMAM) to fuse text, video, and audio features efficiently.
• Show that LMAM can be embedded into existing CER models to boost accuracy with reduced over-fitting risk.

Proposed method

• Introduce LMAM that computes row-wise matching attention between modal features with a shared low-rank weight.
• Apply low-rank decomposition to the weight to reduce parameters to less than one-third of self-attention.
• Fuse intra-modal and inter-modal similarities to exploit contextual and complementary information across text, video, and audio.
• Integrate LMAM via three fusion strategies: early fusion, early fusion with residuals, and late fusion; use residual connections for stability.
• Demonstrate LMAM as a plug-and-play module across multiple backbones (e.g., TextCNN, bc-LSTM, DialogueRNN, DialogueGCN, MM-DFN, M2FNet, EmoCaps).
• Provide complexity analysis showing LMAM’s computational advantage over self-attention (lower time and parameter count).

Experimental results

Research questions

• RQ1 Does LMAM improve cross-modal fusion by leveraging intra-modal context and inter-modal complementarities in CER?
• RQ2 Can LMAM be embedded into diverse state-of-the-art CER architectures to consistently boost performance?
• RQ3 How does LMAM compare to other fusion methods (Add, Concatenate, TFN, LFM) in terms of accuracy and F1 on benchmark CER datasets?
• RQ4 What is the impact of using all three modalities versus single modalities for CER when LMAM is applied?

Key findings

• LMAM improves performance across seven backbone CER methods when integrated in plug-and-play fashion.
• LMAM achieves higher accuracy and F1 than Add, Concatenate, TFN, and LFM on IEMOCAP and MELD with the EmoCaps backbone (best results in Table 4: IEMOCAP Acc/F1 = 73.0/73.0; MELD Acc/F1 = 65.4/64.9).
• Compared to self-attention, LMAM requires substantially fewer parameters and training time (e.g., LMAM 0.62M vs. Self-attention 3.42M; LMAM 17.8s vs. 58.7s per epoch).
• LMAM’s performance gains persist when increasing parameter count to equalized levels, indicating gains are due to LMAM design rather than model size (Table 3).
• Single-modal results show text is strongest for CER, but combining all three modalities yields the best performance (IEMOCAP and MELD, Table 5).
• LMAM’s low-rank weight decomposition reduces parameters while maintaining or improving accuracy across datasets (rank study suggests rank 45 as optimal).