[Paper Review] Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Zebang Cheng, Zhi-Qi Cheng+7 more|arXiv (Cornell University)|Jun 17, 2024

Emotion and Mood Recognition10 citations

TL;DR

Emotion-LLaMA integrates audio, visual, and textual inputs via emotion-specific encoders and instruction tuning to achieve state-of-the-art multimodal emotion recognition and reasoning. It uses the MERR dataset for pre-training and shows strong zero-shot and fine-tuned performance across EMER, MER2023, and DFEW.

ABSTRACT

Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling. However, traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recognizing subtle facial micro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. Furthermore, we propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities. Extensive evaluations show Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on MER2023-SEMI challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on DFEW dataset.

Motivation & Objective

• Motivate accurate emotion perception in real-world, multimodal contexts beyond single-modality approaches.
• Provide a large, diverse dataset (MERR) with coarse- and fine-grained emotional annotations for training multimodal emotion models.
• Propose Emotion-LLaMA that fuses audio, visual, and textual inputs via emotion-specific encoders and instruction tuning.
• Enhance both emotion recognition accuracy and reasoning capabilities across multiple benchmarks.

Proposed method

• Construct the MERR dataset with 28,618 coarse-grained and 4,487 fine-grained samples spanning diverse emotions.
• Use HuBERT as the audio encoder and a multiview visual encoder (MAE, VideoMAE, EVA) to extract complementary visual features.
• Align audio and visual features into a shared space via trainable linear projections to language embedding tokens.
• Incorporate a modified LLaMA language model with instruction tuning to perform multimodal reasoning and generation.
• Train in a coarse-to-fine regime with pretraining on MERR followed by multimodal instruction tuning on MER2023 and DFEW.

Experimental results

Research questions

• RQ1 Can an emotion-specific multimodal encoder improve recognition and reasoning over existing MLLMs?
• RQ2 Does instruction tuning with a diverse, annotated multimodal emotion dataset enhance zero-shot and fine-tuned performance?
• RQ3 How do audio and multiview visual cues contribute to robust emotion understanding and reasoning?
• RQ4 What is the impact of pretraining on MERR versus other data for downstream emotion tasks?

Key findings

• Emotion-LLaMA achieves top Clue Overlap (7.83) and Label Overlap (6.25) on EMER.
• It attains an F1 score of 0.9036 on the MER2023 dataset.
• In zero-shot DFEW evaluations, it reaches the highest UAR (45.59) and WAR (59.37).
• On MER2023, it improves to 0.9036 F1 when using A+V+T modalities.
• Across comparisons, Emotion-LLaMA outperforms other MLLMs on multiple benchmarks.
• The model uses only 34M trainable parameters (~0.495% of total) with efficient fine-tuning.