[Paper Review] Multimodal Local-Global Ranking Fusion for Emotion Recognition

Paul Pu Liang, Amir Zadeh+1 more|arXiv (Cornell University)|Aug 12, 2018

Emotion and Mood Recognition30 references4 citations

TL;DR

This paper proposes Multimodal Local-Global Ranking Fusion (MLRF), a novel approach to multimodal emotion recognition that combines direct prediction of absolute emotion labels with relative ranking of emotion intensity changes between video segments. By dividing the task into local ranking, global ranking via Bayesian inference, and direct-relative fusion, MLRF achieves state-of-the-art performance on the AVEC16 benchmark, improving CCC scores by up to 0.0508 on valence and 0.032 on arousal over strong baselines.

ABSTRACT

Emotion recognition is a core research area at the intersection of artificial intelligence and human communication analysis. It is a significant technical challenge since humans display their emotions through complex idiosyncratic combinations of the language, visual and acoustic modalities. In contrast to traditional multimodal fusion techniques, we approach emotion recognition from both direct person-independent and relative person-dependent perspectives. The direct person-independent perspective follows the conventional emotion recognition approach which directly infers absolute emotion labels from observed multimodal features. The relative person-dependent perspective approaches emotion recognition in a relative manner by comparing partial video segments to determine if there was an increase or decrease in emotional intensity. Our proposed model integrates these direct and relative prediction perspectives by dividing the emotion recognition task into three easier subtasks. The first subtask involves a multimodal local ranking of relative emotion intensities between two short segments of a video. The second subtask uses local rankings to infer global relative emotion ranks with a Bayesian ranking algorithm. The third subtask incorporates both direct predictions from observed multimodal behaviors and relative emotion ranks from local-global rankings for final emotion prediction. Our approach displays excellent performance on an audio-visual emotion recognition benchmark and improves over other algorithms for multimodal fusion.

Motivation & Objective

• To address the challenge of multimodal emotion recognition by integrating both person-independent (absolute) and person-dependent (relative) emotion expression patterns.
• To improve performance on audio-visual emotion recognition benchmarks by modeling emotional intensity changes through pairwise comparisons of video segments.
• To reduce reliance on complex deep architectures by leveraging relative ranking signals that are easier to learn than absolute emotion labeling.
• To demonstrate that combining direct predictions with relative ranking estimates yields superior performance compared to either approach alone.

Proposed method

• The model performs multimodal local ranking by comparing two short, randomly sampled video segments to determine if emotional intensity increased or decreased.
• Local rankings are aggregated using a Bayesian skill rating algorithm to infer global relative emotion ranks across the entire video.
• Direct emotion predictions are generated from multimodal features (audio, visual) using a base model (e.g., EF-LSTM), while relative rankings are derived from the global ranking step.
• A late-fusion mechanism combines direct predictions and relative rankings to produce the final emotion intensity estimates.
• The method uses a configurable number of local comparison pairs (k) and window size (w) for local segment extraction, with default settings of w=200 and k=500 or 1000.
• The framework is trained end-to-end and evaluated on the AVEC16 dataset using Concordance Correlation Coefficient (CCC) for arousal and valence regression.

Experimental results

Research questions

• RQ1 Can relative emotion intensity comparisons between short video segments improve multimodal emotion recognition performance?
• RQ2 Does fusing direct absolute emotion predictions with relative ranking signals lead to better performance than using either alone?
• RQ3 How does the number of local comparison pairs and window size affect the quality of global ranking and final emotion recognition?
• RQ4 Can a simple model with ranking-based supervision outperform more complex architectures like Gated-LSTM or MV-LSTM?

Key findings

• MLRF-1000 achieved a CCC of 0.5049 on arousal and 0.5432 on valence, outperforming all baselines on the AVEC16 dataset.
• The improvement over the best baseline (MV-LSTM fully connected) was 0.032 on arousal and 0.0508 on valence, demonstrating significant gains.
• Increasing the number of local comparison pairs from 500 to 1000 improved performance, indicating that more local rankings enhance global rank estimation.
• Larger window sizes (w=200) significantly outperformed smaller ones (w=10, 50, 100), showing that longer temporal contexts are essential for capturing multimodal dynamics.
• Ablation studies confirmed that fusing both direct and relative predictions was crucial—using only relative predictions yielded poor performance (CCC=0.0402 on valence), while direct-only performance was already strong but suboptimal.
• The model achieved state-of-the-art results without complex architectures, proving that relative ranking signals can be highly effective and efficient for multimodal learning.