[Paper Review] GLA in MediaEval 2018 Emotional Impact of Movies Task

Jennifer J. Sun, Ting Liu+1 more|arXiv (Cornell University)|Nov 27, 2019

Generative Adversarial Networks and Image Synthesis7 references4 citations

TL;DR

This paper presents a multimodal deep learning approach for continuous prediction of valence and arousal in movies using pre-trained features from image, audio, and face modalities. Leveraging GRU-based temporal modeling with a mixture-of-experts regression head and Butterworth filtering, the method achieved top performance in three MediaEval 2018 evaluation metrics, including lowest MSE for valence and arousal and highest PCC for arousal.

ABSTRACT

The visual and audio information from movies can evoke a variety of emotions in viewers. Towards a better understanding of viewer impact, we present our methods for the MediaEval 2018 Emotional Impact of Movies Task to predict the expected valence and arousal continuously in movies. This task, using the LIRIS-ACCEDE dataset, enables researchers to compare different approaches for predicting viewer impact from movies. Our approach leverages image, audio, and face based features computed using pre-trained neural networks. These features were computed over time and modeled using a gated recurrent unit (GRU) based network followed by a mixture of experts model to compute multiclass predictions. We smoothed these predictions using a Butterworth filter for our final result. Our method enabled us to achieve top performance in three evaluation metrics in the MediaEval 2018 task.

Motivation & Objective

• To predict continuous valence and arousal levels in movies based on visual, audio, and facial features.
• To improve emotion prediction performance using transfer learning and multimodal fusion.
• To address temporal dynamics in emotional impact through recurrent sequence modeling.
• To enhance prediction smoothness and accuracy using post-processing filtering techniques.
• To achieve state-of-the-art results on the LIRIS-ACCEDE dataset in the MediaEval 2018 Emotional Impact of Movies Task.

Proposed method

• Extracted image features using pre-trained Inception network on ImageNet (Inception-Image).
• Extracted audio features using AudioSet, a VGG-inspired model pre-trained on YouTube-8M (AudioSet).
• Extracted face features using an Inception-based model trained on facial data (Inception-Face), focusing on the two largest faces per frame.
• Combined modality-specific features using late fusion and processed them through a GRU-based recurrent network to model temporal dependencies.
• Employed a mixture-of-experts model with a softmax gating network for multivariate regression of valence and arousal.
• Applied a Butterworth low-pass filter to smooth high-frequency fluctuations in model predictions.

Experimental results

Research questions

• RQ1 How do pre-trained image, audio, and face features compare to baseline features (e.g., VGG16 + openSMILE) in predicting emotional impact?
• RQ2 What is the optimal sequence length for modeling temporal dynamics in emotional responses to movies?
• RQ3 How does the choice of recurrent architecture (GRU vs. LSTM vs. TCN) affect performance on continuous emotion prediction?
• RQ4 To what extent does ensemble averaging and batch normalization improve prediction robustness and generalization?
• RQ5 Can post-processing with a Butterworth filter effectively reduce noise in regression outputs without degrading correlation?

Key findings

• The model achieved the lowest mean squared error (MSE) for valence (0.0837) and the highest Pearson correlation coefficient (PCC) for arousal (0.3513) among all submissions.
• Incorporating Inception-Face features significantly improved performance, indicating that facial cues are highly informative for emotion prediction.
• GRU-based models outperformed LSTMs and TCNs, with 60-second input sequences yielding better results than shorter sequences.
• Ensemble averaging across five runs reduced MSE by 30% compared to the best single run, demonstrating the benefit of variance reduction.
• Batch normalization during inference increased prediction variance, particularly for arousal, likely due to distribution shift between training and test sets.
• The Butterworth filter outperformed moving average filtering by preserving the flatness of the passband, resulting in smoother and more accurate predictions.