[Paper Review] Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies

Eric Chu, Deb Roy|arXiv (Cornell University)|Dec 8, 2017

Media Influence and Health17 references8 citations

TL;DR

This paper proposes a multimodal machine learning framework that uses deep convolutional neural networks to analyze audio and visual sentiment in movies, constructing emotional arcs that predict audience engagement. It demonstrates that specific emotional arc shapes—particularly those ending with a peak ('end with a bang')—are statistically significant predictors of higher comment counts in online video content.

ABSTRACT

Stories can have tremendous power -- not only useful for entertainment, they can activate our interests and mobilize our actions. The degree to which a story resonates with its audience may be in part reflected in the emotional journey it takes the audience upon. In this paper, we use machine learning methods to construct emotional arcs in movies, calculate families of arcs, and demonstrate the ability for certain arcs to predict audience engagement. The system is applied to Hollywood films and high quality shorts found on the web. We begin by using deep convolutional neural networks for audio and visual sentiment analysis. These models are trained on both new and existing large-scale datasets, after which they can be used to compute separate audio and visual emotional arcs. We then crowdsource annotations for 30-second video clips extracted from highs and lows in the arcs in order to assess the micro-level precision of the system, with precision measured in terms of agreement in polarity between the system's predictions and annotators' ratings. These annotations are also used to combine the audio and visual predictions. Next, we look at macro-level characterizations of movies by investigating whether there exist `universal shapes' of emotional arcs. In particular, we develop a clustering approach to discover distinct classes of emotional arcs. Finally, we show on a sample corpus of short web videos that certain emotional arcs are statistically significant predictors of the number of comments a video receives. These results suggest that the emotional arcs learned by our approach successfully represent macroscopic aspects of a video story that drive audience engagement. Such machine understanding could be used to predict audience reactions to video stories, ultimately improving our ability as storytellers to communicate with each other.

Motivation & Objective

• To model emotional arcs in films using audio and visual sentiment analysis for improved understanding of narrative impact.
• To evaluate micro-level sentiment prediction precision by comparing model outputs with human-annotated emotional peaks and valleys in 30-second video clips.
• To discover universal macro-level emotional arc families through clustering of visual sentiment trajectories.
• To investigate whether specific emotional arc shapes predict audience engagement, using comment counts on online video platforms as a proxy.
• To release public datasets of audio features and sentiment-annotated video clips for future research in multimodal emotion modeling.

Proposed method

• Trained deep convolutional neural networks on large-scale audio and visual sentiment datasets to extract frame-level and audio snippet-level sentiment scores.
• Applied dynamic time warping with Keogh lower bound to compare the shape similarity of emotional arcs across different movies.
• Used k-medoids clustering with a silhouette-based elbow method to identify distinct families of emotional arcs from visual sentiment trajectories.
• Combined audio and visual sentiment predictions using a weighted model, with precision evaluated via human-annotation agreement on emotional polarity.
• Constructed movie embeddings by averaging penultimate layer activations of a biconcept classifier across 10% video segments, then reducing to a 10×1 feature vector.
• Performed regression analysis using metadata and categorical cluster assignments (emotional arc families) to predict comment counts on Vimeo short videos.

Figure 1 : Overview

Experimental results

Research questions

• RQ1 Can audio-visual sentiment analysis accurately model micro-level emotional peaks and valleys in movie clips, as validated by human annotators?
• RQ2 Do distinct, universal families of emotional arcs exist across diverse films and short videos, and if so, how can they be discovered?
• RQ3 Are certain emotional arc shapes statistically predictive of higher audience engagement, as measured by online comment counts?
• RQ4 How does the inclusion of audio and visual sentiment features affect the accuracy and interpretability of emotional arc modeling?
• RQ5 To what extent do emotional arc families correlate with genre or narrative structure in cinematic storytelling?

Key findings

• The combined audio-visual sentiment model achieved a precision of 0.894 in predicting the polarity of emotional peaks and valleys in 30-second video clips, with high agreement to human annotators.
• The k-medoids clustering approach identified five distinct emotional arc families in the short film corpus, with the 'Icarus' shape (rise-fall) and 'end-with-a-bang' arcs being statistically significant predictors of engagement.
• For k=8 clustering, the 'end-with-a-bang' arc family was a statistically significant predictor of comment count (p < 0.05), with a positive correlation to audience engagement.
• The 'Icarus' arc (rise-fall) and two other peak-end arcs (with incline or flat before peak) were also significant predictors of higher comment counts across multiple k-values.
• Movie embeddings derived from visual sentiment showed clear clustering by genre, with romance, adventure, fantasy, and animated films forming distinct groups.
• The use of audio-visual fusion was limited by data sparsity and small ground-truth coverage, resulting in less distinct clusters compared to visual-only arcs.

Figure 2 : Effect of smoothing values for arcs: no smoothing versus $w=0.1*n$