[Paper Review] Spatiotemporal Networks for Video Emotion Recognition

Lijie Fan, Yunjie Ke|arXiv (Cornell University)|Apr 3, 2017

Video Surveillance and Tracking Methods3 citations

TL;DR

This paper proposes a spatiotemporal deep learning framework combining CNN-LSTM for video feature extraction and traditional SVM for multimodal fusion, achieving strong performance on the AFEW 6.0 dataset for video emotion recognition by integrating visual and audio modalities effectively.

ABSTRACT

Our experiment adapts several popular deep learning methods as well as some traditional methods on the problem of video emotion recognition. In our experiment, we use the CNN-LSTM architecture for visual information extraction and classification and utilize traditional methods such as for audio feature classification. For multimodal fusion, we use the traditional Support Vector Machine. Our experiment yields a good result on the AFEW 6.0 Dataset.

Motivation & Objective

• To investigate the effectiveness of deep learning and traditional machine learning methods in video emotion recognition.
• To design a spatiotemporal architecture that captures both spatial facial features and temporal dynamics in video sequences.
• To evaluate multimodal fusion using traditional methods like SVM on visual and audio features.
• To benchmark performance on the AFEW 6.0 dataset, a standard benchmark for facial emotion recognition.

Proposed method

• The visual modality is processed using a CNN-LSTM architecture to extract spatiotemporal features from video frames.
• Audio features are extracted and classified using traditional machine learning methods.
• Multimodal fusion is performed using a Support Vector Machine (SVM) to combine visual and audio predictions.
• The model is trained and evaluated on the AFEW 6.0 dataset, which contains diverse facial expressions in video form.
• The approach leverages the strengths of deep learning for feature learning and traditional methods for robust fusion.

Experimental results

Research questions

• RQ1 How well do CNN-LSTM and traditional methods perform when combined for video emotion recognition?
• RQ2 Can SVM-based fusion effectively integrate visual and audio features for improved recognition accuracy?
• RQ3 What is the performance of this hybrid approach on the AFEW 6.0 benchmark dataset?
• RQ4 How do spatiotemporal features extracted by CNN-LSTM contribute to emotion classification?

Key findings

• The proposed CNN-LSTM and SVM-based fusion model achieves strong performance on the AFEW 6.0 dataset.
• The integration of visual and audio modalities through SVM fusion improves recognition accuracy over unimodal approaches.
• The CNN-LSTM architecture effectively captures temporal dynamics in facial expressions across video sequences.
• Traditional methods like SVM remain effective for multimodal fusion when combined with deep learning features.