[Paper Review] Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis

Wei Liu, Jielin Qiu+2 more|arXiv (Cornell University)|Aug 13, 2019

Emotion and Mood Recognition63 references68 citations

TL;DR

The paper introduces Deep Canonical Correlation Analysis (DCCA) for coordinating representations from multimodal signals and demonstrates state-of-the-art emotion recognition on five datasets.

ABSTRACT

Multimodal signals are more powerful than unimodal data for emotion recognition since they can represent emotions more comprehensively. In this paper, we introduce deep canonical correlation analysis (DCCA) to multimodal emotion recognition. The basic idea behind DCCA is to transform each modality separately and coordinate different modalities into a hyperspace by using specified canonical correlation analysis constraints. We evaluate the performance of DCCA on five multimodal datasets: the SEED, SEED-IV, SEED-V, DEAP, and DREAMER datasets. Our experimental results demonstrate that DCCA achieves state-of-the-art recognition accuracy rates on all five datasets: 94.58% on the SEED dataset, 87.45% on the SEED-IV dataset, 84.33% and 85.62% for two binary classification tasks and 88.51% for a four-category classification task on the DEAP dataset, 83.08% on the SEED-V dataset, and 88.99%, 90.57%, and 90.67% for three binary classification tasks on the DREAMER dataset. We also compare the noise robustness of DCCA with that of existing methods when adding various amounts of noise to the SEED-V dataset. The experimental results indicate that DCCA has greater robustness. By visualizing feature distributions with t-SNE and calculating the mutual information between different modalities before and after using DCCA, we find that the features transformed by DCCA from different modalities are more homogeneous and discriminative across emotions.

Motivation & Objective

• Motivate and enable reliable emotion recognition using multimodal signals beyond unimodal data.
• Propose a coordinated representation framework (DCCA) that learns modality-specific nonlinear transformations with CCA constraints.
• Show how DCCA can fuse modalities with adjustable weights and assess robustness to noise.
• Evaluate DCCA on five benchmark datasets to demonstrate discriminative, robust emotion representations.

Proposed method

• Transform each modality through separate deep neural networks to produce O1 and O2.
• Optimize W1 and W2 by maximizing the correlation between O1 and O2 under the DCCA objective.
• Fuse transformed features via a weighted sum O = α1 O1 + α2 O2 for classification.
• Train an SVM classifier on the fused DCCA features.
• Use regularized covariance estimates and backpropagation to compute gradients for DCCA training.
• Optionally analyze transformed features using MINE to estimate mutual information between modalities.

Experimental results

Research questions

• RQ1 Can DCCA learn coordinated and discriminative nonlinear representations for multimodal emotion data?
• RQ2 How does DCCA perform compared to other fusion strategies on benchmark datasets?
• RQ3 Is DCCA robust to noise affecting one or more modalities?
• RQ4 Can varying modality weights improve fusion performance for specific tasks or datasets?

Key findings

• DCCA achieves state-of-the-art recognition accuracy on SEED (94.58%), SEED-IV (87.45%), DEAP (88.51% for four-category task), SEED-V (83.08%), and DREAMER (88.99%, 90.57%, 90.67% for three binary tasks).
• DCCA shows greater robustness to noise than existing methods on SEED-V under varying noise levels.
• Transformed features via DCCA are more homogeneous and discriminative across emotions as shown by t-SNE visualizations and mutual information analyses.
• DCCA allows flexible fusion by adjusting modality weights, enabling different contributions from each modality to the fused features.