[Paper Review] Complex Emotion Recognition System using basic emotions via Facial Expression, EEG, and ECG Signals: a review

Javad Hassannataj Joloudari, Mohammad Maftoun+3 more|arXiv (Cornell University)|Sep 9, 2024

EEG and Brain-Computer Interfaces4 citations

TL;DR

This review proposes a Complex Emotion Recognition System (CERS) that integrates facial expressions, EEG, and ECG signals using machine learning, deep learning, and meta-learning to enhance recognition of complex emotional states. By fusing multimodal physiological and behavioral data, the system improves accuracy and reliability, with meta-learning emerging as a key enabler for generalization and performance gains in challenging, data-scarce scenarios.

ABSTRACT

The Complex Emotion Recognition System (CERS) deciphers complex emotional states by examining combinations of basic emotions expressed, their interconnections, and the dynamic variations. Through the utilization of advanced algorithms, CERS provides profound insights into emotional dynamics, facilitating a nuanced understanding and customized responses. The attainment of such a level of emotional recognition in machines necessitates the knowledge distillation and the comprehension of novel concepts akin to human cognition. The development of AI systems for discerning complex emotions poses a substantial challenge with significant implications for affective computing. Furthermore, obtaining a sizable dataset for CERS proves to be a daunting task due to the intricacies involved in capturing subtle emotions, necessitating specialized methods for data collection and processing. Incorporating physiological signals such as Electrocardiogram (ECG) and Electroencephalogram (EEG) can notably enhance CERS by furnishing valuable insights into the user's emotional state, enhancing the quality of datasets, and fortifying system dependability. A comprehensive literature review was conducted in this study to assess the efficacy of machine learning, deep learning, and meta-learning approaches in both basic and complex emotion recognition utilizing EEG, ECG signals, and facial expression datasets. The chosen research papers offer perspectives on potential applications, clinical implications, and results of CERSs, with the objective of promoting their acceptance and integration into clinical decision-making processes. This study highlights research gaps and challenges in understanding CERSs, encouraging further investigation by relevant studies and organizations. Lastly, the significance of meta-learning approaches in improving CERS performance and guiding future research endeavors is underscored.

Motivation & Objective

• To investigate the integration of facial expression, EEG, and ECG signals for improved complex emotion recognition.
• To evaluate the effectiveness of machine learning, deep learning, and meta-learning in recognizing both basic and complex emotions.
• To identify research gaps and challenges in data collection, dataset size, and system reliability for CERS.
• To explore clinical applications and implications of CERS for affective computing and decision-making.
• To advocate for meta-learning as a promising approach to enhance model generalization and performance in low-data regimes.

Proposed method

• A comprehensive literature review was conducted on studies using EEG, ECG, and facial expression datasets for emotion recognition.
• The review analyzed the application of machine learning, deep learning, and meta-learning models in both basic and complex emotion recognition tasks.
• Multimodal fusion techniques combining facial expression, EEG, and ECG signals were evaluated for their impact on system accuracy and robustness.
• The study assessed data collection methodologies, preprocessing pipelines, and feature extraction strategies across selected works.
• Performance metrics such as accuracy, F1-score, and AUC were compared across different models and datasets.
• Meta-learning approaches were examined for their ability to adapt to new emotion recognition tasks with limited labeled data.

Experimental results

Research questions

• RQ1 How effective are multimodal fusion approaches combining facial expression, EEG, and ECG in recognizing complex emotions?
• RQ2 What are the performance differences between traditional machine learning, deep learning, and meta-learning in complex emotion recognition?
• RQ3 What are the major challenges in collecting and processing high-quality datasets for complex emotion recognition?
• RQ4 How can physiological signals like ECG and EEG enhance the reliability and depth of emotion recognition systems?
• RQ5 In what ways can meta-learning improve model generalization and performance in low-data or complex emotion recognition scenarios?

Key findings

• The integration of facial expression, EEG, and ECG signals significantly enhances the accuracy and robustness of complex emotion recognition systems.
• Deep learning models outperform traditional machine learning methods in capturing complex, non-linear patterns in multimodal emotion data.
• Meta-learning approaches show strong potential for improving model generalization, especially in low-data or few-shot learning scenarios for emotion recognition.
• Large, high-quality datasets remain a critical bottleneck due to the complexity of capturing subtle emotional states across diverse populations.
• Physiological signals such as ECG and EEG provide valuable, complementary information that improves system reliability and reduces ambiguity in emotion classification.
• Clinical applications of CERS are promising but require further validation and integration into decision-making workflows to gain widespread adoption.