[Paper Review] Emotion Recognition Through Observer's Physiological Signals

Yang Liu, Tom Gedeon+3 more|arXiv (Cornell University)|Feb 19, 2020

EEG and Brain-Computer Interfaces11 references4 citations

TL;DR

This paper introduces the PAFEW dataset, a physiological signal collection from 24 observers viewing emotion-inducing movie clips from AFEW, focusing on electrodermal activity (EDA) for emotion recognition. Using a three-layer neural network baseline, it achieves 42.08% accuracy on single-participant data, with a two-step network improving accuracy by 4.9% when incorporating valence information, demonstrating that valence is a more effective auxiliary signal than arousal for emotion classification in this dataset.

ABSTRACT

Emotion recognition based on physiological signals is a hot topic and has a wide range of applications, like safe driving, health care and creating a secure society. This paper introduces a physiological dataset PAFEW, which is obtained using movie clips from the Acted Facial Expressions in the Wild (AFEW) dataset as stimuli. To establish a baseline, we use the electrodermal activity (EDA) signals in this dataset and extract 6 features from each signal series corresponding to each movie clip to recognize 7 emotions, i.e., Anger, Disgust, Fear, Happy, Surprise, Sad and Neutral. Overall, 24 observers participated in our collection of the training set, including 19 observers who participated in only one session watching 80 videos from 7 classes and 5 observers who participated multiple times and watched all the videos. All videos were presented in an order balanced fashion. Leave-one-observer-out was employed in this classification task. We report the classification accuracy of our baseline, a three-layer network, on this initial training set while training with signals from all participants, only single participants and only multiple participants. We also investigate the recognition accuracy of grouping the dataset by arousal or valence, which achieves 68.66% and 72.72% separately. Finally, we provide a two-step network. The first step is to classify the features into high/low arousal or positive/negative valence by a network. Then the arousal/valence middle output of the first step is concatenated with feature sets as input of the second step for emotion recognition. We found that adding arousal or valence information can help to improve the classification accuracy. In addition, the information of positive/negative valence boosts the classification accuracy to a higher degree on this dataset.

Motivation & Objective

• To develop a large-scale physiological dataset for emotion recognition using real-world stimuli.
• To establish a baseline for emotion classification using EDA signals from multiple observers.
• To investigate whether arousal or valence information can improve classification accuracy in physiological-based emotion recognition.
• To evaluate the effectiveness of a two-step neural network architecture that incorporates arousal/valence as intermediate predictions.

Proposed method

• The PAFEW dataset was collected using E4 wristbands and Eye Tribe eye trackers while 24 observers viewed 80 movie clips from the AFEW dataset.
• Six features were extracted from each EDA signal series corresponding to each video clip to represent physiological responses.
• A three-layer feedforward neural network was used as the baseline model for classifying seven emotions: anger, disgust, fear, happy, surprise, sad, and neutral.
• Leave-one-observer-out cross-validation was applied to ensure robust evaluation across participants.
• A two-step network was designed: the first step classifies arousal (high/low) or valence (positive/negative), and the second step concatenates the intermediate output with original features for final emotion classification.
• The loss function combined both the arousal/valence and emotion classification tasks, enabling end-to-end training with shared hyperparameters.

Experimental results

Research questions

• RQ1 Can physiological signals, particularly EDA, reliably classify seven basic emotions when induced by movie clips?
• RQ2 Does incorporating arousal or valence as intermediate classification tasks improve the performance of a neural network in emotion recognition from EDA?
• RQ3 Is valence a more informative auxiliary signal than arousal for improving emotion classification accuracy in this dataset?
• RQ4 How does model performance vary when training on data from single vs. multiple participants?

Key findings

• The baseline three-layer network achieved a classification accuracy of 42.08% on the single-participant subset of the PAFEW dataset.
• Classifying emotions into high/low arousal achieved 68.66% accuracy, significantly outperforming random guessing (52.03%).
• Classifying emotions into positive/negative valence achieved 72.72% accuracy, which is 22.10% higher than random guessing (50.62%).
• The two-step network improved classification accuracy by 4.9% when using valence information as an auxiliary task, compared to the baseline.
• The improvement from adding valence information was higher than from adding arousal information, indicating valence is a more effective inductive bias for this dataset.
• Data from single participants yielded higher accuracy (42.08%) than from multiple participants (33.07%), suggesting increased noise from repeated sessions.