[Paper Review] A Personalized Affective Memory Neural Model for Improving Emotion Recognition

Pablo Barros, German I. Parisi+1 more|arXiv (Cornell University)|Apr 23, 2019

Emotion and Mood Recognition35 references4 citations

TL;DR

This paper proposes a personalized affective memory model that combines a conditional adversarial autoencoder for prior knowledge learning with Grow-When-Required networks to build individualized emotion representations. The model achieves state-of-the-art performance on in-the-wild datasets by enabling online, adaptive emotion recognition through generative editing of arousal and valence, while maintaining reliable facial feature encoding and dynamic prototype updating.

ABSTRACT

Recent models of emotion recognition strongly rely on supervised deep learning solutions for the distinction of general emotion expressions. However, they are not reliable when recognizing online and personalized facial expressions, e.g., for person-specific affective understanding. In this paper, we present a neural model based on a conditional adversarial autoencoder to learn how to represent and edit general emotion expressions. We then propose Grow-When-Required networks as personalized affective memories to learn individualized aspects of emotion expressions. Our model achieves state-of-the-art performance on emotion recognition when evaluated on extit{in-the-wild} datasets. Furthermore, our experiments include ablation studies and neural visualizations in order to explain the behavior of our model.

Motivation & Objective

• To address the limitations of existing emotion recognition models in handling personalized and online facial expressions.
• To improve adaptability in real-world scenarios by learning individualized emotional representations without requiring full retraining.
• To enable continuous learning of affective prototypes through online updating of neural representations.
• To reduce reliance on strongly supervised data by leveraging unsupervised and adversarial learning for facial expression prior knowledge.
• To enhance model interpretability through neural visualizations and ablation studies.

Proposed method

• A conditional adversarial autoencoder is used to learn a prior knowledge (PK) space that generates facial expressions with specified arousal and valence values.
• The PK model is trained with three discriminators: one for real face images, one for emotion-specific representations, and one for photo-realistic quality, ensuring diverse and realistic facial edits.
• Grow-When-Required (GWR) networks are employed as personalized affective memories (P-AffMem) to dynamically form and update prototype neurons based on individual facial expression patterns.
• The P-AffMem initializes with pre-edited facial representations from the PK, allowing immediate recognition capability at the start of interaction.
• Encoder activations are visualized to analyze how each discriminator influences feature learning and representation quality.
• The model integrates both general prior knowledge and individual-specific learning, enabling online adaptation without retraining.

Experimental results

Research questions

• RQ1 How can a neural model learn general emotional representations in an unsupervised manner to support personalized emotion recognition?
• RQ2 To what extent can a generative adversarial autoencoder improve the diversity and realism of facial expression editing for affective modeling?
• RQ3 Can a GWR-based affective memory dynamically adapt to new facial expressions in real-time without retraining?
• RQ4 How does the integration of prior knowledge and online prototype learning improve recognition performance on in-the-wild datasets?
• RQ5 What role do different discriminators play in shaping the encoder’s focus on emotion-relevant facial regions?

Key findings

• The proposed model achieves state-of-the-art performance on in-the-wild emotion recognition datasets, outperforming existing models in both arousal and valence correlation (CCC) metrics.
• The combination of all three discriminators (realism, emotion, photo-quality) yielded the best objective performance, producing clear facial features without distortions and focusing encoder attention on key emotion-relevant regions like eyes and mouth.
• Neural visualizations confirmed that the encoder trained with the emotion discriminator (D_em) effectively filters facial expression-rich regions, leading to higher CCC than models without it.
• The affective memory (P-AffMem) outperformed the prior knowledge (PK) model after a few seconds of video processing, demonstrating effective online adaptation and improved accuracy over time.
• Ablation studies showed that the PK model trained only with base loss (PK_base) failed to learn meaningful representations, as evidenced by poor image edits and low CCC scores.
• The integration of generative prior knowledge with online prototype learning enabled reliable, continuous emotion recognition from the start of interaction, even before full expression coverage.