[Paper Review] Adversarial-based neural networks for affect estimations in the wild

Decky Aspandi, Adria Mallol-Ragolta+2 more|arXiv (Cornell University)|Feb 3, 2020

Emotion and Mood Recognition36 references4 citations

TL;DR

This paper proposes an adversarial neural network framework that leverages latent features from a denoising autoencoder generator to improve valence and arousal estimation in the wild. By conditioning a multimodal discriminator on these latent features and incorporating audio modality, the model achieves state-of-the-art performance on both the SEWA and ABAW challenge datasets, demonstrating that adversarial training with disentangled latent representations enhances affective computing models.

ABSTRACT

There is a growing interest in affective computing research nowadays given its crucial role in bridging humans with computers. This progress has been recently accelerated due to the emergence of bigger data. One recent advance in this field is the use of adversarial learning to improve model learning through augmented samples. However, the use of latent features, which is feasible through adversarial learning, is not largely explored, yet. This technique may also improve the performance of affective models, as analogously demonstrated in related fields, such as computer vision. To expand this analysis, in this work, we explore the use of latent features through our proposed adversarial-based networks for valence and arousal recognition in the wild. Specifically, our models operate by aggregating several modalities to our discriminator, which is further conditioned to the extracted latent features by the generator. Our experiments on the recently released SEWA dataset suggest the progressive improvements of our results. Finally, we show our competitive results on the Affective Behavior Analysis in-the-Wild (ABAW) challenge dataset

Motivation & Objective

• To explore the use of adversarial-learned latent features for improving affect estimation in real-world (in-the-wild) settings.
• To enhance multimodal affect recognition by conditioning the discriminator on disentangled latent representations from a generator.
• To investigate whether adversarial training with latent feature conditioning improves performance on continuous affect estimation tasks.
• To evaluate the proposed method on large-scale, real-world datasets such as SEWA and Aff-Wild2.
• To demonstrate the effectiveness of combining audio and visual modality with adversarial feature learning for robust affect estimation.

Proposed method

• The model uses a Star-GAN-inspired architecture with a conditional discriminator and an autoencoder-based generator (AEG) for image denoising and latent feature extraction.
• The AEG generates clean images from noisy inputs and produces robust latent features z, which are used to condition the discriminator.
• The conditional discriminator (CD) takes both visual and audio inputs and is conditioned on the latent features z to estimate real/fake authenticity and valence/arousal levels.
• Adversarial training is performed between the AEG and CD, where the AEG minimizes reconstruction loss and fools the CD, while the CD maximizes classification accuracy on real and fake samples.
• Audio modality is aggregated during training by concatenating audio embeddings with visual features in the discriminator.
• The model is trained using standard ℓ² loss for reconstruction and adversarial loss for distribution matching, with progressive training on latent space.

Experimental results

Research questions

• RQ1 Can adversarially learned latent features improve continuous affect estimation in real-world video data?
• RQ2 Does conditioning the discriminator on latent representations extracted from a denoising autoencoder enhance valence and arousal prediction performance?
• RQ3 How does the integration of audio modality affect the performance of multimodal affect estimation in adversarial frameworks?
• RQ4 To what extent does adversarial training with latent feature conditioning outperform standard deep learning baselines on in-the-wild affect recognition benchmarks?
• RQ5 Can the proposed method generalize to diverse datasets such as SEWA and Aff-Wild2, achieving competitive results?

Key findings

• The proposed AEG-CD-SZ model achieved a CCC of 0.430 on the SEWA dataset for arousal, surpassing the baseline (0.427) and the state-of-the-art method (0.392).
• On the ABAW challenge dataset, the AEG-CD-SZ model achieved a CCC of 0.26 for arousal, outperforming the baseline (0.24) and showing consistent improvement over the standard discriminator.
• The inclusion of both latent features z and audio modality (AEG-CD-SZ) led to the highest performance gains, particularly on the arousal dimension, indicating the importance of multimodal and latent-conditioned learning.
• The discriminator-only baseline (Disc) already outperformed the SEWA baseline on both valence and arousal metrics, suggesting that adversarial training alone improves performance.
• The denoising capability of the AEG was visually validated, with generated images showing significant noise reduction and improved clarity compared to raw inputs.
• Progressive improvements were observed across all model variants, with AEG-CD-SZ consistently achieving the best results across both datasets and metrics.