[Paper Review] Deep generative-contrastive networks for facial expression recognition

Youngsung Kim, ByungIn Yoo+3 more|arXiv (Cornell University)|Mar 21, 2017

Face recognition and analysis37 references75 citations

TL;DR

The paper proposes GCNet, a single-image facial expression recognition framework that combines a generative reference image, contrastive representation, and discriminative learning, achieving state-of-the-art results on CK+, MMI, and Oulu-CASIA datasets.

ABSTRACT

As the expressive depth of an emotional face differs with individuals or expressions, recognizing an expression using a single facial image at a moment is difficult. A relative expression of a query face compared to a reference face might alleviate this difficulty. In this paper, we propose to utilize contrastive representation that embeds a distinctive expressive factor for a discriminative purpose. The contrastive representation is calculated at the embedding layer of deep networks by comparing a given (query) image with the reference image. We attempt to utilize a generative reference image that is estimated based on the given image. Consequently, we deploy deep neural networks that embed a combination of a generative model, a contrastive model, and a discriminative model with an end-to-end training manner. In our proposed networks, we attempt to disentangle a facial expressive factor in two steps including learning of a generator network and a contrastive encoder network. We conducted extensive experiments on publicly available face expression databases (CK+, MMI, Oulu-CASIA, and in-the-wild databases) that have been widely adopted in the recent literatures. The proposed method outperforms the known state-of-the art methods in terms of the recognition accuracy.

Motivation & Objective

• Motivate robust expression recognition when expressions are ambiguous in a single image by leveraging a reference (generative) image from the same identity.
• Develop a contrastive latent representation that captures distinctive expressive changes between the input and a reference image.
• Integrate generative, contrastive, and discriminative learning in an end-to-end network to improve recognition performance.
• Demonstrate that a single-image based approach with generative reference and contrastive learning can outperform some multi-image or temporal methods.

Proposed method

• Propose GCNet that jointly learns to generate a reference image from the input, extract a contrastive latent representation, and perform classification in an end-to-end framework.
• Define contrastive representation delta as a distance between normalized latent encodings of the input X and generated reference Xhat_r: delta = d(En(X), En(Xhat_r)).
• Use a generator G consisting of an encoder En and a decoder De to estimate a reference image Xhat_r from X, modeling X ≈ Xhat_r + epsilon.
• Train with multiple losses: discriminative cross-entropy for expression, generative L2 loss between Xhat_r and Xr, contrastive loss in latent space, and reconstruction losses for input and reference decoders.
• Optimize a joint objective L = LCls + lambda_G LGen + lambda_S LContr + LRecon, balancing discriminative, generative, contrastive, and reconstruction objectives.
• Adopt a two-step disentangling strategy: first generate a reference to remove expression-related variation, then learn a contrastive encoder for discriminative features.

Experimental results

Research questions

• RQ1 Does a generative reference image improve discriminative performance for expression recognition?
• RQ2 How can contrastive metric learning be integrated with a generative reference to produce a discriminative latent representation for expressions?
• RQ3 Can a single-image based approach with generation and contrastive learning outperform multi-image or temporal methods?
• RQ4 What is the impact of reconstruction losses on the quality of the contrastive representation and classification accuracy?

Key findings

• GCNet variants using contrastive and/or reconstruction losses outperform several state-of-the-art methods on CK+, including single-image baselines and some multi-image approaches.
• The S1R1 variant (contrastive + reconstruction) achieves the highest reported CK+ accuracy among the proposed models.
• Across CK+ (7 and 8 expressions), MMI, and Oulu-CASIA VIS, the GCNet family delivers competitive or superior results compared to hand-crafted, CNN-based, and some video-based methods.
• Visualizations (t-SNE) show better clustering of expression classes with GCNet features than a CNN baseline, supporting more discriminative latent representations.
• Experiments substituting the generative component with VAEs or GANs indicate compatibility of the framework with different generative backbones.