[Paper Review] Self-supervised learning of a facial attribute embedding from video

Olivia Wiles, A. Sophia Koepke+1 more|arXiv (Cornell University)|Aug 21, 2018

Face recognition and analysis65 references97 citations

TL;DR

FAb-Net learns a low-dimensional embedding of facial attributes by self-supervised frame transformation on video frames, enabling pose, landmarks, and expression tasks with linear classifiers, often rivaling supervised methods.

ABSTRACT

We propose a self-supervised framework for learning facial attributes by simply watching videos of a human face speaking, laughing, and moving over time. To perform this task, we introduce a network, Facial Attributes-Net (FAb-Net), that is trained to embed multiple frames from the same video face-track into a common low-dimensional space. With this approach, we make three contributions: first, we show that the network can leverage information from multiple source frames by predicting confidence/attention masks for each frame; second, we demonstrate that using a curriculum learning regime improves the learned embedding; finally, we demonstrate that the network learns a meaningful face embedding that encodes information about head pose, facial landmarks and facial expression, i.e. facial attributes, without having been supervised with any labelled data. We are comparable or superior to state-of-the-art self-supervised methods on these tasks and approach the performance of supervised methods.

Motivation & Objective

• Motivate learning facial attribute representations without labeled data.
• Leverage video of a single identity across views/expressions to build a shared embedding.
• Show that a linear layer on the embedding can predict landmarks, pose, and expression.
• Demonstrate benefits of multi-frame sources and curriculum learning to improve embeddings.
• Illustrate applications to image retrieval based on facial attributes.

Proposed method

• Train FAb-Net to embed source and target frames into 256-dimensional vectors.
• Concatenate source and target embeddings to predict a flow field mapping source to target via a decoder.
• Use L1 loss between generated and target frames to force embeddings to capture pose/expression.
• In multi-source setup, predict per-frame confidence heatmaps and weight their contributions when reconstructing the target.
• Introduce curriculum learning to progressively increase difficulty by selecting batches organized by training loss percentile.
• Evaluate embeddings by training linear layers for landmark regression, head pose regression, and expression classification on external datasets.

Experimental results

Research questions

• RQ1 Can a self-supervised embedding learned from video capture facial attributes such as pose, landmarks, and expressions without labels?
• RQ2 Do multi-source frames and curriculum learning improve the quality of the facial attribute embedding?
• RQ3 Is the learned embedding transferable to downstream tasks with simple linear probes on external datasets?
• RQ4 Can the embedding support tasks beyond training data, e.g., image retrieval by facial attributes?

Key findings

• The learned embedding encodes facial attributes including head pose, landmarks, and expression without supervision.
• Adding multiple source frames and confidence maps improves the embedding quality.
• Curriculum learning yields performance gains over standard training. The embedding approaches, and in some settings rivals supervised methods for pose and landmark tasks, and is competitive for expression recognition.
• Linear probes on external datasets achieve competitive results for landmarks, pose, and expression, despite domain shifts from VoxCeleb+ to CelebA/AffectNet/EmotioNet.
• The embedding enables image retrieval across identities using cosine similarity, retrieving images with similar pose/expression rather than identity or decoration.