[Paper Review] FEAFA: A Well-Annotated Dataset for Facial Expression Analysis and 3D Facial Animation

Yanfu Yan, Ke Lü+3 more|arXiv (Cornell University)|Apr 2, 2019

Face recognition and analysis33 references4 citations

TL;DR

FEAFA is a novel, highly detailed facial expression dataset with 99,356 manually annotated video frames from 122 participants recorded in real-world conditions. It provides continuous, floating-point intensity annotations (0–1) for 9 symmetrical and 10 unilateral FACS action units, plus 4 descriptors, enabling precise AU value regression via deep CNNs and enabling 3D facial animation without 3D reconstruction by driving blendshape models from 2D video predictions.

ABSTRACT

Facial expression analysis based on machine learning requires large number of well-annotated data to reflect different changes in facial motion. Publicly available datasets truly help to accelerate research in this area by providing a benchmark resource, but all of these datasets, to the best of our knowledge, are limited to rough annotations for action units, including only their absence, presence, or a five-level intensity according to the Facial Action Coding System. To meet the need for videos labeled in great detail, we present a well-annotated dataset named FEAFA for Facial Expression Analysis and 3D Facial Animation. One hundred and twenty-two participants, including children, young adults and elderly people, were recorded in real-world conditions. In addition, 99,356 frames were manually labeled using Expression Quantitative Tool developed by us to quantify 9 symmetrical FACS action units, 10 asymmetrical (unilateral) FACS action units, 2 symmetrical FACS action descriptors and 2 asymmetrical FACS action descriptors, and each action unit or action descriptor is well-annotated with a floating point number between 0 and 1. To provide a baseline for use in future research, a benchmark for the regression of action unit values based on Convolutional Neural Networks are presented. We also demonstrate the potential of our FEAFA dataset for 3D facial animation. Almost all state-of-the-art algorithms for facial animation are achieved based on 3D face reconstruction. We hence propose a novel method that drives virtual characters only based on action unit value regression of the 2D video frames of source actors.

Motivation & Objective

• To address the lack of fine-grained, continuous facial expression annotations in existing datasets, which are typically limited to binary or five-level intensity scales.
• To enable accurate regression of continuous facial action unit (AU) intensities for improved facial expression analysis and animation.
• To provide a benchmark for joint AU value regression using deep learning on real-world, diverse demographic data.
• To demonstrate a novel 3D facial animation pipeline that bypasses 3D shape reconstruction by regressing AU values directly from 2D video frames.
• To support future research in facial expression recognition, image manipulation, and animation through a publicly available, well-annotated dataset.

Proposed method

• The FEAFA dataset was constructed by recording 123 webcam videos of 122 participants across diverse age groups and real-world conditions.
• Each frame was manually annotated using a custom Expression Quantification Tool to assign floating-point intensities (0 to 1) to 9 symmetrical AUs, 10 unilateral AUs, 2 symmetrical ADs, and 2 unilateral ADs.
• A baseline deep learning system was developed using Convolutional Neural Networks (CNNs) to perform joint regression of all AU intensities from 2D video frames.
• The 3D facial animation pipeline uses the predicted AU values to drive a blendshape model, where the virtual character’s face is reconstructed as a linear combination of neutral and expression-specific blendshapes.
• The method leverages dynamic AU dependencies and co-occurrence patterns to improve animation realism and temporal consistency.
• The system was implemented using features from AlexNet and achieves real-time performance (>18 fps) on standard hardware without requiring 3D reconstruction.

Experimental results

Research questions

• RQ1 Can continuous, high-resolution AU intensity annotations (0–1) improve the accuracy of facial expression analysis compared to discrete five-level scales?
• RQ2 To what extent can 2D video frames alone drive realistic 3D facial animation without explicit 3D shape reconstruction?
• RQ3 How effective is joint AU value regression using deep CNNs on a diverse, real-world dataset with fine-grained annotations?
• RQ4 Can the proposed method generate more naturalistic and verisimilar facial animations by modeling AU relationships and dynamic dependencies?
• RQ5 How does the FEAFA dataset compare to existing benchmarks in supporting AU detection and intensity estimation tasks?

Key findings

• The FEAFA dataset contains 99,356 frames from 123 videos of 122 participants, with continuous AU intensity annotations across 25 facial motion components (9 symmetrical AUs, 10 unilateral AUs, 2 symmetrical ADs, 2 unilateral ADs).
• The proposed CNN-based baseline system achieves robust joint regression of AU intensities from 2D video frames, enabling accurate and continuous expression representation.
• The 3D facial animation pipeline successfully generates realistic character animations using only 2D video input and AU value regression, eliminating the need for time-consuming 3D reconstruction.
• The animation system runs at over 18 fps on standard consumer hardware, demonstrating real-time feasibility for practical applications.
• The continuous AU intensity annotations enable more precise AU detection and intensity estimation than discrete five-level scales, with AU values mapped to FACS intensity levels A–E via predefined intervals.
• The dataset supports a wide range of future applications, including facial expression recognition, image manipulation, and improved human-computer interaction.