[Paper Review] ACE-Net: Fine-Level Face Alignment through Anchors and Contours Estimation

Jihua Huang, Amir Tamrakar|arXiv (Cornell University)|Dec 2, 2020

Face recognition and analysis48 references4 citations

TL;DR

ACE-Net is a deep learning framework that predicts fine-grained facial anchors and contours instead of traditional sparse landmarks, using a weakly supervised loss to leverage existing landmark datasets and synthetic data to bridge the annotation gap. It achieves superior contour accuracy over landmark-interpolated methods and outperforms fully supervised models on the HELEN dataset, even when trained with only 36 or 68 landmarks.

ABSTRACT

We propose a novel facial Anchors and Contours Estimation framework, ACE-Net, for fine-level face alignment tasks. ACE-Net predicts facial anchors and contours that are richer than traditional facial landmarks while overcoming ambiguities and inconsistencies in their definitions. We introduce a weakly supervised loss enabling ACE-Net to learn from existing facial landmarks datasets without the need for reannotation. Instead, synthetic data, from which GT contours can be easily obtained, is used during training to bridge the density gap between landmarks and true facial contours. We evaluate the face alignment accuracy of ACE-Net with respect to the HELEN dataset which has 194 annotated facial landmarks, while it is trained with only 68 or 36 landmarks from the 300-W dataset. We show that ACE-Net generated contours are better than contours interpolated straight from the 68 GT landmarks and ACE-Net also outperforms models trained only with full supervision from GT landmarks-based contours.

Motivation & Objective

• To address the limitations of sparse, inconsistent, and ambiguous facial landmark representations in fine-level face alignment.
• To eliminate the need for costly manual annotation of facial contours by leveraging weak supervision from existing landmark datasets and synthetic data.
• To develop a unified representation—Anchors and Contours (AC)—that provides denser, more accurate facial structure constraints for downstream tasks like mesh fitting.
• To enable high-accuracy contour estimation even in regions with sparse training annotations, improving robustness in low-annotation-density areas.
• To establish a new evaluation benchmark for AC representations using densely annotated datasets like HELEN.

Proposed method

• ACE-Net employs a two-module architecture: an AC prediction module that regresses heatmaps for anchors and contours, and an AC extraction module that refines these heatmaps to sub-pixel accuracy.
• It introduces a novel 'contourness' loss that enforces predicted contours to pass through existing facial landmarks, enabling weak supervision without re-annotation.
• The model uses synthetic data with ground-truth contours to provide full supervision, bridging the density gap between sparse landmarks and dense contours.
• The contourness loss is derived from steerable filters and enables efficient computation of contour confidence via convolutions, with closed-form solutions for contour orientation and normal maps.
• Intermediate supervisory signals are applied to low-resolution feature maps using line-segment contours as pseudo-ground truth, improving training stability.
• The framework is architecture-agnostic and can be integrated with various backbone networks, including those with intermediate supervision.

Experimental results

Research questions

• RQ1 Can a deep learning model predict dense facial contours more accurately than interpolation from sparse landmarks?
• RQ2 Can weak supervision from existing landmark datasets effectively train a model to predict accurate contours without re-annotation?
• RQ3 Does incorporating synthetic data with ground-truth contours improve contour localization in regions between landmarks, despite domain gap?
• RQ4 Can the proposed AC representation outperform standard landmark-based alignment in fine-level face alignment tasks?
• RQ5 Is the contourness loss effective in aligning predicted contours with known landmark positions across diverse datasets?

Key findings

• ACE-Net outperforms contour interpolation from 68 landmarks on the HELEN dataset, demonstrating that learned contours are more accurate than linear or spline-based interpolation.
• Even when trained with only 36 landmarks, ACE-Net achieves competitive performance, showing that weak supervision from sparse landmarks is sufficient for high-quality contour prediction.
• The model trained with synthetic data and weak supervision surpasses models trained solely with full supervision from landmark-derived contours, proving the benefit of synthetic data in bridging the annotation density gap.
• ACE-Net generates sub-pixel accurate contours that better align with true facial geometry, as validated by comparison to the densely annotated HELEN dataset.
• The contourness loss enables effective weak supervision, allowing the model to generalize across different landmark definitions and datasets without re-annotation.
• Qualitative results show that ACE-Net produces smoother, more anatomically plausible contours, especially in challenging regions like the nose and mouth.