[Paper Review] Unsupervised Learning of Landmarks based on Inter-Intra Subject Consistencies

Weijian Li, Haofu Liao+3 more|arXiv (Cornell University)|Apr 16, 2020

Face recognition and analysis34 references4 citations

TL;DR

This paper proposes an unsupervised facial landmark detection method that enforces both inter-subject and intra-subject consistency by using a subject mapping module and cycle-consistent image translation. By leveraging auxiliary images to guide structural transformations and enforcing landmark consistency across subjects and within subjects, the model achieves state-of-the-art performance on MAFL and AFLW datasets, with superior accuracy and stability in landmark localization.

ABSTRACT

We present a novel unsupervised learning approach to image landmark discovery by incorporating the inter-subject landmark consistencies on facial images. This is achieved via an inter-subject mapping module that transforms original subject landmarks based on an auxiliary subject-related structure. To recover from the transformed images back to the original subject, the landmark detector is forced to learn spatial locations that contain the consistent semantic meanings both for the paired intra-subject images and between the paired inter-subject images. Our proposed method is extensively evaluated on two public facial image datasets (MAFL, AFLW) with various settings. Experimental results indicate that our method can extract the consistent landmarks for both datasets and achieve better performances compared to the previous state-of-the-art methods quantitatively and qualitatively.

Motivation & Objective

• To address the high annotation cost and subjectivity of fully supervised facial landmark detection.
• To improve unsupervised landmark discovery by enforcing semantic consistency across different subjects.
• To enhance intra-subject landmark consistency through a cycle-consistent image translation framework.
• To reduce reliance on human-annotated landmarks while maintaining high localization accuracy.
• To develop a self-supervised method that generalizes well across diverse facial appearances and poses.

Proposed method

• The method uses a subject mapping module that transforms input images based on landmark heatmaps extracted from an auxiliary image of a different subject.
• It applies a two-path, cycle-consistent design: forward translation from input to transformed image, and backward translation to reconstruct the original, enforcing intra-subject consistency.
• Landmark heatmaps are generated from feature maps using a Gaussian-like function, serving as structural guidance for image transformations.
• The model is trained with a combination of reconstruction loss and perceptual loss (using VGG-16) to preserve semantic and structural details.
• The network is end-to-end trainable, with the landmark detector jointly optimized to predict consistent, semantically meaningful keypoints across subjects.
• The method leverages image reconstruction as a self-supervision signal, with structural consistency enforced through cross-subject and intra-subject transformation cycles.

Experimental results

Research questions

• RQ1 Can inter-subject landmark consistency be effectively enforced in unsupervised facial landmark detection without human-annotated labels?
• RQ2 How does incorporating a cycle-consistent translation path improve intra-subject landmark localization stability?
• RQ3 To what extent does using auxiliary images from different subjects enhance the semantic consistency of detected landmarks?
• RQ4 Does combining perceptual and reconstruction losses lead to better landmark localization than using reconstruction alone?
• RQ5 How does the model perform under low-data regimes, and does it generalize across diverse facial datasets?

Key findings

• The proposed method achieves a mean error of 3.08 on MAFL and 6.20 on AFLW with K=10 landmarks, outperforming previous state-of-the-art methods.
• On AFLW, the method achieves better performance than Jakab et al. [11] and Sanchez et al. [12], especially at K=50, indicating improved robustness to higher landmark counts.
• The model shows strong generalization with low data, achieving near-saturation performance even with only 100 training images, and maintaining low standard deviation across runs.
• Qualitative results show that predicted landmarks are more stable and semantically meaningful (e.g., eye corners, nose) compared to baselines, which often predict landmarks on hair or neck.
• Abllation studies confirm that combining perceptual and reconstruction losses yields the best performance, with VGG-16 outperforming VGG-19 for feature extraction.
• The model exhibits reduced drift in landmark locations across subjects, particularly in forehead and eye regions, though minor variance remains under occlusion or pose variation.