[Paper Review] Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs

Adrian Bulat, Georgios Tzimiropoulos|arXiv (Cornell University)|Dec 7, 2017

Face recognition and analysis4 citations

TL;DR

This paper proposes Super-FAN, the first end-to-end GAN-based framework that jointly performs facial super-resolution and landmark localization on real-world low-resolution faces in arbitrary poses. By integrating a heatmap regression sub-network with a novel heatmap loss into a residual-based super-resolution architecture, Super-FAN achieves state-of-the-art performance in both tasks, producing sharp, detailed results on real-world data including profile and occluded faces.

ABSTRACT

This paper addresses 2 challenging tasks: improving the quality of low resolution facial images and accurately locating the facial landmarks on such poor resolution images. To this end, we make the following 5 contributions: (a) we propose Super-FAN: the very first end-to-end system that addresses both tasks simultaneously, i.e. both improves face resolution and detects the facial landmarks. The novelty or Super-FAN lies in incorporating structural information in a GAN-based super-resolution algorithm via integrating a sub-network for face alignment through heatmap regression and optimizing a novel heatmap loss. (b) We illustrate the benefit of training the two networks jointly by reporting good results not only on frontal images (as in prior work) but on the whole spectrum of facial poses, and not only on synthetic low resolution images (as in prior work) but also on real-world images. (c) We improve upon the state-of-the-art in face super-resolution by proposing a new residual-based architecture. (d) Quantitatively, we show large improvement over the state-of-the-art for both face super-resolution and alignment. (e) Qualitatively, we show for the first time good results on real-world low resolution images.

Motivation & Objective

• To address the challenge of improving facial image quality and accurately localizing landmarks on very low-resolution, real-world faces across all facial poses.
• To overcome the limitations of prior methods that rely on synthetic data or frontal, pre-aligned faces.
• To integrate facial structural information into super-resolution via joint optimization of landmark detection and image reconstruction.
• To demonstrate that joint training of super-resolution and face alignment leads to superior performance on both tasks.

Proposed method

• Super-FAN employs a GAN-based super-resolution network with a residual architecture to enhance low-resolution facial images.
• A dedicated sub-network performs heatmap regression for facial landmark localization, providing structural supervision.
• A novel heatmap loss is introduced to guide the super-resolution network using predicted landmark heatmaps.
• The model is trained end-to-end using a combination of perceptual loss, pixel-wise L1 loss, and the novel heatmap loss.
• Data augmentation includes random Gaussian blur, JPEG artifacts, and color distortion to improve robustness on real-world images.
• The framework jointly optimizes super-resolution and landmark detection, enabling mutual improvement.

Experimental results

Research questions

• RQ1 Can joint learning of facial super-resolution and landmark localization improve performance on very low-resolution, real-world faces in arbitrary poses?
• RQ2 Does incorporating facial structural information via heatmap supervision enhance super-resolution quality and landmark accuracy?
• RQ3 How does the proposed method perform on real-world data compared to state-of-the-art methods trained only on synthetic or frontal images?
• RQ4 To what extent does the novel heatmap loss improve alignment and image reconstruction compared to standard losses?
• RQ5 Can the model generalize to extreme poses, occlusions, and heavy blur in unconstrained environments?

Key findings

• Super-FAN achieves a 67.0% AUC on the LS3D-W dataset for faces with poses >60°, significantly outperforming prior methods.
• On real-world WiderFace images, Super-FAN produces sharper, more detailed results than SR-GAN and CBN, especially in challenging conditions.
• The addition of the heatmap loss improves landmark localization AUC by 5.5 percentage points compared to the baseline with only pixel and feature losses.
• Even when trained on artificially downsampled images, Super-FAN outperforms retrained FAN models and bilinear baselines on real-world data after data augmentation.
• The model shows a 20.85 PSNR on the LS3D-W dataset for poses >60°, indicating strong performance despite pose variation.
• Failure cases are primarily due to extreme poses, occlusions, and heavy blur, suggesting room for improvement in robustness to such distortions.