[Paper Review] Progressive Face Super-Resolution via Attention to Facial Landmark

Deokyun Kim, Minseon Kim+2 more|arXiv (Cornell University)|Aug 22, 2019

Advanced Image Processing TechniquesComputer Science25 references93 citations

TL;DR

Introduces a progressive training framework for 8x face super-resolution using facial landmark attention and a distilled FAN to improve perceptual and structural fidelity.

ABSTRACT

Face Super-Resolution (SR) is a subfield of the SR domain that specifically targets the reconstruction of face images. The main challenge of face SR is to restore essential facial features without distortion. We propose a novel face SR method that generates photo-realistic 8x super-resolved face images with fully retained facial details. To that end, we adopt a progressive training method, which allows stable training by splitting the network into successive steps, each producing output with a progressively higher resolution. We also propose a novel facial attention loss and apply it at each step to focus on restoring facial attributes in greater details by multiplying the pixel difference and heatmap values. Lastly, we propose a compressed version of the state-of-the-art face alignment network (FAN) for landmark heatmap extraction. With the proposed FAN, we can extract the heatmaps suitable for face SR and also reduce the overall training time. Experimental results verify that our method outperforms state-of-the-art methods in both qualitative and quantitative measurements, especially in perceptual quality.

Motivation & Objective

• Motivate high-fidelity face SR that preserves facial attributes without relying on external priors.
• Develop a stable progressive training scheme that grows generator and discriminator across steps.
• Introduce a facial attention loss to emphasize detail near facial landmarks.
• Distill a lightweight FAN to produce effective landmark heatmaps for SR.
• Show that the method outperforms state-of-the-art in perceptual quality and efficiency.

Proposed method

• Use a three-step progressive SR network that upscales by factors 2x, 2x, and 2x (8x total) with step-wise fusion and linear-increasing alpha weighting.
• Propose a facial attention loss that multiplies pixel-wise error by landmark heatmap values to focus restoration around landmarks.
• Distill a compact FAN via hint-based training to produce SR-oriented landmark heatmaps with comparable accuracy.
• Train with a combination of pixel, perceptual, and WGAN-GP losses, plus heatmap and attention losses in later steps.
• Employ a distilled FAN heatmap as attention weights to guide SR without requiring excessive priors.

Experimental results

Research questions

• RQ1 Can progressive training improve stability and quality in 8x face SR?
• RQ2 Does incorporating landmark-focused attention enhance preservation of facial attributes around key regions?
• RQ3 Is a distilled FAN sufficient to provide effective landmark heatmaps for SR tasks?
• RQ4 How do perceptual, adversarial, heatmap, and attention losses interact to improve SR results?

Key findings

• The method achieves photo-realistic 8x SR with retained facial details and higher perceptual quality.
• Distilled FAN provides comparable landmark heatmaps with far fewer parameters than the original FAN.
• Facial attention loss improves PSNR, SSIM, and MS-SSIM over ablations.
• Progressive training yields better facial detail preservation than non-progressive training.
• MOS tests indicate perceptual superiority over competing methods.