[Paper Review] Hallucinating very low-resolution and obscured face images

Lianping Yang, Bin Shao+3 more|arXiv (Cornell University)|Nov 12, 2018

Advanced Image Processing Techniques26 references4 citations

TL;DR

This paper proposes OFHNet, a novel deep learning framework for hallucinating high-resolution (HR) face images from very low-resolution (16×16 pixels) and severely occluded inputs with an 8× scaling factor. The method uses a two-stage approach: first, an encoder-decoder inpainting network restores the missing regions in the LR image, followed by a super-resolution upsampling network enhanced with adversarial loss, semantic structure loss (using fixed facial landmarks), and L2 loss, achieving state-of-the-art perceptual quality and structural fidelity on challenging occluded inputs.

ABSTRACT

Most of the face hallucination methods are designed for complete inputs. They will not work well if the inputs are very tiny or contaminated by large occlusion. Inspired by this fact, we propose an obscured face hallucination network(OFHNet). The OFHNet consists of four parts: an inpainting network, an upsampling network, a discriminative network, and a fixed facial landmark detection network. The inpainting network restores the low-resolution(LR) obscured face images. The following upsampling network is to upsample the output of inpainting network. In order to ensure the generated high-resolution(HR) face images more photo-realistic, we utilize the discriminative network and the facial landmark detection network to better the result of upsampling network. In addition, we present a semantic structure loss, which makes the generated HR face images more pleasing. Extensive experiments show that our framework can restore the appealing HR face images from 1/4 missing area LR face images with a challenging scaling factor of 8x.

Motivation & Objective

• To address the challenge of hallucinating high-resolution face images from extremely low-resolution (16×16 pixels) and heavily occluded inputs with an 8× scaling factor.
• To overcome the limitations of existing face hallucination methods that fail on incomplete or very small inputs.
• To improve perceptual quality and structural fidelity in generated HR face images by incorporating facial landmark priors as a semantic regularization loss.
• To develop a two-stage framework that first restores missing regions in LR images before super-resolution, reducing reconstruction difficulty.

Proposed method

• An encoder-decoder-based inpainting network is used to restore the missing regions in low-resolution, occluded face images, ensuring semantic consistency between known and restored areas.
• A separate upsampling network performs 8× super-resolution on the inpainted LR images, combining L2 loss, adversarial loss, and a novel semantic structure loss based on fixed facial landmark detection.
• The semantic structure loss is derived from a fixed pre-trained facial landmark detection network, which provides supervision to preserve facial geometry and improve visual realism.
• Adversarial training is applied to the upsampling network to enhance photo-realism by distinguishing generated HR images from real HR images.
• The framework is trained end-to-end with a multi-loss objective: L2 loss for pixel-level reconstruction, adversarial loss for realism, and semantic structure loss for facial layout fidelity.
• The overall architecture is an encoder-decoder-decoder framework: inpainting network (first decoder), upsampling network (second decoder), with shared encoder for feature extraction.

Experimental results

Research questions

• RQ1 Can a deep learning framework effectively restore high-resolution face images from 16×16-pixel inputs with 1/4 area occlusion and an 8× scaling factor?
• RQ2 How does incorporating facial landmark priors as a semantic structure loss improve the perceptual quality and structural accuracy of hallucinated face images?
• RQ3 Does a two-stage approach—first inpainting the LR image and then super-resolving it—yield better results than direct super-resolution on occluded inputs?
• RQ4 To what extent does the combination of L2, adversarial, and semantic structure losses outperform standard loss functions in generating realistic and structurally accurate HR face images?

Key findings

• OFHNet achieves a PSNR of 20.44 dB and SSIM of 0.63 on the benchmark dataset, outperforming URDGN by 0.44 dB in PSNR and 10.5% in SSIM.
• The inclusion of semantic structure loss improves visual quality significantly, as shown in qualitative comparisons where OFHNet generates more realistic and detailed facial features than L2-loss-only baselines.
• OFHNet achieves a lower NRMSE (5.33) and higher IoU (0.60) in face alignment and parsing tasks compared to URDGN (NRMSE: 13.61, IoU: 0.40), indicating superior geometric fidelity.
• The model generalizes well to random occlusion patterns (top-left, top-right, bottom-left, bottom-right), consistently producing plausible and coherent HR face reconstructions.
• As occlusion size increases from 4×4 to 8×8 pixels, PSNR decreases monotonically, confirming the method's robustness under increasing reconstruction difficulty.
• Ablation studies confirm that the semantic structure loss is critical for visual quality, with results showing marked improvement over L2 loss alone or with adversarial loss only.