[Paper Review] Global-Local Face Upsampling Network

Oncel Tuzel, Yuichi Taguchi+1 more|arXiv (Cornell University)|Mar 23, 2016

Advanced Image Processing Techniques39 references21 citations

TL;DR

This paper proposes a deep learning-based Global-Local Face Upsampling Network that jointly learns global facial structure and local texture details for high-quality face hallucination from very low-resolution inputs (e.g., 10×12 pixels). By combining a reconstruction loss with an adversarial loss for perceptual quality, the method achieves state-of-the-art results in both controlled and uncontrolled settings, significantly improving visual fidelity and detail recovery over prior methods.

ABSTRACT

Face hallucination, which is the task of generating a high-resolution face image from a low-resolution input image, is a well-studied problem that is useful in widespread application areas. Face hallucination is particularly challenging when the input face resolution is very low (e.g., 10 x 12 pixels) and/or the image is captured in an uncontrolled setting with large pose and illumination variations. In this paper, we revisit the algorithm introduced in [1] and present a deep interpretation of this framework that achieves state-of-the-art under such challenging scenarios. In our deep network architecture the global and local constraints that define a face can be efficiently modeled and learned end-to-end using training data. Conceptually our network design can be partitioned into two sub-networks: the first one implements the holistic face reconstruction according to global constraints, and the second one enhances face-specific details and enforces local patch statistics. We optimize the deep network using a new loss function for super-resolution that combines reconstruction error with a learned face quality measure in adversarial setting, producing improved visual results. We conduct extensive experiments in both controlled and uncontrolled setups and show that our algorithm improves the state of the art both numerically and visually.

Motivation & Objective

• To address the challenge of face hallucination in extreme low-resolution and uncontrolled conditions (e.g., large pose, illumination variations).
• To overcome limitations of prior two-step methods, such as reliance on linear eigenface models and computationally expensive patch searches.
• To develop an end-to-end trainable deep network that jointly optimizes global facial constraints and local patch statistics.
• To improve visual quality beyond PSNR/SSIM by incorporating a learned adversarial loss for perceptual realism.

Proposed method

• The network consists of two sub-networks: one for holistic face reconstruction based on global constraints (e.g., symmetry, pose), and another for enhancing local details using patch-level statistics.
• Global constraints are modeled via a deep encoder-decoder architecture that learns high-level facial structure from training data.
• Local details are enhanced through a refinement sub-network that enforces statistical consistency with high-resolution face patches.
• The model is trained using a hybrid loss combining mean-squared reconstruction error and an adversarial loss from a discriminator network that evaluates face quality.
• The adversarial loss is optimized with a weighting factor λ to balance fidelity and perceptual realism, reducing artifacts while enhancing sharpness.
• Color upsampling is performed by processing the luminance (Y) channel and fusing bicubic-upsampled chrominance (u, v) channels.

Experimental results

Research questions

• RQ1 Can a deep end-to-end network jointly model global facial structure and local texture details to improve face hallucination in low-resolution and uncontrolled settings?
• RQ2 How does combining reconstruction loss with an adversarial loss affect visual quality and perceptual realism in super-resolution?
• RQ3 To what extent does the proposed method outperform prior state-of-the-art methods in terms of quantitative metrics and visual fidelity?
• RQ4 How sensitive is the performance to the weighting of the adversarial loss, and what trade-offs exist between PSNR and perceptual quality?
• RQ5 What are the failure modes of the method under extreme pose, expression, or occlusion variations?

Key findings

• The proposed Global-Local Network (GLN) achieves 30.34 dB PSNR and 0.884 SSIM on FRGC at 8× upsampling, outperforming prior methods in both metrics.
• Adversarial fine-tuning improves visual quality significantly, producing sharper images with more facial details, though PSNR decreases slightly by 0.25 dB at 8× upsampling.
• The GLN with λ=8×10³ for 8× upsampling produces the sharpest results with enhanced facial features, though some high-frequency artifacts appear.
• The GLN-Only and LN-Only ablation variants show that both global and local modules are essential, with GLN8 achieving the best performance.
• Failure cases occur primarily under large pose, expression changes, or occlusion, where the network struggles to reconstruct accurate facial geometry.
• Color upsampling results (Figures 9–10) confirm that the method preserves perceptual quality when applied to YUV color space, with realistic skin tones and textures.