[Paper Review] FaceFormer: Scale-aware Blind Face Restoration with Transformers

Aijin Li, Gen Li+2 more|arXiv (Cornell University)|Jul 20, 2022

Face recognition and analysis4 citations

TL;DR

FaceFormer proposes a scale-aware blind face restoration framework using a novel Facial Feature Up-sampling (FFUP) module and a hierarchical Transformer-based Facial Feature Embedding (FFE) module, enabling high-fidelity, realistic, and symmetrical face restoration across arbitrary input scales. Trained on synthetic data, it generalizes better to real-world low-quality images than state-of-the-art methods.

ABSTRACT

Blind face restoration usually encounters with diverse scale face inputs, especially in the real world. However, most of the current works support specific scale faces, which limits its application ability in real-world scenarios. In this work, we propose a novel scale-aware blind face restoration framework, named FaceFormer, which formulates facial feature restoration as scale-aware transformation. The proposed Facial Feature Up-sampling (FFUP) module dynamically generates upsampling filters based on the original scale-factor priors, which facilitate our network to adapt to arbitrary face scales. Moreover, we further propose the facial feature embedding (FFE) module which leverages transformer to hierarchically extract diversity and robustness of facial latent. Thus, our FaceFormer achieves fidelity and robustness restored faces, which possess realistic and symmetrical details of facial components. Extensive experiments demonstrate that our proposed method trained with synthetic dataset generalizes better to a natural low quality images than current state-of-the-arts.

Motivation & Objective

• To address the limitation of existing face restoration methods that are restricted to specific input scales, especially in real-world scenarios with arbitrary degradation.
• To improve restoration fidelity and robustness by leveraging scale-aware feature learning and hierarchical attention mechanisms.
• To enable generalization from synthetic training data to real-world low-quality face images without relying on ground-truth degradation priors.
• To maintain identity consistency and realistic facial details such as pupils, eyebrows, and eyelashes under extreme upscaling and severe degradation.

Proposed method

• The Facial Feature Up-sampling (FFUP) module dynamically generates scale-aware upsampling filters based on input scale-factor priors, enabling arbitrary-scale face restoration without fixed resolution constraints.
• The Facial Feature Embedding (FFE) module uses a hierarchical Transformer architecture with non-overlapping local windows and cross-window connections to extract diverse and robust facial latent features.
• The framework integrates FFUP and FFE within a unified face restoration pipeline, combining scale-aware feature refinement with attention-based feature enhancement.
• The method leverages a super-resolution backbone with a fractional sampler to refine scale-aware features before feeding them into the FFE and generator modules.
• The FFE module enhances feature representation by modeling long-range dependencies and local facial component structures through multi-head self-attention in a hierarchical manner.
• The entire framework is trained end-to-end on synthetic datasets and generalizes effectively to real-world low-quality images without fine-tuning.

Experimental results

Research questions

• RQ1 Can a blind face restoration model generalize effectively across arbitrary input scales without requiring fixed resolution inputs?
• RQ2 How does a scale-aware upsampling mechanism improve restoration quality and reduce artifacts in extreme upscaling scenarios?
• RQ3 To what extent does a hierarchical Transformer-based feature embedding module enhance the fidelity and realism of restored facial components?
• RQ4 Does the proposed framework outperform state-of-the-art methods on real-world low-quality face images despite being trained on synthetic data?

Key findings

• FaceFormer achieves state-of-the-art performance on real-world datasets LFW-test and CelebChild, with a FID of 13.17 and LPIPS of 0.3237 on CelebA-Test, outperforming prior methods.
• Ablation studies show that removing the FFUP module increases LPIPS by 0.0265 and FID by 4.65, indicating significant degradation in perceptual quality and identity preservation.
• Without the FFE module, FID increases dramatically to 42.62, and LPIPS rises to 0.3646, demonstrating its critical role in maintaining facial structure and detail fidelity.
• On extreme upscaling factors (×6.0 and ×8.0), FaceFormer produces perceptually superior results compared to GFP-GAN, with clearer eyebrows and more realistic skin texture.
• The method successfully preserves consistent pupil color across faces, as seen in the 4th row of qualitative results, indicating strong identity consistency.
• Visual comparisons show that FaceFormer avoids transparent artifacts and blurring common in methods like HiFaceGAN and Wan et al., especially in eyes and mouth regions.