[Paper Review] Learning Oracle Attention for High-fidelity Face Completion

Tong Zhou, Changxing Ding+3 more|arXiv (Cornell University)|Mar 31, 2020

Face recognition and analysis38 references4 citations

TL;DR

This paper proposes a novel face completion framework using a Dual Spatial Attention (DSA) module with oracle supervision and multi-discriminator training to enhance high-fidelity image generation. By enforcing accurate attention scores and focusing adversarial learning on key facial components, the method achieves state-of-the-art performance on CelebA-HQ and Flickr-Faces-HQ, significantly reducing artifacts and improving texture realism and symmetry.

ABSTRACT

High-fidelity face completion is a challenging task due to the rich and subtle facial textures involved. What makes it more complicated is the correlations between different facial components, for example, the symmetry in texture and structure between both eyes. While recent works adopted the attention mechanism to learn the contextual relations among elements of the face, they have largely overlooked the disastrous impacts of inaccurate attention scores; in addition, they fail to pay sufficient attention to key facial components, the completion results of which largely determine the authenticity of a face image. Accordingly, in this paper, we design a comprehensive framework for face completion based on the U-Net structure. Specifically, we propose a dual spatial attention module to efficiently learn the correlations between facial textures at multiple scales; moreover, we provide an oracle supervision signal to the attention module to ensure that the obtained attention scores are reasonable. Furthermore, we take the location of the facial components as prior knowledge and impose a multi-discriminator on these regions, with which the fidelity of facial components is significantly promoted. Extensive experiments on two high-resolution face datasets including CelebA-HQ and Flickr-Faces-HQ demonstrate that the proposed approach outperforms state-of-the-art methods by large margins.

Motivation & Objective

• Address the challenge of high-fidelity face completion due to complex facial textures and structural correlations.
• Overcome inaccurate attention scores in existing attention-based methods that lead to artifacts and semantic inconsistencies.
• Improve fidelity of key facial components—eyes, nose, mouth—by leveraging landmark priors and targeted discriminators.
• Ensure reliable attention mechanisms through oracle supervision to guide feature selection during inpainting.
• Achieve superior visual quality and quantitative performance compared to state-of-the-art methods on high-resolution datasets.

Proposed method

• Propose a Dual Spatial Attention (DSA) module that combines foreground self-attention and foreground-background cross-attention to model pixel-level correlations across multiple scales.
• Introduce an oracle supervision signal to constrain attention scores, ensuring they are semantically meaningful and reduce hallucination in generated textures.
• Use facial landmarks from ground truth images as prior knowledge to identify and localize four key facial components for targeted refinement.
• Deploy four separate discriminators on each of the four facial components to enforce high-fidelity texture generation through adversarial training.
• Integrate the DSA module and multi-discriminator into a U-Net-based generator, with discriminators removed during inference to preserve efficiency.
• Train the model end-to-end using perceptual and adversarial losses, with supervision applied specifically to attention maps to improve reliability.

Experimental results

Research questions

• RQ1 Can oracle-supervised attention learning improve the reliability of feature selection in face completion, reducing artifacts caused by inaccurate attention scores?
• RQ2 Does focusing adversarial training on key facial components—eyes, nose, mouth—lead to higher texture fidelity compared to global or local discriminators alone?
• RQ3 To what extent does dual spatial attention with pixel-level correlation modeling outperform patch-based attention mechanisms in preserving facial symmetry and detail?
• RQ4 How does the integration of landmark-based priors enhance the spatial alignment and realism of generated facial components?
• RQ5 Can a multi-discriminator strategy improve high-frequency texture details without degrading inference speed?

Key findings

• The proposed method achieves state-of-the-art performance on CelebA-HQ and Flickr-Faces-HQ, outperforming prior methods in both qualitative and quantitative evaluations.
• The DSA module with oracle supervision reduces artifacts such as color discrepancies between eyes and texture distortions in the nose and mouth regions.
• Quantitative results show a PSNR improvement from 26.23 to 26.36 and SSIM from 0.9062 to 0.9107 on CelebA-HQ when using supervised attention, with lower LPIPS (0.0706 vs. 0.0726), indicating better perceptual quality.
• The multi-discriminator setup significantly enhances texture sharpness and realism, especially in facial components, as shown in visual comparisons and zoom-in analysis.
• The method maintains inference efficiency by removing discriminators at test time, ensuring no performance overhead.
• Abllation studies confirm that both the oracle supervision on attention and the component-specific discriminators are essential for achieving high-fidelity results.