[Paper Review] LaFIn: Generative Landmark Guided Face Inpainting

Yang Yang, Xiaojie Guo+3 more|arXiv (Cornell University)|Nov 26, 2019

Face recognition and analysisComputer Science36 references19 citations

TL;DR

LaFIn proposes a generative face inpainting method that leverages facial landmarks as structural priors to improve realism and attribute consistency. It uses a two-stream network: a landmark predictor to estimate face topology and expression, and a U-Net-based inpaintor conditioned on predicted landmarks, achieving state-of-the-art performance on CelebA-HQ and CelebA with FID scores of 4.98 under center mask and 5.31 under random mask.

ABSTRACT

It is challenging to inpaint face images in the wild, due to the large variation of appearance, such as different poses, expressions and occlusions. A good inpainting algorithm should guarantee the realism of output, including the topological structure among eyes, nose and mouth, as well as the attribute consistency on pose, gender, ethnicity, expression, etc. This paper studies an effective deep learning based strategy to deal with these issues, which comprises of a facial landmark predicting subnet and an image inpainting subnet. Concretely, given partial observation, the landmark predictor aims to provide the structural information (e.g. topological relationship and expression) of incomplete faces, while the inpaintor is to generate plausible appearance (e.g. gender and ethnicity) conditioned on the predicted landmarks. Experiments on the CelebA-HQ and CelebA datasets are conducted to reveal the efficacy of our design and, to demonstrate its superiority over state-of-the-art alternatives both qualitatively and quantitatively. In addition, we assume that high-quality completed faces together with their landmarks can be utilized as augmented data to further improve the performance of (any) landmark predictor, which is corroborated by experimental results on the 300W and WFLW datasets.

Motivation & Objective

• Address the challenge of realistic face inpainting under large occlusions, varying poses, expressions, and identities.
• Overcome limitations of prior methods that rely on edges or parsing maps, which are prone to error and redundancy.
• Improve structural fidelity and attribute consistency (e.g., gender, ethnicity, expression) in inpainted faces.
• Demonstrate that high-quality generated faces with predicted landmarks can serve as effective data augmentation for landmark detection.
• Achieve superior performance in both qualitative and quantitative evaluation on benchmark face datasets.

Proposed method

• Employ a two-stage deep learning framework: a facial landmark prediction subnet and a conditional image inpainting subnet.
• Use a U-Net-based generator with dilated residual blocks and long-short term attention to enhance feature aggregation across long-range spatial dependencies.
• Condition the inpaintor on predicted landmarks to guide the generation of semantically consistent and realistic face textures.
• Integrate instance normalization and spectral normalization in the generator and discriminator networks to stabilize training.
• Apply a multi-scale discriminator with spectral normalization and leaky ReLU to improve adversarial loss optimization.
• Use data augmentation by generating synthetic, high-quality face inpainting results with predicted landmarks to fine-tune landmark predictors on 300W and WFLW.

Experimental results

Research questions

• RQ1 Can facial landmarks serve as a more robust and effective structural prior than edges or parsing maps for face inpainting?
• RQ2 To what extent does conditioning the inpainting network on predicted landmarks improve realism and attribute consistency?
• RQ3 Can the generated, high-quality face inpainting results with accurate landmarks be used to improve the performance of existing landmark detection models?
• RQ4 How does the proposed method compare to state-of-the-art methods in terms of FID, LPIPS, and perceptual quality under diverse occlusion patterns?
• RQ5 Does the model generalize well to large occlusions and diverse facial attributes such as pose, expression, and ethnicity?

Key findings

• LaFIn achieves a FID score of 4.98 on CelebA-HQ under center mask, outperforming PIC and other SOTA methods in terms of distributional similarity.
• The model achieves a FID of 5.31 under random mask, demonstrating strong robustness to irregular occlusion patterns.
• Qualitative results show that LaFIn preserves topological structure (eyes, nose, mouth) and attribute consistency (pose, expression, ethnicity) more effectively than baseline methods.
• The method significantly reduces perceptual flaws caused by structural inconsistencies, especially in large occlusion regions.
• The proposed data-augmentation strategy improves landmark detection performance on 300W and WFLW, validating the utility of generated faces as synthetic training data.
• Ablation studies confirm that landmark conditioning is more effective than edge or parsing map supervision, as it provides a compact yet informative structural prior.