[Paper Review] Extreme 3D Face Reconstruction: Seeing Through Occlusions

Anh Tran, Tal Hassner+4 more|arXiv (Cornell University)|Dec 14, 2017

Face recognition and analysis52 references4 citations

TL;DR

This paper proposes a novel deep learning framework for extreme 3D face reconstruction that decouples global face shape estimation from mid-level facial details using a bump map representation. By estimating a robust foundation shape and separately predicting detailed surface variations via a convolutional encoder-decoder, the method achieves high-fidelity 3D reconstructions even under severe occlusions and out-of-plane rotations, outperforming state-of-the-art methods in both quantitative accuracy and qualitative robustness on challenging, unconstrained inputs.

ABSTRACT

Existing single view, 3D face reconstruction methods can produce beautifully detailed 3D results, but typically only for near frontal, unobstructed viewpoints. We describe a system designed to provide detailed 3D reconstructions of faces viewed under extreme conditions, out of plane rotations, and occlusions. Motivated by the concept of bump mapping, we propose a layered approach which decouples estimation of a global shape from its mid-level details (e.g., wrinkles). We estimate a coarse 3D face shape which acts as a foundation and then separately layer this foundation with details represented by a bump map. We show how a deep convolutional encoder-decoder can be used to estimate such bump maps. We further show how this approach naturally extends to generate plausible details for occluded facial regions. We test our approach and its components extensively, quantitatively demonstrating the invariance of our estimated facial details. We further provide numerous qualitative examples showing that our method produces detailed 3D face shapes in viewing conditions where existing state of the art often break down.

Motivation & Objective

• To address the limitation of existing 3D face reconstruction methods that fail under extreme conditions such as severe occlusions and large head rotations.
• To enable detailed 3D face reconstruction while maintaining robustness to challenging viewing conditions.
• To decouple global shape estimation from mid-level facial detail prediction, improving generalization and handling of occluded regions.
• To develop a method that generates plausible, identity-preserving facial details in occluded regions using example-based hole filling and 3D soft symmetry.
• To demonstrate that bump map representations are invariant to occlusions and can preserve identity information even when parts of the face are hidden.

Proposed method

• The method uses a deep convolutional encoder-decoder network to estimate facial bump maps, which encode mid-level surface details such as wrinkles and pores.
• A coarse 3D face shape is first estimated as a foundation, which is then refined by layering the predicted bump map to add fine details.
• For occluded regions, an example-based hole-filling technique is applied using reference 3D faces to synthesize plausible facial details.
• 3D soft symmetry is used to infer self-occluded facial features by leveraging symmetric structure across the face.
• The bump map estimation is trained end-to-end using a loss function that encourages identity preservation and geometric consistency.
• The system is trained and evaluated on the CASIA 3D face dataset, with transfer learning from ImageNet-pretrained models.

Experimental results

Research questions

• RQ1 Can a deep learning-based bump map estimation method produce detailed 3D face reconstructions under extreme viewing conditions, including occlusions and large rotations?
• RQ2 To what extent do bump maps preserve identity information when parts of the face are occluded?
• RQ3 Can example-based hole filling generate plausible and identity-consistent facial details in occluded regions?
• RQ4 How does the modular separation of global shape and local details improve robustness compared to end-to-end 3D reconstruction networks?
• RQ5 Does the use of 3D soft symmetry enhance the recovery of self-occluded facial features?

Key findings

• The proposed bump map estimation method achieves high recognition accuracy on LFW images, with a 92.1% top-1 accuracy on the CASIA dataset when using the AlexNet-based verification system.
• When 75×75 pixel occlusions were introduced, recognition accuracy on bump maps dropped from 92.1% to 68.3%, demonstrating the sensitivity of raw bump maps to occlusions.
• The example-based hole-filling method recovered 89.4% of the original recognition accuracy, indicating that the generated details are highly plausible and identity-preserving.
• Qualitative comparisons show that the method produces detailed, realistic 3D faces even in cases where state-of-the-art methods fail or produce generic shapes.
• The method outperforms end-to-end systems like Sela et al. [44] in handling occluded and rotated faces, maintaining both detail and structural consistency.
• The ablation study confirms that the modular design—separating shape foundation from bump map details—leads to more robust and accurate reconstructions under extreme conditions.