[Paper Review] A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images

Biwen Lei, Jianqiang Ren+3 more|arXiv (Cornell University)|Feb 28, 2023

Face recognition and analysis4 citations

TL;DR

This paper proposes a Hierarchical Representation Network (HRN) for high-fidelity 3D face reconstruction from in-the-wild images by disentangling facial geometry into low-, mid-, and high-frequency components using face-wise blendshapes, vertex-wise deformations, and pixel-wise displacement maps. The method integrates 3D priors of facial details and a de-retouching module to improve accuracy and realism, achieving state-of-the-art performance on single- and multi-view benchmarks, including a new high-quality dataset, FaceHD-100.

ABSTRACT

Limited by the nature of the low-dimensional representational capacity of 3DMM, most of the 3DMM-based face reconstruction (FR) methods fail to recover high-frequency facial details, such as wrinkles, dimples, etc. Some attempt to solve the problem by introducing detail maps or non-linear operations, however, the results are still not vivid. To this end, we in this paper present a novel hierarchical representation network (HRN) to achieve accurate and detailed face reconstruction from a single image. Specifically, we implement the geometry disentanglement and introduce the hierarchical representation to fulfill detailed face modeling. Meanwhile, 3D priors of facial details are incorporated to enhance the accuracy and authenticity of the reconstruction results. We also propose a de-retouching module to achieve better decoupling of the geometry and appearance. It is noteworthy that our framework can be extended to a multi-view fashion by considering detail consistency of different views. Extensive experiments on two single-view and two multi-view FR benchmarks demonstrate that our method outperforms the existing methods in both reconstruction accuracy and visual effects. Finally, we introduce a high-quality 3D face dataset FaceHD-100 to boost the research of high-fidelity face reconstruction. The project homepage is at https://younglbw.github.io/HRN-homepage/.

Motivation & Objective

• To address the limitation of 3DMM-based methods in capturing high-frequency facial details like wrinkles and dimples.
• To improve reconstruction accuracy and visual fidelity by disentangling facial geometry into low-, mid-, and high-frequency components.
• To decouple geometry and appearance more effectively by introducing a de-retouching module to reduce confusion between skin blemishes and lighting effects.
• To extend the framework to multi-view reconstruction for improved performance with sparse input views.
• To provide a high-quality 3D face dataset, FaceHD-100, to advance research in high-fidelity, sparse-view face reconstruction.

Proposed method

• The framework employs a hierarchical representation strategy that models low-frequency geometry via 3DMM coefficients, mid-frequency details via vertex-wise deformation maps, and high-frequency details via pixel-wise displacement maps.
• Two image translation networks are used to predict deformation and displacement maps from a single image, enabling coarse-to-fine reconstruction.
• 3D priors of mid- and high-frequency facial details are learned from real face scans and incorporated to guide realistic geometry modeling.
• A de-retouching module is introduced to refine the base texture, reducing artifacts caused by skin texture and illumination ambiguity.
• The method is extended to multi-view face reconstruction (MV-HRN) by enforcing consistency of detailed geometry across views.
• The framework is trained end-to-end with a contour-aware loss to improve facial contour accuracy and a multi-scale loss for geometric fidelity.

Experimental results

Research questions

• RQ1 Can a hierarchical representation strategy effectively disentangle and model mid- and high-frequency facial details beyond the limitations of standard 3DMM?
• RQ2 How can 3D priors of facial details improve the realism and accuracy of reconstructed geometry?
• RQ3 To what extent does a de-retouching module enhance the disentanglement of geometry and appearance in in-the-wild images?
• RQ4 Can the single-view HRN framework be effectively extended to multi-view reconstruction with sparse input?
• RQ5 How does the proposed FaceHD-100 dataset compare to existing benchmarks in supporting high-fidelity, sparse-view face reconstruction?

Key findings

• HRN achieves a median reconstruction error of 1.13mm on the multi-view FaceScape benchmark, outperforming prior methods such as MVFNet (1.76mm) and DFNRMVS (1.79mm).
• On the single-view FFHQ benchmark, HRN reduces median error to 1.13mm, surpassing SOTA methods like DFNRMVS (1.79mm) and MVFNet (1.76mm).
• Ablation studies show that hierarchical modeling reduces error by ~0.5mm compared to base 3DMM, and 3D detail priors contribute ~0.3mm improvement.
• The de-retouching module reduces geometry-appearance confusion, producing more realistic textures and eliminating spurious details caused by skin blemishes.
• MV-HRN achieves consistent performance with only two input views (1.17mm error), improving with more views, and outperforms a naive averaging baseline.
• The introduction of FaceHD-100, a dataset of 2,000 high-fidelity 3D face models with high-definition multi-view images, supports future research in sparse-view, high-fidelity face reconstruction.