[Paper Review] 3D Face Reconstruction from Light Field Images: A Model-free Approach

Mingtao Feng, Syed Zulqarnain Gilani+2 more|arXiv (Cornell University)|Nov 16, 2017

Face recognition and analysis34 references3 citations

TL;DR

This paper proposes a model-free 3D face reconstruction method from single light field images using a densely connected CNN (FaceLFnet) that regresses 3D facial curves from horizontal and vertical Epipolar Plane Images (EPIs). By synthesizing photo-realistic light field data from 3D facial scans, the approach achieves a 20%+ reduction in reconstruction error over state-of-the-art methods, even with only 14K training images of 80 identities.

ABSTRACT

Reconstructing 3D facial geometry from a single RGB image has recently instigated wide research interest. However, it is still an ill-posed problem and most methods rely on prior models hence undermining the accuracy of the recovered 3D faces. In this paper, we exploit the Epipolar Plane Images (EPI) obtained from light field cameras and learn CNN models that recover horizontal and vertical 3D facial curves from the respective horizontal and vertical EPIs. Our 3D face reconstruction network (FaceLFnet) comprises a densely connected architecture to learn accurate 3D facial curves from low resolution EPIs. To train the proposed FaceLFnets from scratch, we synthesize photo-realistic light field images from 3D facial scans. The curve by curve 3D face estimation approach allows the networks to learn from only 14K images of 80 identities, which still comprises over 11 Million EPIs/curves. The estimated facial curves are merged into a single pointcloud to which a surface is fitted to get the final 3D face. Our method is model-free, requires only a few training samples to learn FaceLFnet and can reconstruct 3D faces with high accuracy from single light field images under varying poses, expressions and lighting conditions. Comparison on the BU-3DFE and BU-4DFE datasets show that our method reduces reconstruction errors by over 20% compared to recent state of the art.

Motivation & Objective

• Address the ill-posed problem of 3D face reconstruction from single RGB images, which suffers from depth ambiguity and scale/bas-relief ambiguities.
• Overcome limitations of model-based approaches that rely on prior 3D morphable models and are constrained by training data distribution.
• Enable accurate 3D face reconstruction under varying poses, expressions, and lighting conditions without using facial landmarks or explicit shape models.
• Develop a large-scale, photo-realistic synthetic light field dataset with ground truth 3D facial scans to train deep learning models.
• Demonstrate that curve-by-curve regression from EPIs enables high-accuracy 3D reconstruction with significantly fewer training identities than end-to-end methods.

Proposed method

• Synthesize photo-realistic light field images from 3D facial scans in the BU-3DFE and BU-4DFE datasets, varying pose, lighting, and expression to increase realism.
• Extract horizontal and vertical Epipolar Plane Images (EPIs) from the synthesized light fields to capture depth-related ray variations.
• Train two separate densely connected CNNs (FaceLFnet) to regress 3D facial curves from the respective EPIs, leveraging feature reuse and gradient flow.
• Use camera parameters to merge predicted depth maps from horizontal and vertical EPIs into a single 3D point cloud.
• Fit a surface to the merged point cloud to generate the final 3D facial mesh.
• Leverage the densenet architecture to capture subtle depth gradients in low-resolution EPIs, improving estimation accuracy.

Experimental results

Research questions

• RQ1 Can 3D face reconstruction from light field images be achieved without relying on 3D morphable models or explicit shape priors?
• RQ2 To what extent can a curve-by-curve regression approach from EPIs improve reconstruction accuracy compared to end-to-end 3D face estimation?
• RQ3 How effective is a synthetic light field dataset derived from real 3D facial scans in enabling high-accuracy 3D reconstruction with minimal real training data?
• RQ4 How robust is the proposed method to variations in facial pose, expression, and lighting conditions?
• RQ5 Can a model-free approach achieve state-of-the-art performance with only 14K training images of 80 identities?

Key findings

• The proposed method reduces mean reconstruction error by 2.78 mm on the BU-3DFE dataset, outperforming prior state-of-the-art methods (e.g., Kemelmacher et al. at 3.89 mm).
• The method achieves a 20%+ reduction in reconstruction error compared to recent SOTA, with a median error of 1.73 mm and 90% largest error of 5.30 mm.
• Reconstruction error increases by only 0.31 mm when facial pose varies by 30 degrees, demonstrating strong robustness to pose variation.
• Under extreme facial expressions, the RMSE increases from 2.49 mm (neutral) to 2.98 mm (surprise), with the highest error observed in sad expressions.
• On the BU-4DFE dataset, the method achieves an NME of 3.72, outperforming 3DDFA (5.14), EOS (5.33), and VRN-Guided (4.71).
• The method generalizes well across ethnicities and lighting conditions due to the model-free, EPI-based learning approach.