[Paper Review] A Simple, Fast and Highly-Accurate Algorithm to Recover 3D Shape from 2D Landmarks on a Single Image

Ruiqi Zhao, Yan Wang+1 more|arXiv (Cornell University)|Sep 28, 2016

Face recognition and analysis17 references4 citations

TL;DR

This paper proposes a feed-forward deep neural network that accurately recovers 3D shape from 2D landmarks on a single image, achieving reconstruction errors below 0.004 for faces, 0.0022 for cars, 0.022 for bodies, and 0.0004 for flags—outperforming state-of-the-art methods by up to two-fold. The model is trained quickly, runs at over 1,000 frames per second, and is robust to noise and missing data via innovative data augmentation.

ABSTRACT

Three-dimensional shape reconstruction of 2D landmark points on a single image is a hallmark of human vision, but is a task that has been proven difficult for computer vision algorithms. We define a feed-forward deep neural network algorithm that can reconstruct 3D shapes from 2D landmark points almost perfectly (i.e., with extremely small reconstruction errors), even when these 2D landmarks are from a single image. Our experimental results show an improvement of up to two-fold over state-of-the-art computer vision algorithms; 3D shape reconstruction of human faces is given at a reconstruction error < .004, cars at .0022, human bodies at .022, and highly-deformable flags at an error of .0004. Our algorithm was also a top performer at the 2016 3D Face Alignment in the Wild Challenge competition (done in conjunction with the European Conference on Computer Vision, ECCV) that required the reconstruction of 3D face shape from a single image. The derived algorithm can be trained in a couple hours and testing runs at more than 1, 000 frames/s on an i7 desktop. We also present an innovative data augmentation approach that allows us to train the system efficiently with small number of samples. And the system is robust to noise (e.g., imprecise landmark points) and missing data (e.g., occluded or undetected landmark points).

Motivation & Objective

• To develop a general-purpose algorithm that reconstructs 3D shapes from 2D landmarks on a single image, applicable to rigid, articulated, and highly deformable objects.
• To address the challenge of low accuracy in existing methods when dealing with non-dense or sparse 2D fiducials and non-linear deformations.
• To enable real-time performance (>1,000 fps) while maintaining high accuracy and robustness to noise and missing data.
• To train effectively even with small datasets using a novel data augmentation strategy based on multiple camera models.

Proposed method

• A feed-forward deep neural network is trained to learn the mapping from 2D landmark coordinates to 3D shape, using hierarchical nonlinear transformations to recover depth.
• The network is trained end-to-end using a loss function that minimizes the L2 distance between predicted and ground-truth 3D coordinates.
• An innovative data augmentation technique generates synthetic 2D landmark points by simulating multiple camera views, scales, and viewpoints from a single 3D point cloud.
• Mini-batch training is used for large datasets, while the same architecture efficiently handles small datasets through augmented data.
• The model is invariant to global scaling and estimates 3D shape independently of scale factors.
• Robustness to noise and missing data is achieved through training with corrupted or incomplete landmark sets.

Experimental results

Research questions

• RQ1 Can a single deep neural network generalize across diverse object categories, including rigid, articulated, and highly deformable shapes, using only 2D landmarks from a single image?
• RQ2 How can 3D shape reconstruction accuracy be significantly improved over state-of-the-art methods while maintaining real-time inference speed?
• RQ3 To what extent can a deep learning model remain accurate and robust when 2D landmarks are noisy or partially missing?
• RQ4 Can effective training be achieved with limited real-world 3D-2D data pairs using synthetic data augmentation?

Key findings

• The proposed method achieves a mean 3D reconstruction error of 0.004 on human faces, significantly outperforming prior state-of-the-art methods.
• For cars, the mean reconstruction error is 0.0022, demonstrating high accuracy on rigid, detailed objects.
• On human bodies, the mean error is 0.022, indicating strong performance on articulated structures.
• For highly deformable flags, the mean error is as low as 0.0004, highlighting exceptional performance on non-rigid, dynamic shapes.
• The algorithm runs at over 1,000 frames per second on an i7 desktop, exceeding real-time performance.
• The model maintains low error even with up to 5% Gaussian noise on landmark positions and with up to one missing landmark per sample.