[Paper Review] Neural Generative Models for 3D Faces with Application in 3D Texture Free Face Recognition

Ahmed El-Sayed, Elif Kongar+3 more|arXiv (Cornell University)|Nov 11, 2018

Face recognition and analysis13 references4 citations

TL;DR

This paper proposes a neural generative modeling approach to create compact, canonical representations of 3D face point clouds, reducing storage size and enabling accurate regeneration. By training neural models on 3D scans and using their weights as input to a Siamese CNN, the method achieves 100% verification accuracy on the Bosphorus dataset—surpassing state-of-the-art results—while supporting data augmentation and compatibility with heterogeneous depth sensors.

ABSTRACT

Using heterogeneous depth cameras and 3D scanners in 3D face verification causes variations in the resolution of the 3D point clouds. To solve this issue, previous studies use 3D registration techniques. Out of these proposed techniques, detecting points of correspondence is proven to be an efficient method given that the data belongs to the same individual. However, if the data belongs to different persons, the registration algorithms can convert the 3D point cloud of one person to another, destroying the distinguishing features between the two point clouds. Another issue regarding the storage size of the point clouds. That is, if the captured depth image contains around 50 thousand points in the cloud for a single pose for one individual, then the storage size of the entire dataset will be in order of giga if not tera bytes. With these motivations, this work introduces a new technique for 3D point clouds generation using a neural modeling system to handle the differences caused by heterogeneous depth cameras, and to generate a new face canonical compact representation. The proposed system reduces the stored 3D dataset size, and if required, provides an accurate dataset regeneration. Furthermore, the system generates neural models for all gallery point clouds and stores these models to represent the faces in the recognition or verification processes. For the probe cloud to be verified, a new model is generated specifically for that particular cloud and is matched against pre-stored gallery model presentations to identify the query cloud. This work also introduces the utilization of Siamese deep neural network in 3D face verification using generated model representations as raw data for the deep network, and shows that the accuracy of the trained network is comparable all published results on Bosphorus dataset.

Motivation & Objective

• To address storage bloat and heterogeneity in 3D face point clouds from diverse depth cameras.
• To develop a compact, canonical representation of 3D faces using neural generative modeling.
• To enable accurate 3D face verification using learned model weights as input to a Siamese deep neural network.
• To support data augmentation and super-resolution for small or noisy datasets.
• To achieve state-of-the-art performance on the Bosphorus 3D face recognition dataset using texture-free neutral-expression scans.

Proposed method

• A neural generative model is trained to learn a compact, canonical representation of 3D face point clouds from raw depth scans.
• The model uses regression to generate smoothed, noise-reduced 3D point clouds that preserve facial geometry.
• Trained model weights are used as raw input data for a Siamese convolutional neural network (Siamese CNN).
• Positive and negative pairs of model weights are generated via data augmentation to train the Siamese network for verification.
• The Siamese network is trained and tested using 50,000 positive and 70,000 negative pairs, with 50% of data used for training and 50% for testing.
• The system supports regeneration of 3D point clouds from model weights, enabling compatibility with existing 3D face recognition techniques.

Experimental results

Research questions

• RQ1 Can a neural generative model produce accurate, compact, and noise-reduced representations of 3D face point clouds from heterogeneous depth sensors?
• RQ2 Can the weights of trained generative models serve as effective input for a Siamese CNN in 3D face verification?
• RQ3 Does the proposed method achieve superior verification accuracy compared to state-of-the-art techniques on the Bosphorus dataset?
• RQ4 To what extent can the generative model support data augmentation and super-resolution for small or noisy datasets?
• RQ5 Can the model representation preserve distinguishing facial features while reducing storage size and enabling regeneration?

Key findings

• The proposed neural generative model produces high-quality, smoothed 3D face point clouds that closely match the original data, even with significant noise in the input.
• The Siamese CNN trained on generated model weights achieved 100% verification accuracy on the Bosphorus dataset, outperforming all previously published results.
• The training and testing loss curves show no overfitting, indicating stable and effective learning over 50,000 iterations.
• The ROC and Precision-Recall curves demonstrate strong discriminative performance, confirming the robustness of the learned representations.
• The method enables data augmentation by generating additional positive and negative pairs from a single model, addressing small dataset limitations.
• The model weights serve as a compact, lossy representation that reduces 3D point cloud storage size while allowing full regeneration of the original geometry.