[Paper Review] Single-View Hair Reconstruction using Convolutional Neural Networks

Yi Zhou, Liwen Hu+5 more|arXiv (Cornell University)|Jun 19, 2018

Advanced Vision and ImagingComputer Science3 citations

TL;DR

This paper proposes a real-time, deep learning-based method for single-view 3D hair reconstruction using a convolutional neural network that takes a 2D orientation field as input and generates 30K evenly distributed hair strands with realistic collision and visibility modeling. The approach enables compact, continuous hairstyle representation and interpolation, achieving 1000x speedup over prior methods while generalizing to real Internet images via synthetic-to-real domain adaptation.

ABSTRACT

We introduce a deep learning-based method to generate full 3D hair geometry from an unconstrained image. Our method can recover local strand details and has real-time performance. State-of-the-art hair modeling techniques rely on large hairstyle collections for nearest neighbor retrieval and then perform ad-hoc refinement. Our deep learning approach, in contrast, is highly efficient in storage and can run 1000 times faster while generating hair with 30K strands. The convolutional neural network takes the 2D orientation field of a hair image as input and generates strand features that are evenly distributed on the parameterized 2D scalp. We introduce a collision loss to synthesize more plausible hairstyles, and the visibility of each strand is also used as a weight term to improve the reconstruction accuracy. The encoder-decoder architecture of our network naturally provides a compact and continuous representation for hairstyles, which allows us to interpolate naturally between hairstyles. We use a large set of rendered synthetic hair models to train our network. Our method scales to real images because an intermediate 2D orientation field, automatically calculated from the real image, factors out the difference between synthetic and real hairs. We demonstrate the effectiveness and robustness of our method on a wide range of challenging real Internet pictures and show reconstructed hair sequences from videos.

Motivation & Objective

• To address the inefficiency and high storage demands of state-of-the-art hair modeling techniques that rely on large hairstyle collections and nearest neighbor retrieval.
• To enable real-time 3D hair reconstruction from unconstrained single images with high geometric fidelity and local strand detail.
• To develop a compact, continuous representation of hairstyles that supports natural interpolation between different styles.
• To bridge the domain gap between synthetic training data and real-world images using an intermediate 2D orientation field.

Proposed method

• A U-Net-like encoder-decoder CNN architecture processes the 2D orientation field of a hair image to predict strand features distributed on a parameterized 2D scalp.
• The network incorporates a collision loss to ensure physically plausible hair strand distribution and reduce intersections.
• Visibility of each strand is modeled as a weight term to improve reconstruction accuracy by emphasizing visible regions.
• The method uses a large dataset of rendered synthetic hair models for end-to-end training, enabling generalization to real images.
• An intermediate 2D orientation field is automatically extracted from real images to factor out differences between synthetic and real hair appearance.
• The continuous, compact representation from the encoder allows for natural interpolation between learned hairstyles.

Experimental results

Research questions

• RQ1 Can a deep learning model reconstruct detailed 30K-strand 3D hair geometry from a single unconstrained image in real time?
• RQ2 How can a neural network learn a compact and continuous representation of hairstyles that enables natural interpolation?
• RQ3 What loss functions are most effective for synthesizing plausible hair strand configurations with realistic collision and visibility?
• RQ4 To what extent can a model trained on synthetic data generalize to real-world images without fine-tuning?
• RQ5 How does the use of an intermediate 2D orientation field improve domain generalization from synthetic to real hair images?

Key findings

• The method achieves real-time performance, running 1000 times faster than state-of-the-art retrieval-based approaches while generating 30K hair strands.
• The inclusion of a collision loss significantly improves the plausibility of reconstructed hairstyles by reducing strand intersections.
• Visibility-based weighting enhances reconstruction accuracy by focusing on regions where strands are perceptually visible.
• The model generalizes robustly to diverse real-world images from the Internet, including challenging lighting and pose variations.
• The learned latent space enables smooth and natural interpolation between different hairstyles, demonstrating a continuous and meaningful representation.
• The use of an intermediate 2D orientation field enables effective domain adaptation, allowing the model to generalize from synthetic training data to real images without retraining.