[Paper Review] High-Res Facial Appearance Capture from Polarized Smartphone Images

Dejan Azinović, Olivier Maury+3 more|arXiv (Cornell University)|Dec 2, 2022

Face recognition and analysis4 citations

TL;DR

This paper presents a low-cost, smartphone-based method for high-resolution facial appearance capture using polarization foils attached to the camera and flashlight. By capturing cross-polarized and parallel-polarized sequences, the method disentangles diffuse and specular reflectance, enabling high-fidelity reconstruction of 4K diffuse albedo, specular gain, and normal maps via a coarse-to-fine differentiable rendering pipeline, resulting in photo-realistic 3D face rendering in novel views and lighting.

ABSTRACT

We propose a novel method for high-quality facial texture reconstruction from RGB images using a novel capturing routine based on a single smartphone which we equip with an inexpensive polarization foil. Specifically, we turn the flashlight into a polarized light source and add a polarization filter on top of the camera. Leveraging this setup, we capture the face of a subject with cross-polarized and parallel-polarized light. For each subject, we record two short sequences in a dark environment under flash illumination with different light polarization using the modified smartphone. Based on these observations, we reconstruct an explicit surface mesh of the face using structure from motion. We then exploit the camera and light co-location within a differentiable renderer to optimize the facial textures using an analysis-by-synthesis approach. Our method optimizes for high-resolution normal textures, diffuse albedo, and specular albedo using a coarse-to-fine optimization scheme. We show that the optimized textures can be used in a standard rendering pipeline to synthesize high-quality photo-realistic 3D digital humans in novel environments.

Motivation & Objective

• To enable high-quality, photo-realistic 3D facial appearance capture using only a commodity smartphone and inexpensive polarization foils.
• To overcome the limitations of traditional light stage setups by simplifying capture to a single device with minimal hardware modification.
• To disentangle diffuse and specular surface responses using polarization, enabling accurate reconstruction of albedo, normal, and specular maps.
• To develop a coarse-to-fine, differentiable rendering optimization strategy that enhances texture sharpness and reduces blur.

Proposed method

• Equipping a smartphone with polarization foils on both the camera lens and flashlight to enable controlled polarization states during capture.
• Capturing two short video sequences in a dark room: one with cross-polarized light (to emphasize diffuse reflection) and one with parallel-polarized light (to preserve specular response).
• Reconstructing facial geometry using structure-from-motion and multi-view stereo, followed by non-rigid fitting of a FLAME parametric face model to ensure consistent UV parameterization.
• Performing a two-stage photometric optimization: first estimating high-resolution diffuse albedo and initial normal map from cross-polarized data, then refining specular gain and final normal map using parallel-polarized views.
• Employing a coarse-to-fine optimization with mipmapping, where each resolution level uses only pixels with corresponding UV-space footprints to preserve texture sharpness.
• Integrating a differentiable renderer that models directional flashlight attenuation, Fresnel effects, and ambient light to improve reconstruction accuracy.

Experimental results

Research questions

• RQ1 Can a smartphone with low-cost polarization foils achieve high-resolution facial texture reconstruction comparable to professional light stage systems?
• RQ2 How effective is polarization-based separation of diffuse and specular reflectance in a co-located camera-light setup for consumer-grade capture?
• RQ3 To what extent does a coarse-to-fine, mipmapping-based optimization improve texture sharpness compared to direct high-resolution optimization?
• RQ4 How do modeling assumptions—such as directional flashlight attenuation and Fresnel effects—affect the accuracy of the reconstructed appearance?
• RQ5 Can the resulting textures be used in standard rendering pipelines to produce photo-realistic results under novel lighting and viewpoints?

Key findings

• The method achieves high-resolution 4K diffuse albedo, specular gain, and normal map reconstruction from only two smartphone video sequences with polarization foils.
• Coarse-to-fine optimization with mipmapping significantly improves texture sharpness, reducing blur compared to direct high-resolution optimization.
• Accounting for directional flashlight attenuation and Fresnel effects leads to lower re-projection error and more accurate shading across multiple views.
• Joint optimization of all textures without polarization leads to leakage of specular information into the diffuse texture, degrading realism.
• The full pipeline requires approximately 2.5 hours of computation (including 1 hour for MVS), with 30 GB GPU memory at 4096×4096 texture resolution, outperforming NLT (10h) and NextFace (6h) in speed for the same input.
• The reconstructed textures are compatible with standard rendering software like Blender and enable photo-realistic rendering with subsurface scattering and novel lighting.