[Paper Review] Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

Xingang Pan, Bo Dai+3 more|arXiv (Cornell University)|Nov 2, 2020

Generative Adversarial Networks and Image Synthesis43 references51 citations

TL;DR

The paper shows that off-the-shelf 2D GANs trained on RGB images contain latent 3D geometric cues, enabling unsupervised 3D shape reconstruction from a single image by iteratively exploring GAN image manifolds with a weak shape prior and differentiable rendering.

ABSTRACT

Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an off-the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cats, cars, and buildings. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code is available at https://github.com/XingangPan/GAN2Shape.

Motivation & Objective

• Investigate whether pre-trained 2D GANs inherently capture 3D geometry.
• Develop an unsupervised pipeline to reconstruct 3D shape from a single 2D image without 2D keypoints or 3D annotations.
• Enable 3D-aware image manipulations such as rotation and relighting from GAN-derived 3D shapes.
• Evaluate against prior unsupervised 3D reconstruction methods and demonstrate cross-category applicability.

Proposed method

• Use a weak convex shape prior (ellipsoid) to initialize depth for the target image.
• Render pseudo samples with random viewpoints and lighting using the current depth/albedo, then GAN-invert these samples to obtain projected samples within the GAN manifold.
• Learn a 3D shape through a photo-geometric autoencoding pipeline by jointly optimizing depth, albedo, viewpoint, and light for projected samples to match the reconstructed projections.
• Iteratively refine the 3D shape and networks across cycles to capture finer details.
• Incorporate a regularized latent-offset strategy to keep GAN latent codes within the intermediate latent space and improve reconstruction quality.
• Optionally extend to joint multi-instance training by using relative latent offsets for better generalization.

Experimental results

Research questions

• RQ1 Can 2D GANs trained on RGB images implicitly encode 3D geometry for objects like faces, cars, and buildings?
• RQ2 Can an unsupervised pipeline reconstruct accurate 3D shapes from a single 2D image by leveraging GAN image manifolds and a weak shape prior?
• RQ3 What is the impact of shape priors and latent regularization on 3D reconstruction quality and symmetry assumptions?
• RQ4 How well do the recovered 3D shapes enable 3D-aware manipulations such as rotation and relighting?
• RQ5 How does the proposed method compare to existing unsupervised 3D shape learning methods on standard benchmarks?

Key findings

• The method reconstructs 3D shapes with high fidelity for faces, cats, cars, and buildings from single images.
• Compared to Unsup3d, the approach better handles asymmetry and large viewpoint variations and avoids reliance on symmetry priors.
• Quantitative results on BFM show favorable SIDE and MAD metrics versus baselines, demonstrating improved 3D reconstruction without keypoints or 3D supervision.
• The recovered 3D shapes enable realistic 3D-aware manipulations like rotation and relighting, closely aligned with the underlying 3D geometry.