[Paper Review] Unsupervised 3D Shape Learning from Image Collections in the Wild

Attila Szabó, Paolo Favaro|arXiv (Cornell University)|Nov 26, 2018

3D Shape Modeling and Analysis27 references4 citations

TL;DR

This paper proposes a fully unsupervised method to learn 3D shapes and textures from unpaired image collections without any annotations, templates, or 3D supervision. It uses a generative adversarial network (GAN) with a differentiable renderer to train a model that synthesizes realistic 3D objects and views, achieving state-of-the-art unsupervised 3D reconstruction on real and synthetic data.

ABSTRACT

We present a method to learn the 3D surface of objects directly from a collection of images. Previous work achieved this capability by exploiting additional manual annotation, such as object pose, 3D surface templates, temporal continuity of videos, manually selected landmarks, and foreground/background masks. In contrast, our method does not make use of any such annotation. Rather, it builds a generative model, a convolutional neural network, which, given a noise vector sample, outputs the 3D surface and texture of an object and a background image. These 3 components combined with an additional random viewpoint vector are then fed to a differential renderer to produce a view of the sampled object and background. Our general principle is that if the output of the renderer, the generated image, is realistic, then its input, the generated 3D and texture, should also be realistic. To achieve realism, the generative model is trained adversarially against a discriminator that tries to distinguish between the output of the renderer and real images from the given data set. Moreover, our generative model can be paired with an encoder and trained as an autoencoder, to automatically extract the 3D shape, texture and pose of the object in an image. Our trained generative model and encoder show promising results both on real and synthetic data, which demonstrate for the first time that fully unsupervised 3D learning from image collections is possible.

Motivation & Objective

• To enable 3D shape and texture reconstruction from unpaired 2D image collections without any manual annotations or 3D templates.
• To eliminate reliance on pose, landmarks, masks, or video sequences typically used in prior 3D reconstruction methods.
• To develop a generative model that learns 3D structure directly from image data via adversarial training with a differentiable renderer.
• To demonstrate that realistic 3D shapes can be learned purely from image data by enforcing view consistency through adversarial realism.
• To validate the method on both real (CelebA) and synthetic (ShapeNet) datasets, showing feasibility of fully unsupervised 3D learning.

Proposed method

• Train a generative model that maps a noise vector to a 3D mesh, texture, and background image.
• Use a differentiable renderer to synthesize novel views from the generated 3D object and viewpoint vector.
• Train the generator adversarially against a discriminator that distinguishes real images from rendered ones.
• Integrate a smoothing loss ($\mathcal{L}_S$) to reduce high-frequency artifacts in the 3D mesh by penalizing non-smooth triangle normals.
• Train an encoder as part of an autoencoder setup to infer 3D shape, texture, and pose from a single input image.
• Optimize the combined objective: $\mathcal{L}_{GAN}(G,D) + \lambda_S \mathcal{L}_S(G)$, with viewpoint sampling to cover diverse views.

Experimental results

Research questions

• RQ1 Can 3D shapes be learned from unpaired 2D images without any supervision, templates, or annotations?
• RQ2 Is it possible to train a generative model to produce realistic 3D objects and views using only image-level realism as a supervisory signal?
• RQ3 How can high-frequency artifacts in generated 3D meshes be mitigated while preserving meaningful shape details?
• RQ4 To what extent can the model generalize to real-world image collections like CelebA without prior knowledge of object categories?
• RQ5 Can the autoencoder component effectively reconstruct 3D shape, texture, and pose from a single image in a fully unsupervised manner?

Key findings

• The method achieves plausible 3D shape and texture reconstruction on the CelebA dataset without any annotations, demonstrating the feasibility of fully unsupervised 3D learning.
• The inclusion of a smoothing loss ($\mathcal{L}_S$) significantly reduces high-frequency artifacts in the 3D mesh while preserving major facial features like the nose and lips.
• Without smoothing, the generated 3D meshes exhibit noticeable high-frequency noise; with excessive smoothing, the model collapses to an average ellipsoidal shape.
• The autoencoder variant successfully reconstructs 3D shapes and textures from single images, though with less detail than supervised methods.
• The system mitigates the hollow-mask ambiguity by constraining object size via resizing when the radial extent exceeds a threshold.
• On the ShapeNet car category, the method successfully generalizes to synthetic data with full 360° view rendering, confirming robustness across datasets.