[Paper Review] AG3D: Learning to Generate 3D Avatars from 2D Image Collections

Zijian Dong, Xu Chen+4 more|arXiv (Cornell University)|May 3, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

AG3D proposes a novel adversarial generative model that learns to synthesize high-fidelity 3D avatars with realistic geometry and appearance from unstructured 2D image collections, using a holistic 3D generator with an efficient articulation module and multiple specialized discriminators—including for normal maps and facial regions—achieving state-of-the-art performance in both geometry and appearance quality while naturally modeling loose clothing like dresses and skirts.

ABSTRACT

While progress in 2D generative models of human appearance has been rapid, many applications require 3D avatars that can be animated and rendered. Unfortunately, most existing methods for learning generative models of 3D humans with diverse shape and appearance require 3D training data, which is limited and expensive to acquire. The key to progress is hence to learn generative models of 3D avatars from abundant unstructured 2D image collections. However, learning realistic and complete 3D appearance and geometry in this under-constrained setting remains challenging, especially in the presence of loose clothing such as dresses. In this paper, we propose a new adversarial generative model of realistic 3D people from 2D images. Our method captures shape and deformation of the body and loose clothing by adopting a holistic 3D generator and integrating an efficient and flexible articulation module. To improve realism, we train our model using multiple discriminators while also integrating geometric cues in the form of predicted 2D normal maps. We experimentally find that our method outperforms previous 3D- and articulation-aware methods in terms of geometry and appearance. We validate the effectiveness of our model and the importance of each component via systematic ablation studies.

Motivation & Objective

• Address the lack of diverse, high-quality 3D avatars for animation and rendering, which are limited by the scarcity and cost of 3D training data.
• Overcome the challenges of generating realistic 3D human geometry and appearance from unstructured 2D image collections, especially for articulated bodies and loose clothing.
• Eliminate reliance on 3D supervision by learning a generative model solely from 2D images, enabling scalable and diverse 3D avatar synthesis.
• Improve geometric fidelity and perceptual realism in highly under-constrained settings through multi-discriminator training and geometric supervision via 2D normal maps.

Proposed method

• Employ a monolithic 3D generator using tri-plane representation for efficient volume rendering and holistic 3D shape and appearance modeling.
• Integrate an efficient articulation module based on Fast-SNARF to enable flexible deformation of the body and loose clothing, surpassing SMPL-based nearest-neighbor methods.
• Apply empty-space skipping during rendering, guided by a coarse human body prior, to accelerate inference without sacrificing quality.
• Use multiple discriminators: one for RGB images, one for predicted 2D normal maps, and a dedicated face discriminator to enhance realism in critical regions.
• Train with adversarial loss on both image and normal map branches, leveraging off-the-shelf monocular normal estimators to provide geometric supervision.
• Optimize the generator using a multi-branch GAN setup, where each discriminator specializes in a specific aspect (e.g., face, surface normals) to improve localized fidelity.

Figure 2 : Method Overview. Holistic 3D Human Generation: Given a latent vector $\mathbf{z}$ , our method generates human shape $d$ and appearance $\mathbf{c}$ in canonical space. In addition, we compute surface normals $\mathbf{n}$ via the spatial derivatives of the canonical shape which is represe

Experimental results

Research questions

• RQ1 Can a generative 3D model learn realistic human geometry and appearance from unstructured 2D image collections without any 3D supervision?
• RQ2 How can a generative model effectively capture the complex deformations of loose clothing such as dresses and skirts without part-based decomposition?
• RQ3 To what extent does incorporating 2D normal map supervision improve the geometric quality of generated 3D avatars in an under-constrained 2D-to-3D generation setting?
• RQ4 How do specialized discriminators for faces and normal maps contribute to improving perceptual realism compared to a single global discriminator?
• RQ5 Can a holistic 3D generator with an efficient articulation module outperform part-based models in modeling continuous, flowing garments?

Key findings

• AG3D achieves state-of-the-art performance in both geometry and appearance quality, outperforming prior 3D- and articulation-aware methods on quantitative metrics including FID and FID_normal.
• The inclusion of a normal map discriminator significantly reduces surface noise and holes, especially on faces, as evidenced by a substantial improvement in FID_normal.
• Face-specific discriminators lead to a large drop in FID_face when removed, confirming their critical role in enhancing facial realism.
• The Fast-SNARF-based articulation module successfully deforms skirts and long hair without splitting or discontinuities, unlike SMPL-based baselines that suffer from artifacts far from the body surface.
• AG3D generates 3D avatars at 9.5 FPS for 512² images—over three times faster than EVA3D—while achieving better performance in geometry and appearance.
• Ablation studies confirm that each component (normal discriminator, face discriminator, and advanced deformer) contributes significantly to the final quality, with the full model outperforming ablated variants across all metrics.

(a) View Control