[Paper Review] Unsupervised 3D Shape Completion through GAN Inversion

Junzhe Zhang, Xinyi Chen+7 more|arXiv (Cornell University)|Apr 27, 2021

3D Shape Modeling and Analysis38 references9 citations

TL;DR

This paper proposes ShapeInversion, an unsupervised 3D shape completion method that leverages GAN inversion to reconstruct complete shapes from partial inputs without requiring paired training data. By optimizing a latent code in a pre-trained GAN’s space to best reconstruct the partial input, ShapeInversion achieves state-of-the-art performance among unsupervised methods and generalizes robustly across diverse partial forms and real-world scans, while enabling additional capabilities like multi-output generation, shape jittering, and interpolation.

ABSTRACT

Most 3D shape completion approaches rely heavily on partial-complete shape pairs and learn in a fully supervised manner. Despite their impressive performances on in-domain data, when generalizing to partial shapes in other forms or real-world partial scans, they often obtain unsatisfactory results due to domain gaps. In contrast to previous fully supervised approaches, in this paper we present ShapeInversion, which introduces Generative Adversarial Network (GAN) inversion to shape completion for the first time. ShapeInversion uses a GAN pre-trained on complete shapes by searching for a latent code that gives a complete shape that best reconstructs the given partial input. In this way, ShapeInversion no longer needs paired training data, and is capable of incorporating the rich prior captured in a well-trained generative model. On the ShapeNet benchmark, the proposed ShapeInversion outperforms the SOTA unsupervised method, and is comparable with supervised methods that are learned using paired data. It also demonstrates remarkable generalization ability, giving robust results for real-world scans and partial inputs of various forms and incompleteness levels. Importantly, ShapeInversion naturally enables a series of additional abilities thanks to the involvement of a pre-trained GAN, such as producing multiple valid complete shapes for an ambiguous partial input, as well as shape manipulation and interpolation.

Motivation & Objective

• To address the domain generalization gap in 3D shape completion, where supervised methods fail on out-of-domain partial scans or non-standard partial forms.
• To eliminate the need for paired partial-complete shape datasets by leveraging pre-trained generative models as inductive bias.
• To enable multiple plausible completions, shape manipulation, and interpolation through latent space operations in a GAN-based framework.
• To overcome challenges in 3D GAN inversion, including non-uniform point cloud distribution and ambiguous point correspondences in unordered point clouds.

Proposed method

• Introduces a novel PatchVariance loss to improve point cloud uniformity during GAN training by minimizing variance in average distances between patch centers and their nearest neighbors.
• Proposes a dynamic k-Mask mechanism to estimate point correspondences between partial inputs and generated shapes by assigning each partial point to its k-nearest neighbors in the prediction.
• Performs GAN inversion by optimizing a latent code z such that the generated shape from the GAN best reconstructs the given partial input, using the k-Mask for reconstruction loss computation.
• Splits the inversion process into multiple sub-stages with decaying learning rates for z and generator parameters θ to improve convergence and stability.
• Utilizes a pre-trained GAN (e.g., r-GAN or tree-GAN) on complete shapes, enabling zero-shot completion without fine-tuning on real scans.
• Enables shape jittering and interpolation by perturbing or interpolating latent codes z, leveraging the disentangled and continuous latent space of the GAN.

Figure 1: ShapeInversion incorporates the prior captured by a well-trained GAN. It shows exceptional generalization ability for shape completion: is invariant to partial form changes, i.e. , (a) virtual scans, generated by back-projecting 2.5D depth images into 3D, (b) ball-holed partial shapes, gen

Experimental results

Research questions

• RQ1 Can GAN inversion be effectively adapted to 3D shape completion without paired training data?
• RQ2 How well can a GAN-based method generalize to diverse partial forms, including virtual scans, ball-holed shapes, and real-world scans?
• RQ3 Can the latent space of a pre-trained GAN naturally support multi-modal completion, shape manipulation, and interpolation?
• RQ4 How do non-uniform point distributions and unordered point cloud structures affect GAN inversion in 3D, and how can they be mitigated?

Key findings

• ShapeInversion outperforms the SOTA unsupervised method pcl2pcl on the ShapeNet benchmark, achieving comparable results to supervised methods despite no paired data.
• The method generalizes robustly across all tested partial forms, including virtual scans, ball-holed shapes, semantic part removal, and real-world scans from ScanNet and KITTI.
• On real-world scans, ShapeInversion achieves higher fidelity and better alignment with the input than pcl2pcl, which requires retraining on real scans.
• The PatchVariance loss effectively improves uniformity in generated point clouds across all shape categories, including fine and bulk structures, without architectural changes.
• The k-Mask mechanism enables dynamic, robust correspondence estimation even when semantic parts are spatially misaligned, outperforming fixed-threshold or voxel-based baselines.
• ShapeInversion naturally supports multi-modal completion, shape jittering, and interpolation due to the disentangled latent space of the pre-trained GAN.

Figure 2: GAN inversion for shape completion. A latent vector $\mathbf{z}$ is used by the pre-trained generator $G$ to reconstruct a complete shape $\mathbf{x}_{c}$ . The degradation function $M$ (Sec. 3.2 ) then transforms $\mathbf{x}_{c}$ into a partial shape $\mathbf{x}_{p}$ . The supervision sig