[Paper Review] Point Cloud Generation with Continuous Conditioning

Larissa T. Triess, Andre Bühler+3 more|arXiv (Cornell University)|Feb 17, 2022

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes a novel conditional GAN framework for 3D point cloud generation that enables continuous control over object dimensions (e.g., width, height) using continuous conditioning parameters. By training the generator with labels sampled from a kernel density estimation (KDE) of the data distribution, the method achieves high-quality, diverse, and semantically meaningful shape generation, generalizing effectively to out-of-distribution dimensions not seen during training.

ABSTRACT

Generative models can be used to synthesize 3D objects of high quality and diversity. However, there is typically no control over the properties of the generated object.This paper proposes a novel generative adversarial network (GAN) setup that generates 3D point cloud shapes conditioned on a continuous parameter. In an exemplary application, we use this to guide the generative process to create a 3D object with a custom-fit shape. We formulate this generation process in a multi-task setting by using the concept of auxiliary classifier GANs. Further, we propose to sample the generator label input for training from a kernel density estimation (KDE) of the dataset. Our ablations show that this leads to significant performance increase in regions with few samples. Extensive quantitative and qualitative experiments show that we gain explicit control over the object dimensions while maintaining good generation quality and diversity.

Motivation & Objective

• To enable explicit, continuous control over 3D object dimensions (e.g., width, height) during point cloud generation.
• To address the challenge of generating realistic and diverse 3D shapes when conditioning on continuous, infinitely large parameter spaces.
• To improve generalization to out-of-distribution dimensions by enhancing training data coverage in low-density regions.
• To maintain high generation quality and diversity while enabling custom-fit shape synthesis.
• To propose a novel label sampling strategy based on kernel density estimation (KDE) that improves performance in data-sparse regions.

Proposed method

• Formulates continuous conditioning as a multi-task learning problem within a GAN framework, using an auxiliary classifier GAN (AcGAN) setup with a regression head for continuous parameters.
• Uses a weighted loss combining adversarial loss and regression loss to train the discriminator and generator jointly.
• Samples conditioning labels from a kernel density estimation (KDE) of the real data distribution during training, rather than from discrete dataset labels.
• Applies a k-nearest neighbor (k-NN) classifier with KDE-based regions to evaluate generalization across in- and out-of-distribution dimensions.
• Employs standard 3D generation metrics including Chamfer Distance (CD), Earth Mover’s Distance (EMD), Fréchet Point Cloud Distance (FPD), and coverage (COV) for evaluation.
• Trains the generator with a fixed latent vector z and varying continuous conditioning vectors y to produce diverse shapes for the same latent code.

Experimental results

Research questions

• RQ1 Can a GAN-based 3D point cloud generator be effectively conditioned on continuous parameters such as object dimensions?
• RQ2 Does sampling conditioning labels from a KDE of the data distribution improve generation quality and generalization, especially in low-density regions?
• RQ3 Can the model generate realistic, diverse, and semantically meaningful 3D shapes for out-of-distribution dimensions not present in the training data?
• RQ4 How does the proposed KDE-based label sampling strategy compare to standard label sampling from the dataset in terms of performance and diversity?
• RQ5 To what extent does continuous conditioning preserve generation quality while enabling precise control over object dimensions?

Key findings

• The proposed method achieves a Fréchet Point Cloud Distance (FPD) of 0.8691 on the Table class, significantly outperforming the baseline (1.0209) and baseline 2 (1.6613), indicating superior generation quality.
• For the Lamp class, the model achieves an FPD of 7.5012 with KDE-based sampling, compared to 36.5025 with standard dataset sampling, demonstrating a substantial improvement in generation quality in low-density regions.
• In the most out-of-distribution region (σ₃), the MSE for dimension regression drops from 52.01 (dataset sampling) to 17.49 (KDE sampling), showing improved generalization in sparse regions.
• The method generalizes to out-of-distribution dimensions: generated shapes for conditioning values beyond the training data range (e.g., outside the dotted shape in Figure 1) remain realistic and semantically meaningful.
• The KDE-based label sampling strategy reduces FPD and MSE across all object classes, especially in high-σ regions, confirming its effectiveness in improving performance in data-sparse areas.
• The model maintains high diversity, as evidenced by low MSE and FPD values across multiple conditioning parameters, even when generating from the same latent vector z.