[Paper Review] RealPoint3D: Point Cloud Generation from a Single Image with Complex Background

Yan Xia, Yang Zhang+4 more|arXiv (Cornell University)|Sep 8, 2018

3D Shape Modeling and Analysis19 references4 citations

TL;DR

RealPoint3D proposes a novel framework for 3D point cloud generation from single real images with complex backgrounds and arbitrary viewpoints by integrating prior 3D shape knowledge through a retrieved shape model. By combining a 2D image with a nearest-neighbor 3D shape from a pre-built database, the method achieves state-of-the-art performance in fine-grained 3D reconstruction, outperforming existing volumetric and point-based methods on both synthetic and real-world data.

ABSTRACT

3D point cloud generation by the deep neural network from a single image has been attracting more and more researchers' attention. However, recently-proposed methods require the objects be captured with relatively clean backgrounds, fixed viewpoint, while this highly limits its application in the real environment. To overcome these drawbacks, we proposed to integrate the prior 3D shape knowledge into the network to guide the 3D generation. By taking additional 3D information, the proposed network can handle the 3D object generation from a single real image captured from any viewpoint and complex background. Specifically, giving a query image, we retrieve the nearest shape model from a pre-prepared 3D model database. Then, the image together with the retrieved shape model is fed into the proposed network to generate the fine-grained 3D point cloud. The effectiveness of our proposed framework has been verified on different kinds of datasets. Experimental results show that the proposed framework achieves state-of-the-art accuracy compared to other volumetric-based and point set generation methods. Furthermore, the proposed framework works well for real images in complex backgrounds with various view angles.

Motivation & Objective

• To address the limitation of existing 3D generation methods that fail on real-world images with complex backgrounds and variable viewpoints.
• To improve 3D reconstruction fidelity by incorporating prior 3D shape knowledge from a pre-existing 3D model database.
• To enable end-to-end 3D point cloud generation from a single image without requiring object masks or clean backgrounds.
• To achieve state-of-the-art performance in 3D reconstruction on real-world datasets such as ObjectNet3D.

Proposed method

• The framework first retrieves the nearest 3D shape from a pre-built 3D model database using image features extracted from the input image.
• Image features are compared against stored features from 2D renderings of 3D models at multiple viewpoints to find the most similar shape.
• The retrieved 3D shape and the input image are jointly fed into a deep neural network to generate a complete, fine-grained 3D point cloud.
• The network architecture includes a 3D encoder branch to process the retrieved shape and a 2D image encoder to process the input image, with feature fusion for generation.
• The method is trained end-to-end to reconstruct detailed 3D geometry, including occluded parts, by leveraging the retrieved shape as inductive bias.
• The framework is designed to be robust to viewpoint changes, occlusions, and complex backgrounds by relying on sparse, distinctive features for retrieval.

Experimental results

Research questions

• RQ1 Can prior 3D shape knowledge improve 3D point cloud generation from single real images with complex backgrounds?
• RQ2 How does integrating a retrieved 3D shape from a database affect reconstruction quality compared to mask-free or mask-reliant methods?
• RQ3 To what extent does the retrieval-based approach outperform existing volumetric and point-based generative models on real-world images?
• RQ4 How does the method perform on objects with complex structures (e.g., sofas, chairs) under occlusion and viewpoint variation?
• RQ5 Can the framework generalize to real-world images without requiring object segmentation masks?

Key findings

• RealPoint3D achieved state-of-the-art performance on both synthetic and real-world datasets, with an IoU score of 0.63 on the sofa category, outperforming PSGN and OGN.
• On the ObjectNet3D dataset, RealPoint3D successfully reconstructed fine-grained details like bench legs that PSGN missed even with object masks.
• The method demonstrated robustness to complex backgrounds and viewpoint variations, with significant performance gains over baseline methods in challenging cases.
• The ablation study showed that the full model (RP3D-v2) with the 3D encoder branch outperformed the variant without it (RP3D-v1), confirming the benefit of prior shape knowledge.
• Inference time was approximately 0.1 seconds per image on CPU, making it significantly faster than OGN (1.6s) and comparable to PSGN.
• The retrieval stage was found to be robust to occlusions and viewpoint changes, enabling reliable shape matching even when input images were noisy or cluttered.