[Paper Review] Multi-view Convolutional Neural Networks for 3D Shape Recognition

Hang Su, Subhransu Maji+2 more|arXiv (Cornell University)|May 5, 2015

3D Surveying and Cultural Heritage30 references279 citations

TL;DR

This paper proposes a multi-view convolutional neural network (MVCNN) that recognizes 3D shapes by treating multiple 2D renderings as inputs, outperforming 3D representation-based methods. By fusing features from multiple views into a compact descriptor, MVCNN achieves state-of-the-art performance in 3D shape classification and sketch-based retrieval, with a 77% to 85% accuracy gain over prior 3D methods using only a single view.

ABSTRACT

A longstanding question in computer vision concerns the representation of 3D shapes for recognition: should 3D shapes be represented with descriptors operating on their native 3D formats, such as voxel grid or polygon mesh, or can they be effectively represented with view-based descriptors? We address this question in the context of learning to recognize 3D shapes from a collection of their rendered views on 2D images. We first present a standard CNN architecture trained to recognize the shapes' rendered views independently of each other, and show that a 3D shape can be recognized even from a single view at an accuracy far higher than using state-of-the-art 3D shape descriptors. Recognition rates further increase when multiple views of the shapes are provided. In addition, we present a novel CNN architecture that combines information from multiple views of a 3D shape into a single and compact shape descriptor offering even better recognition performance. The same architecture can be applied to accurately recognize human hand-drawn sketches of shapes. We conclude that a collection of 2D views can be highly informative for 3D shape recognition and is amenable to emerging CNN architectures and their derivatives.

Motivation & Objective

• To investigate whether 2D image-based representations can outperform direct 3D representation learning for 3D shape recognition.
• To develop a deep learning architecture that effectively combines multiple 2D views of a 3D shape into a single, compact, and discriminative descriptor.
• To enable accurate 3D shape retrieval using hand-drawn sketches by leveraging learned 2D representations.
• To explore the feasibility of using pre-trained CNNs on ImageNet to improve generalization on 3D shape recognition tasks.

Proposed method

• The method uses a two-stage CNN architecture: first, a CNN processes each 2D view independently to extract view-specific features.
• Second, features from multiple views are pooled across views and fed into a second CNN to generate a compact, unified shape descriptor.
• The network is trained using cross-entropy loss on 3D shape categories, with data augmentation via view jittering during training.
• The model leverages pre-trained ImageNet weights for feature initialization, followed by fine-tuning on 3D shape datasets.
• Saliency maps are generated by back-propagating gradients to identify the most informative views and regions within views.
• For sketch-based retrieval, the same descriptor is used to match hand-drawn sketches to 3D shapes without additional fine-tuning.

Experimental results

Research questions

• RQ1 Can 2D image-based representations of 3D shapes outperform direct 3D representation learning in shape recognition?
• RQ2 How effective is a multi-view CNN architecture in combining information from multiple 2D projections into a compact, discriminative shape descriptor?
• RQ3 Can the learned descriptor support accurate 3D shape retrieval using hand-drawn sketches?
• RQ4 What is the impact of view selection and view diversity on recognition performance?
• RQ5 Can pre-trained CNNs on ImageNet be effectively fine-tuned for 3D shape recognition using only 2D renderings?

Key findings

• Using only a single 2D view, the MVCNN achieves 85% top-1 accuracy on ModelNet40, an 8% improvement over the best prior 3D representation-based method.
• With 12 views, the model achieves 86.4% top-1 accuracy on ModelNet40, significantly outperforming previous 3D CNNs.
• The model achieves 36.1% mAP on sketch-based 3D shape retrieval using a pre-trained VGG-M network without any fine-tuning on sketches.
• Saliency maps identify the most informative views and discriminative parts, such as the front of a bench or the faucet of a bathtub.
• The multi-view CNN outperforms standard jittering-based data augmentation on sketch recognition benchmarks, demonstrating its effectiveness beyond 3D shape recognition.
• The model generalizes well to real-world 3D objects and video-based reconstruction, suggesting broader applicability beyond synthetic meshes.