[Paper Review] FusionNet: 3D Object Classification Using Multiple Data Representations

Vishakh Hegde, Reza Bosagh Zadeh|arXiv (Cornell University)|Jul 19, 2016

Image Processing and 3D Reconstruction22 references168 citations

TL;DR

FusionNet combines volumetric CNNs on voxel data with multi-view CNNs on 2D projections to improve 3D object classification, achieving state-of-the-art results on ModelNet40/ModelNet10.

ABSTRACT

High-quality 3D object recognition is an important component of many vision and robotics systems. We tackle the object recognition problem using two data representations, to achieve leading results on the Princeton ModelNet challenge. The two representations: 1. Volumetric representation: the 3D object is discretized spatially as binary voxels - $1$ if the voxel is occupied and $0$ otherwise. 2. Pixel representation: the 3D object is represented as a set of projected 2D pixel images. Current leading submissions to the ModelNet Challenge use Convolutional Neural Networks (CNNs) on pixel representations. However, we diverge from this trend and additionally, use Volumetric CNNs to bridge the gap between the efficiency of the above two representations. We combine both representations and exploit them to learn new features, which yield a significantly better classifier than using either of the representations in isolation. To do this, we introduce new Volumetric CNN (V-CNN) architectures.

Motivation & Objective

• Motivate robust 3D object classification using complementary data representations.
• Develop volumetric CNNs (V-CNNs) that learn long-range 3D features with fewer parameters.
• Leverage multi-view 2D projections with transfer learning to exploit pre-trained networks.
• Fuse predictions from heterogeneous networks to surpass single-representation performance.

Proposed method

• Introduce two new volumetric CNNs for voxel data with orientations augmentation.
• Use 60 orientations per object to learn long-range spatial correlations with shared weights across orientations.
• In V-CNN II, adopt an inception-style module to capture multi-scale voxel features.
• Apply data augmentation by random vertex displacements to increase robustness.
• Combine voxel-based networks and MV-CNN via a linear score fusion at the final layer to form FusionNet.

Experimental results

Research questions

• RQ1 Can combining voxel-based and multi-view representations improve 3D object classification beyond what each representation achieves alone?
• RQ2 Do volumetric networks with rotated/oriented inputs complement 2D projection-based networks for ModelNet benchmarks?
• RQ3 What is the effect of transfer learning from ImageNet on MV-CNN performance for 3D shape classification?

Key findings

• V-CNN I and V-CNN II perform comparably but learn complementary features; combining them yields better results.
• MV-CNN based on AlexNet with ImageNet pre-training outperforms individual volumetric CNNs.
• FusionNet (combining V-CNN I, V-CNN II, and MV-CNN) achieves the best results on ModelNet10 and ModelNet40.
• Fine-tuning AlexNet-based MV-CNN provides notable gains; combining with voxel networks yields additional improvements.
• FusionNet styles show 60-view voxel ensembles plus 20-view MV-CNN surpass single-representation methods.