[Paper Review] Learning Discriminative 3D Shape Representations by View Discerning Networks

Biao Leng, Cheng Zhang+3 more|arXiv (Cornell University)|Aug 11, 2018

3D Shape Modeling and Analysis26 references4 citations

TL;DR

This paper proposes View Discerning Network (VDN), a deep learning framework that enhances 3D shape recognition by dynamically assessing view quality via a Score Generation Unit, which assigns attention weights to multi-view features. The method improves recognition accuracy and robustness under occlusion and clutter, outperforming state-of-the-art methods on ModelNet and ShapeNet Core55 with faster convergence and reduced computational cost.

ABSTRACT

In view-based 3D shape recognition, extracting discriminative visual representation of 3D shapes from projected images is considered the core problem. Projections with low discriminative ability can adversely influence the final 3D shape representation. Especially under the real situations with background clutter and object occlusion, the adverse effect is even more severe. To resolve this problem, we propose a novel deep neural network, View Discerning Network, which learns to judge the quality of views and adjust their contributions to the representation of shapes. In this network, a Score Generation Unit is devised to evaluate the quality of each projected image with score vectors. These score vectors are used to weight the image features and the weighted features perform much better than original features in 3D shape recognition task. In particular, we introduce two structures of Score Generation Unit, Channel-wise Score Unit and Part-wise Score Unit, to assess the quality of feature maps from different perspectives. Our network aggregates features and scores in an end-to-end framework, so that final shape descriptors are directly obtained from its output. Our experiments on ModelNet and ShapeNet Core55 show that View Discerning Network outperforms the state-of-the-arts in terms of the retrieval task, with excellent robustness against background clutter and object occlusion.

Motivation & Objective

• To address the negative impact of low-quality views in view-based 3D shape recognition, especially under real-world conditions like background clutter and object occlusion.
• To develop a data-driven, end-to-end trainable network that can automatically assess and prioritize informative views during feature aggregation.
• To improve the efficiency and robustness of multi-view 3D shape representation learning by replacing standard max-pooling with quality-aware feature weighting.
• To reduce computational cost by enabling effective recognition with fewer high-quality views rather than uniformly sampling many views.

Proposed method

• The network introduces a Score Generation Unit that evaluates the quality of each projected 2D view using score vectors derived from feature maps.
• Two variants of the Score Generation Unit are proposed: Channel-wise Score Unit and Part-wise Score Unit, which assess feature maps from different perspectives.
• The score vectors are used to weight the CNN-embedded features of each view, producing a weighted feature representation that emphasizes informative views.
• The network aggregates these weighted features in an end-to-end fashion, directly producing a discriminative 3D shape descriptor.
• The Score Generation Unit is trained jointly with the main network, enabling backpropagation to refine both feature extraction and view quality scoring.
• The method is evaluated using a modified ShapeNet Core55 dataset with occluded images to simulate real-world challenges.

Experimental results

Research questions

• RQ1 How does the quality of individual views affect the performance of multi-view 3D shape recognition under real-world conditions like occlusion and clutter?
• RQ2 Can a learnable, attention-like mechanism improve feature aggregation by selectively emphasizing high-quality views?
• RQ3 To what extent can a smaller number of high-quality views replace uniformly sampled views to maintain or improve recognition accuracy?
• RQ4 How does the proposed score-based weighting mechanism compare to standard max-pooling in terms of robustness and convergence speed?
• RQ5 What are the failure modes of the view quality scoring mechanism, particularly for complex or ambiguous shapes?

Key findings

• Using only 10 high-quality views achieved better recognition performance than using 64 uniformly sampled views, demonstrating the critical role of view quality.
• The View Discerning Network outperformed state-of-the-art methods on both ModelNet and ShapeNet Core55 benchmarks, especially under occlusion and clutter.
• The loss curve during training showed faster and more stable convergence when using the Score Generation Unit, confirming improved training efficiency.
• The method reduced computational cost by enabling effective recognition with fewer views while maintaining high accuracy.
• The Part-wise Score Unit failed to recognize good views for complex plant models and guitar models placed on larger objects, due to content-oblivious block division and foreground-background confusion.
• Classification accuracy for plant models dropped to around 45% on the occluded ModelNet40 dataset, indicating a key limitation in handling high-complexity, cluttered scenes.