[Paper Review] Multiple Discrimination and Pairwise CNN for View-based 3D Object Retrieval

Zan Gao, Kaixin Xue+1 more|arXiv (Cornell University)|Feb 27, 2020

Advanced Image and Video Retrieval Techniques42 references4 citations

TL;DR

This paper proposes MDPCNN, a novel pairwise CNN framework for view-based 3D object retrieval that enhances feature discrimination by simultaneously processing multiple batches and views using Slice and Concat layers. It improves performance through hard sample mining via clustering and jointly optimizes contrastive and contrastive-center loss, achieving state-of-the-art results on ModelNet40, MVRED, and NTU60 datasets with mAP improvements up to 87.6%.

ABSTRACT

With the rapid development and wide application of computer, camera device, network and hardware technology, 3D object (or model) retrieval has attracted widespread attention and it has become a hot research topic in the computer vision domain. Deep learning features already available in 3D object retrieval have been proven to be better than the retrieval performance of hand-crafted features. However, most existing networks do not take into account the impact of multi-view image selection on network training, and the use of contrastive loss alone only forcing the same-class samples to be as close as possible. In this work, a novel solution named Multi-view Discrimination and Pairwise CNN (MDPCNN) for 3D object retrieval is proposed to tackle these issues. It can simultaneously input of multiple batches and multiple views by adding the Slice layer and the Concat layer. Furthermore, a highly discriminative network is obtained by training samples that are not easy to be classified by clustering. Lastly, we deploy the contrastive-center loss and contrastive loss as the optimization objective that has better intra-class compactness and inter-class separability. Large-scale experiments show that the proposed MDPCNN can achieve a significant performance over the state-of-the-art algorithms in 3D object retrieval.

Motivation & Objective

• To address the limitations of existing 3D object retrieval methods that rely on hand-crafted features with unstable performance under varying lighting and viewpoints.
• To overcome the lack of consideration for multi-view image selection and insufficient inter-class separability in current deep learning-based retrieval networks.
• To develop an end-to-end, highly discriminative CNN architecture that improves intra-class compactness and inter-class separability for better retrieval accuracy.
• To investigate the impact of multi-batch training, hard sample selection, and combined loss functions on retrieval performance.

Proposed method

• Introduces a novel network architecture, MDPCNN, that uses Slice and Concat layers to enable simultaneous input of multiple batches and multiple views during training.
• Employs a clustering-based sample selection strategy to identify and prioritize hard-to-classify samples for pairwise training, enhancing model discrimination.
• Applies a dual loss function combining contrastive loss and contrastive-center loss to promote intra-class compactness and inter-class separability.
• Utilizes an end-to-end training paradigm that avoids post-processing of deep features, enabling direct optimization of retrieval performance.
• Deploys pairwise training with positive and negative pairs derived from multiple views of the same 3D object, increasing training sample diversity.
• Employs fine-tuning with low-rank Mahalanobis metric learning to further refine feature representations.

Experimental results

Research questions

• RQ1 How does multi-batch and multi-view input via Slice and Concat layers affect the training dynamics and performance of 3D object retrieval networks?
• RQ2 To what extent does hard sample mining based on clustering improve model discrimination in view-based 3D retrieval?
• RQ3 How do the combination of contrastive loss and contrastive-center loss contribute to intra-class compactness and inter-class separability?
• RQ4 What is the relative contribution of batch structure, sample selection, and loss function design to overall retrieval accuracy?

Key findings

• MDPCNN achieves a mean average precision (mAP) of 87.6% on the ModelNet40 dataset, significantly outperforming state-of-the-art methods.
• The ablation study confirms that the multi-batch structure alone improves performance over single-batch training, with further gains from hard sample selection.
• The clustering-based sample selection method enhances model generalization by focusing on the most difficult-to-classify samples, leading to measurable performance gains.
• The joint optimization of contrastive and contrastive-center loss significantly improves feature discriminability, especially on MVRED and NTU60 datasets.
• The network converges rapidly, achieving convergence within 3 epochs on ETH and 5 epochs on MVRED and NTU60, indicating strong training stability.
• The proposed method demonstrates robustness and scalability across diverse benchmark datasets, including ModelNet40, MVRED, and NTU60.