[Paper Review] Dominant Set Clustering and Pooling for Multi-View 3D Object Recognition

Chu Wang, Marcello Pelillo+1 more|arXiv (Cornell University)|Jun 4, 2019

Human Pose and Action Recognition21 references37 citations

TL;DR

Introduces a recurrent dominant-set clustering and pooling layer to fuse multi-view CNN features, boosting 3D object recognition on ModelNet40 to state-of-the-art levels.

ABSTRACT

View based strategies for 3D object recognition have proven to be very successful. The state-of-the-art methods now achieve over 90% correct category level recognition performance on appearance images. We improve upon these methods by introducing a view clustering and pooling layer based on dominant sets. The key idea is to pool information from views which are similar and thus belong to the same cluster. The pooled feature vectors are then fed as inputs to the same layer, in a recurrent fashion. This recurrent clustering and pooling module, when inserted in an off-the-shelf pretrained CNN, boosts performance for multi-view 3D object recognition, achieving a new state of the art test set recognition accuracy of 93.8% on the ModelNet 40 database. We also explore a fast approximate learning strategy for our cluster-pooling CNN, which, while sacrificing end-to-end learning, greatly improves its training efficiency with only a slight reduction of recognition accuracy to 93.3%. Our implementation is available at https://github.com/fate3439/dscnn.

Motivation & Objective

• Motivate improving multi-view 3D object recognition beyond winner-take-all pooling.
• Develop a view clustering and pooling layer based on dominant sets to fuse similar views.
• Enable end-to-end or fast training by integrating the layer with pretrained CNNs for improved accuracy.

Proposed method

• Construct a view similarity graph where nodes are view feature vectors (relu outputs) and edge weights are inner products of CNN features.
• Extract dominant sets using a replicator dynamics-based algorithm to form coherent view clusters.
• Pool within each dominant set (max or average) and feed the results back for recurrent refinement until clusters stabilize.
• Attach the recurrent clustering and pooling layer after relu6/relu7 in a pretrained VGG-M network and perform full-stride pooling to produce a unified multi-view feature vector.
• Optionally train end-to-end by fine-tuning CNN layers around the recurrent layer; fast training variants forego end-to-end learning for efficiency.
• Explore additional feature modalities (depth, surface normals) alongside RGB to improve accuracy.

Experimental results

Research questions

• RQ1 Does dominant-set based clustering improve multi-view feature fusion over max pooling or pairwise view methods?
• RQ2 Can the recurrent clustering and pooling layer be inserted into a pretrained CNN to yield end-to-end trainable networks for multi-view recognition?
• RQ3 How does the number of views and inclusion of additional modalities affect recognition performance on ModelNet40?
• RQ4 What is the trade-off between training efficiency and accuracy when using fast training versus end-to-end training?

Key findings

• Achieves state-of-the-art test accuracy on ModelNet40 with 93.8% full-set accuracy using RGB views and the recurrent clustering and pooling layer.
• End-to-end training yields higher accuracy (92.2% full-set) than fast training (91.9% full-set) for RGB views.
• Incorporating depth and surface normals with RGB raises full-set accuracy to 93.3% in fast training and 93.8% in end-to-end training variants.
• Increasing the number of views generally improves performance for the proposed method, with consistent gains over MVCNN across view counts.
• Fast training significantly reduces per-epoch cost with only slight reductions in accuracy compared to end-to-end training.
• The method outperforms state-of-the-art MVCNN and related approaches on ModelNet40 under comparable settings.