[Paper Review] Mask-CNN: Localizing Parts and Selecting Descriptors for Fine-Grained Image Recognition

Xiu-Shen Wei, Chen-Wei Xie+1 more|ArXiv.org|May 23, 2016

Advanced Neural Network ApplicationsComputer Science14 references103 citations

TL;DR

Mask-CNN introduces an end-to-end four-stream CNN that uses FCN-predicted object/part masks to select and pool deep descriptors for fine-grained recognition, achieving state-of-the-art accuracy on CUB-200-2011 with a compact model.

ABSTRACT

Fine-grained image recognition is a challenging computer vision problem, due to the small inter-class variations caused by highly similar subordinate categories, and the large intra-class variations in poses, scales and rotations. In this paper, we propose a novel end-to-end Mask-CNN model without the fully connected layers for fine-grained recognition. Based on the part annotations of fine-grained images, the proposed model consists of a fully convolutional network to both locate the discriminative parts (e.g., head and torso), and more importantly generate object/part masks for selecting useful and meaningful convolutional descriptors. After that, a four-stream Mask-CNN model is built for aggregating the selected object- and part-level descriptors simultaneously. The proposed Mask-CNN model has the smallest number of parameters, lowest feature dimensionality and highest recognition accuracy when compared with state-of-the-arts fine-grained approaches.

Motivation & Objective

• Motivate fine-grained recognition which requires discriminating subtle inter-class differences.
• Propose an end-to-endMask-CNN without fully connected layers that uses part-based masks to select descriptors.
• Leverage a four-stream architecture (image, head, torso, object) to jointly model object- and part-level information.
• Demonstrate high accuracy and efficiency compared to state-of-the-art methods on CUB-200-2011.

Proposed method

• Use FCN to generate object/part masks from part annotations, treating part localization as a three-class segmentation task.
• Discard fully connected layers and keep conv layers to obtain 512-d deep descriptors per spatial location.
• Resize head/torso/object masks to 7x7 and apply as binary selectors to keep only object-relevant descriptors.
• Compute per-stream features by average and max pooling of selected descriptors, followed by L2 normalization.
• Concatenate four 1024-d features (after pooling and normalization) into a 4096-d representation per stream and train a 200-way classifier end-to-end.
• Optionally enrich features by extracting activations from relu5_2 in addition to pool5 and combining for an 8192-d representation with SVD whitening to 4096-d.

Experimental results

Research questions

• RQ1 Can part localization via FCN masks enable effective descriptor selection for fine-grained recognition without testing-time supervision?
• RQ2 Does a four-stream architecture (image, head, torso, object) outperform single-stream or partially supervised baselines on CUB-200-2011?
• RQ3 What are the effects of descriptor selection versus standard pooling on recognition accuracy?
• RQ4 How does Mask-CNN perform in terms of model size and feature dimensionality compared to state-of-the-art methods?

Key findings

• Achieves 83.1% accuracy with 224x224 input and 4-stream M-CNN (no FCs in streams).
• Improves to 85.2% with 448x448 inputs for all streams; 85.4% with 4-stream M-CNN + (448) when combining pool5 and relu5_2 features.
• With SVD whitening to 4096-d, accuracy reaches 85.5%.
• Head localization: 84.62% PCP; Torso localization: 89.83% PCP (using 50% IOU threshold).
• Object segmentation mean IU on test set: 72.41%.
• Four-stream M-CNN has fewer parameters and lower feature dimensionality than competing methods (e.g., 60.49M parameters, 8,192-d features for 4-stream M-CNN +; AlexNet variant has 9.74M parameters and 2,048-d features).
• On CUB-200-2011, Mask-CNN achieves 85.5% classification accuracy, higher than prior state-of-the-art methods that do not require testing-time bounding boxes or parts.