[Paper Review] Do Convnets Learn Correspondence?

Jonathan Long, Ning Zhang+1 more|arXiv (Cornell University)|Nov 4, 2014

Advanced Neural Network ApplicationsComputer Science35 references161 citations

TL;DR

This paper investigates whether convolutional neural networks (ConvNets) learn effective correspondence features despite large receptive fields and weak supervision. Using features from a pretrained ImageNet model, the authors demonstrate that ConvNet activations enable fine-grained localization, outperforming SIFT in keypoint prediction on PASCAL VOC 2011, with a mean PCK accuracy of 42.5% using conv5 features and prior, significantly exceeding SIFT's 28.4%.

ABSTRACT

Convolutional neural nets (convnets) trained from massive labeled datasets have substantially improved the state-of-the-art in image classification and object detection. However, visual understanding requires establishing correspondence on a finer level than object category. Given their large pooling regions and training from whole-image labels, it is not clear that convnets derive their success from an accurate correspondence model which could be used for precise localization. In this paper, we study the effectiveness of convnet activation features for tasks requiring correspondence. We present evidence that convnet features localize at a much finer scale than their receptive field sizes, that they can be used to perform intraclass alignment as well as conventional hand-engineered features, and that they outperform conventional features in keypoint prediction on objects from PASCAL VOC 2011.

Motivation & Objective

• To determine whether ConvNets trained for image classification learn features capable of precise correspondence at the pixel level.
• To evaluate the effectiveness of ConvNet features in intraclass alignment and keypoint prediction tasks.
• To compare the performance of learned ConvNet features against handcrafted SIFT features in correspondence tasks.
• To investigate whether large receptive fields in ConvNets diminish their ability to model fine-grained spatial correspondence.
• To demonstrate that end-to-end learned features can surpass conventional features in localization accuracy for generic object categories.

Proposed method

• Utilized a pretrained ImageNet ConvNet (Caffe reference model) with features extracted from the conv5 layer as the primary representation.
• Performed nonparametric image reconstruction from feature maps to visualize effective receptive fields and spatial specificity.
• Applied SIFT flow with deep features to align images within the same class, using cosine similarity on pool5 features for correspondence.
• Trained linear SVMs with hard negative mining for keypoint detection, using features within a 3×3 receptive field neighborhood as positive samples.
• Combined local detector scores with a spherical Gaussian prior based on nearest neighbor matching in pool5 feature space to improve localization accuracy.
• Optimized the fusion of detector and prior scores using a tradeoff parameter η=0.1 via cross-validation, with final predictions selected as the highest-scoring candidate.

Experimental results

Research questions

• RQ1 Can ConvNet features from a classification network perform as well as handcrafted features in tasks requiring fine-grained correspondence?
• RQ2 To what extent do large receptive fields in ConvNets limit their ability to model precise spatial correspondence?
• RQ3 Do features from deeper layers of a pretrained ConvNet support accurate intraclass alignment across object instances?
• RQ4 Can learned features from a single network architecture outperform SIFT in keypoint prediction for generic object categories?
• RQ5 How does incorporating prior knowledge via nearest neighbor matching improve keypoint localization using ConvNet features?

Key findings

• ConvNet features from the conv5 layer localize at a finer scale than their receptive field size suggests, indicating high spatial specificity.
• The authors demonstrate that conv5 features can perform intraclass alignment as effectively as conventional hand-engineered features.
• In keypoint prediction on PASCAL VOC 2011, conv5 features outperformed SIFT, achieving a mean PCK accuracy of 42.5% when combined with a prior.
• The addition of a spherical Gaussian prior improved performance on both SIFT and conv5 features, with conv5+prior reaching 42.5% mean PCK.
• The results represent the first reported keypoint prediction performance on the PASCAL VOC 2011 dataset using deep features.
• Visualizations show that conv5 features produce more accurate and consistent keypoint predictions than SIFT, especially for challenging categories like horse and potted plant.