[Paper Review] Return of the Devil in the Details: Delving Deep into Convolutional Nets

Ken Chatfield, Karen Simonyan+2 more|arXiv (Cornell University)|May 14, 2014

Advanced Neural Network Applications31 references666 citations

TL;DR

This paper conducts a rigorous empirical comparison of deep convolutional neural networks (CNNs) and shallow visual representations like the Improved Fisher Vector (IFV), demonstrating that data augmentation significantly boosts IFV performance and that CNN features—especially when fine-tuned—achieve state-of-the-art results on PASCAL VOC and Caltech datasets with minimal architectural complexity.

ABSTRACT

The latest generation of Convolutional Neural Networks (CNN) have achieved impressive results in challenging benchmarks on image recognition and object detection, significantly raising the interest of the community in these methods. Nevertheless, it is still unclear how different CNN methods compare with each other and with previous state-of-the-art shallow representations such as the Bag-of-Visual-Words and the Improved Fisher Vector. This paper conducts a rigorous evaluation of these new techniques, exploring different deep architectures and comparing them on a common ground, identifying and disclosing important implementation details. We identify several useful properties of CNN-based representations, including the fact that the dimensionality of the CNN output layer can be reduced significantly without having an adverse effect on performance. We also identify aspects of deep and shallow methods that can be successfully shared. In particular, we show that the data augmentation techniques commonly applied to CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost. Source code and models to reproduce the experiments in the paper is made publicly available.

Motivation & Objective

• To rigorously evaluate and compare deep CNN-based representations with shallow handcrafted features like IFV on standard benchmarks.
• To identify key implementation details—especially data augmentation—that significantly impact performance across both deep and shallow methods.
• To determine whether pre-trained CNN features can achieve state-of-the-art performance without extensive retraining or complex pipelines.
• To provide a common evaluation framework and publicly released code/models to enable reproducibility and future benchmarking in image representation research.

Proposed method

• Employs three scenarios: shallow IFV encoding, pre-trained deep CNN features, and fine-tuned deep CNN features on target datasets.
• Applies data augmentation (e.g., random crops, color jittering) uniformly across both shallow and deep methods to assess its impact.
• Uses feature normalization and dimensionality reduction techniques to optimize performance and efficiency.
• Trains linear SVMs with hinge loss on extracted features for classification, ensuring fair comparison across methods.
• Employs standard datasets: ILSVRC-2012, PASCAL VOC-2007/2012, Caltech-101, and Caltech-256 for evaluation.
• Reimplements and validates CNN architectures (CNN-F, CNN-M, CNN-S) using consistent training and evaluation protocols to ensure reproducibility.

Experimental results

Research questions

• RQ1 How do different deep CNN architectures compare to each other and to shallow representations like IFV when evaluated under identical conditions?
• RQ2 To what extent does data augmentation improve performance in both deep and shallow feature learning frameworks?
• RQ3 Can dimensionality reduction of deep CNN features be applied without degrading performance, and what are the optimal settings?
• RQ4 Does fine-tuning pre-trained CNN features on target datasets yield significant performance gains, and how does it compare to end-to-end training?
• RQ5 Can the performance gap between deep and shallow methods be reduced by applying deep learning best practices (e.g., data augmentation) to shallow models?

Key findings

• Data augmentation significantly improves IFV performance, raising its top result to 68.02% mAP on PASCAL VOC-2007, demonstrating that such techniques are beneficial beyond CNNs.
• Fine-tuned CNN-S features achieve 82.42% mAP on PASCAL VOC-2007, setting a new state of the art for models pre-trained only on ILSVRC-2012.
• The CNN-M-2048 II feature representation is 42× more compact (2K vs. 84K dimensions) and 50× faster to compute than the best-performing IFV II.
• Pre-trained CNN features on ILSVRC-2012 generalize well to other datasets, with performance on ILSVRC being a strong predictor of performance on PASCAL VOC and Caltech.
• Fine-tuning with ranking loss on VOC-2012 yields a marginal improvement over classification loss, aligning with the dataset’s evaluation metric.
• Despite improvements, deep CNNs still outperform even augmented shallow methods by a large margin, confirming the superiority of hierarchical, learned representations.