[Paper Review] High-Performance Neural Networks for Visual Object Classification

Dan Cireşan, Ueli Meier+3 more|arXiv (Cornell University)|Feb 1, 2011

Advanced Neural Network Applications22 references220 citations

TL;DR

This paper presents a high-performance, fully parameterizable GPU implementation of deep convolutional neural networks (CNNs) for visual object classification. Using end-to-end back-propagation training on GPUs, the authors achieve state-of-the-art error rates of 0.35% on MNIST, 2.53% on NORB, and 19.51% on CIFAR10, demonstrating that deep, sparsely connected CNNs trained with online stochastic gradient descent outperform shallower models and require no unsupervised pretraining.

ABSTRACT

We present a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants. Our feature extractors are neither carefully designed nor pre-wired, but rather learned in a supervised way. Our deep hierarchical architectures achieve the best published results on benchmarks for object classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple back-propagation perform better than more shallow ones. Learning is surprisingly rapid. NORB is completely trained within five epochs. Test error rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs, respectively.

Motivation & Objective

• To develop a fast, flexible, and fully parameterizable GPU-based implementation of deep convolutional neural networks for visual object classification.
• To investigate the impact of network depth, width, and architectural components (e.g., max-pooling, pre-processing) on classification performance across standard benchmarks.
• To demonstrate that deep CNNs trained via simple back-propagation can outperform shallower architectures without unsupervised pretraining.
• To enable rapid exploration of large hyperparameter spaces by drastically reducing training time compared to CPU-based methods.

Proposed method

• The method employs a fully parameterizable CNN architecture with alternating convolutional and max-pooling layers, allowing flexible configuration of kernel sizes, skipping factors, and number of feature maps.
• Each convolutional layer uses shared weights across receptive fields and applies filters with specified kernel size and stride, computed via the formula: $ M^{n}_{x} = \frac{M^{n-1}_{x} - K^{n}_{x}}{S^{n}_{x} + 1} + 1 $.
• Max-pooling layers downsample feature maps by taking the maximum activation over non-overlapping $ K_{x} \times K_{y} $ regions, enhancing position invariance.
• The network uses online stochastic gradient descent with learning rate decay (multiplied by 0.993 per epoch) for end-to-end training.
• An optional image processing layer applies fixed filters (e.g., Sobel, Scharr, or contrast-extracting filters) to enhance input representation.
• The implementation is optimized for GPU execution, achieving 10–60× speedup over CPU-optimized versions, enabling rapid training of large networks.

Experimental results

Research questions

• RQ1 Does training deep, fully connected CNNs via end-to-end back-propagation yield superior performance compared to shallower networks on standard benchmarks?
• RQ2 How do architectural choices such as max-pooling, kernel size, and number of feature maps affect classification accuracy and generalization?
• RQ3 Can data augmentation via image translations improve generalization in CNNs, even with inherent translation invariance?
• RQ4 Does pre-processing with fixed filters (e.g., edge detectors) improve performance, or is it detrimental compared to raw input?
• RQ5 To what extent does GPU acceleration enable efficient exploration of large hyperparameter spaces in CNN design?

Key findings

• The proposed GPU-accelerated CNN achieved a test error rate of 0.35% on the MNIST benchmark, setting a new state-of-the-art at the time.
• On the NORB dataset, the network achieved a test error rate of 2.53%, outperforming previous methods.
• For CIFAR10, the best network achieved a test error rate of 19.51%, surpassing prior SOTA results of 20.40% and 25.50%.
• Increasing the number of feature maps per layer from 100 to 300 yielded the lowest error rate (19.51%), with further increases to 400 maps showing no improvement.
• Training with image translations reduced the CIFAR10 error rate from 28.87% (no augmentation) to 20.26%, demonstrating the benefit of data augmentation.
• The GPU implementation was more than 60 times faster than a CPU-optimized version for large networks, reducing training time from days to hours.