[Paper Review] Analysis and Optimization of Convolutional Neural Network Architectures

Martin Thoma|arXiv (Cornell University)|Jul 31, 2017

Advanced Neural Network Applications56 references57 citations

TL;DR

This master thesis analyzes CNN analysis and topology construction, introduces a confusion-matrix based visualization, evaluates hierarchical classifiers, and reports on techniques that boost accuracy, including data augmentation and small batch sizes, culminating in a compact state-of-the-art model across multiple datasets.

ABSTRACT

Convolutional Neural Networks (CNNs) dominate various computer vision tasks since Alex Krizhevsky showed that they can be trained effectively and reduced the top-5 error from 26.2 % to 15.3 % on the ImageNet large scale visual recognition challenge. Many aspects of CNNs are examined in various publications, but literature about the analysis and construction of neural network architectures is rare. This work is one step to close this gap. A comprehensive overview over existing techniques for CNN analysis and topology construction is provided. A novel way to visualize classification errors with confusion matrices was developed. Based on this method, hierarchical classifiers are described and evaluated. Additionally, some results are confirmed and quantified for CIFAR-100. For example, the positive impact of smaller batch sizes, averaging ensembles, data augmentation and test-time transformations on the accuracy. Other results, such as the positive impact of learned color transformation on the test accuracy could not be confirmed. A model which has only one million learned parameters for an input size of 32x32x3 and 100 classes and which beats the state of the art on the benchmark dataset Asirra, GTSRB, HASYv2 and STL-10 was developed.

Motivation & Objective

• Provide a comprehensive overview of CNN analysis techniques and topology learning methods.
• Introduce and evaluate a confusion-matrix based visualization for classification errors.
• Develop and assess hierarchical classifiers (CMO) and their impact on performance.
• Validate known accuracy-enhancing practices (data augmentation, small batch sizes, test-time transforms) on CIFAR-100 and other benchmarks.
• Present a compact CNN model (<1M parameters) achieving strong results on several datasets (Asirra, GTSRB, HASYv2, STL-10).

Proposed method

• Review existing CNN analysis and topology construction methods.
• Develop a novel confusion matrix based visualization for classification errors.
• Describe and evaluate hierarchical classifiers built from confusion matrix insights.
• Experimentally validate techniques such as smaller batch sizes, averaging ensembles, data augmentation, and test-time transformations on CIFAR-100.
• Investigate learned color space transformation and other potential accuracy boosters.
• Construct a compact CNN model with ~1 million parameters that benchmarks competitively on multiple datasets.

Experimental results

Research questions

• RQ1 How can visualization of classification errors inform CNN architecture design?
• RQ2 Do hierarchical classifiers improve overall accuracy and robustness?
• RQ3 Which training techniques (batch size, data augmentation, test-time transforms) reliably improve accuracy across datasets?
• RQ4 Is it possible to design a compact CNN that achieves state-of-the-art results on diverse benchmarks?

Key findings

• Confusion-matrix based visualization was developed to analyze CNN errors.
• Hierarchical classifiers derived from the method were described and evaluated.
• Smaller batch sizes, averaging ensembles, data augmentation, and test-time transformations positively impacted accuracy on CIFAR-100.
• Learned color space transformation did not show confirmed gains in test accuracy.
• A model with under 1 million parameters achieved competitive results and set new state-of-the-art on Asirra, GTSRB, HASYv2, and STL-10 benchmarks.