[Paper Review] Do Deep Convolutional Nets Really Need to be Deep and Convolutional?

Gregor Urban, Krzysztof J. Geras+7 more|PolyPublie (École Polytechnique de Montréal)|Mar 17, 2016

Advanced Neural Network ApplicationsComputer Science22 references107 citations

TL;DR

The paper empirically shows that, for CIFAR-10, shallow models cannot match deep convolutional nets even with distillation and hyperparameter optimization; multiple convolutional layers are essential for high accuracy within the same parameter budget.

ABSTRACT

Yes, they do. This paper provides the first empirical demonstration that deep convolutional models really need to be both deep and convolutional, even when trained with methods such as distillation that allow small or shallow models of high accuracy to be trained. Although previous research showed that shallow feed-forward nets sometimes can learn the complex functions previously learned by deep nets while using the same number of parameters as the deep models they mimic, in this paper we demonstrate that the same methods cannot be used to train accurate models on CIFAR-10 unless the student models contain multiple layers of convolution. Although the student models do not have to be as deep as the teacher model they mimic, the students need multiple convolutional layers to learn functions of comparable accuracy as the deep convolutional teacher.

Motivation & Objective

• Motivate whether shallow nets can match deep convolutional CNNs under equal parameter budgets on CIFAR-10.
• Assess the effectiveness of distillation (teacher-student) with Bayesian hyperparameter optimization for shallow CNNs.
• Compare performance of shallow models trained on hard targets vs. soft targets from an ensemble of deep teachers.
• Quantify how many convolutional layers are needed in shallow models to approach deep-model accuracy.

Proposed method

• Train state-of-the-art deep convolutional teacher ensemble (16 CNNs) on CIFAR-10 with extensive augmentation.
• Use distillation to train shallow student models trained to mimic the ensemble logits (soft targets) rather than hard one-hot labels.
• Apply linear bottlenecks to shallow models with 0–1 convolutional layers to speed up learning.
• Perform Bayesian hyperparameter optimization (Gaussian processes via Spearmint) over learning rate, momentum, weight scales, and network widths.
• Augment data with HSV-based shifts and random crops/mirrors to create a large transfer set for model compression.
• Evaluate shallow students across architectures (1–4 conv layers, various parameter budgets) vs. the deep teacher ensemble.

Experimental results

Research questions

• RQ1 Can shallow networks with comparable parameter counts to deep CNNs achieve CIFAR-10 accuracy close to deep models when trained via distillation?
• RQ2 Does distillation with soft targets enable shallow architectures to bridge the gap to deep conv nets on CIFAR-10?
• RQ3 How many convolutional layers are necessary for shallow models to achieve competitive performance under a fixed parameter budget?
• RQ4 What is the role of data augmentation and hyperparameter optimization in training effective shallow mimics?

Key findings

• Shallow models cannot match deep convolutional nets within the same parameter budget, even with distillation.
• An ensemble of 16 deep CNNs achieves 93.8% validation accuracy (94.0% on the validation set; 93.8% on the final test set).
• Shallow student CNNs require multiple convolutional layers (3–4) to reach high CIFAR-10 accuracy; with 1–2 conv layers they lag behind the deep models.
• Distillation significantly improves shallow models over hard-target training, especially for very shallow architectures (e.g., 1 conv layer).
• Shallow fully-connected MLPs without convolution perform markedly worse (e.g., ~70%+ with distillation vs. >90% with CNNs).
• Even with hyperparameter optimization and distillation, there remains a sizable “convolutional gap” that deeper conv nets close but do not fully bridge for shallow students.
• The best single-layer MLP achieved 70.2% accuracy, illustrating the limits of non-convolutional shallow models on CIFAR-10, even with distillation.