[Paper Review] Do Deep Nets Really Need to be Deep?

Jimmy Ba, Rich Caruana|arXiv (Cornell University)|Dec 21, 2013

Generative Adversarial Networks and Image Synthesis21 references1,480 citations

TL;DR

This paper demonstrates that shallow feed-forward neural networks can achieve performance comparable to deep convolutional networks on TIMIT phoneme recognition and CIFAR-10 image classification by training them to mimic deep models via knowledge distillation. Using logits from a pre-trained deep net as targets, the shallow student networks match or exceed the accuracy of the original deep models despite having fewer or comparable parameters, suggesting depth is not inherently necessary for high performance.

ABSTRACT

Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this extended abstract, we show that shallow feed-forward networks can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow neural nets can learn these deep functions using a total number of parameters similar to the original deep model. We evaluate our method on the TIMIT phoneme recognition task and are able to train shallow fully-connected nets that perform similarly to complex, well-engineered, deep convolutional architectures. Our success in training shallow neural nets to mimic deeper models suggests that there probably exist better algorithms for training shallow feed-forward nets than those currently available.

Motivation & Objective

• To investigate whether deep neural networks truly require depth to achieve state-of-the-art performance on vision and speech tasks.
• To determine if shallow networks can learn complex functions previously attributed to deep architectures.
• To evaluate whether model compression via distillation enables shallow nets to match deep net accuracy with comparable parameter counts.
• To assess whether the performance gain of deep nets stems from architectural depth or from better training procedures and inductive biases.

Proposed method

• Train a deep neural network (teacher) on original labeled data using standard training with cross-entropy loss.
• Use the trained deep net to generate logits (pre-softmax values) on unlabeled data to create synthetic training targets.
• Train a shallow fully-connected feedforward network (student) to regress the logits from the deep net using L2 loss.
• Employ knowledge distillation by training the student on the soft labels (logits) rather than hard labels, enabling better generalization and finer-grained learning.
• Compare student performance against the original deep net and other shallow baselines on test sets.
• Use ensemble models as teachers to improve student accuracy, demonstrating scalability of the distillation approach.

Experimental results

Research questions

• RQ1 Can a shallow feedforward network achieve performance comparable to a deep convolutional neural network on speech and image recognition tasks?
• RQ2 Does the performance advantage of deep networks stem from architectural depth or from better inductive biases and training procedures?
• RQ3 Can knowledge distillation from a deep teacher model enable a shallow student network to match or exceed the accuracy of the teacher with fewer or similar parameters?
• RQ4 Is there a fundamental representational capacity limit in shallow networks, or is the bottleneck primarily in optimization and regularization?
• RQ5 To what extent does the availability of unlabeled data or higher-accuracy teachers improve the performance of shallow student models?

Key findings

• Shallow feedforward networks trained to mimic deep models achieved test accuracies comparable to state-of-the-art deep convolutional networks on both TIMIT and CIFAR-10.
• On TIMIT, a shallow net with 160K parameters (SNN-MIMIC-160K) matched the performance of a deeper model with 10 times fewer parameters, demonstrating that depth is not essential for high accuracy.
• Increasing teacher accuracy via ensemble models led to proportional improvements in student model performance, indicating that the student's capacity is not the limiting factor.
• Despite having more parameters than the deep model, the shallow mimic networks trained 6–12 times faster than the deep models, completing in 1–2 hours versus 8–12 hours on GPUs.
• The performance gap between directly trained shallow nets and mimic-trained shallow nets was significant, showing that current learning algorithms struggle to train shallow nets directly on original data.
• Shallow models with the same number of parameters as deep models could achieve similar accuracy when trained via distillation, suggesting that the function learned by deep nets does not inherently require depth.