[Paper Review] Net2Net: Accelerating Learning via Knowledge Transfer

Tianqi Chen, Ian Goodfellow+1 more|arXiv (Cornell University)|Nov 18, 2015

Adversarial Robustness in Machine Learning13 references164 citations

TL;DR

Net2Net accelerates deep learning model training by transferring knowledge from a smaller, pre-trained neural network to a larger, wider or deeper network through function-preserving weight initialization. The method enables faster convergence and achieves a new state-of-the-art 78.5% top-1 accuracy on ImageNet without retraining from scratch.

ABSTRACT

We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net. During real-world workflows, one often trains very many different neural networks during the experimentation and design process. This is a wasteful process in which each new model is trained from scratch. Our Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network. Our techniques are based on the concept of function-preserving transformations between neural network specifications. This differs from previous approaches to pre-training that altered the function represented by a neural net when adding layers to it. Using our knowledge transfer mechanism to add depth to Inception modules, we demonstrate a new state of the art accuracy rating on the ImageNet dataset.

Motivation & Objective

• Accelerate the iterative design of deep neural networks by reusing knowledge from previously trained models.
• Reduce training time in real-world workflows where multiple model architectures are explored sequentially.
• Enable efficient exploration of wider and deeper model architectures without training from random initialization.
• Support lifelong learning systems by allowing seamless scaling of model capacity as training data grows.
• Demonstrate that knowledge transfer can be achieved without loss in performance when scaling network width or depth.

Proposed method

• Net2WiderNet initializes a wider network by splitting each neuron in the teacher network into multiple neurons with scaled weights to preserve the same function.
• Net2DeeperNet initializes a deeper network by inserting identity-preserving residual blocks (vertical-horizontal convolution pairs) into each Inception module.
• The initialization ensures the student network computes the same output as the teacher network before any fine-tuning.
• Function-preserving transformations are applied to maintain the same input-output mapping during architectural changes.
• The initialized student network is then trained with standard optimization, benefiting from the pre-learned function.
• The approach avoids the need for additional distillation heads or auxiliary loss terms, unlike prior knowledge transfer methods.

Experimental results

Research questions

• RQ1 Can knowledge from a smaller, pre-trained neural network be transferred to a larger, wider or deeper network without performance degradation?
• RQ2 Can function-preserving weight initialization significantly reduce training time for larger models compared to random initialization?
• RQ3 Does Net2Net enable faster exploration of model architecture space in deep learning workflows?
• RQ4 Can Net2Net be used to scale up models in a way that maintains or improves accuracy on large-scale benchmarks like ImageNet?
• RQ5 Is the performance gain from Net2Net due to better initialization or additional inductive bias from the architectural transformation?

Key findings

• Net2WiderNet achieved the same final accuracy as a randomly initialized wider network, confirming no performance loss from knowledge transfer.
• Net2DeeperNet enabled faster convergence in both training and validation accuracy compared to random initialization for deeper models.
• By combining width and depth scaling via Net2Net, the authors achieved a new state-of-the-art 78.5% top-1 accuracy on the ImageNet validation set.
• The convergence curves of Net2Net-initialized models were consistently faster than those of randomly initialized models, even for larger architectures.
• The method reduced the time required to train new model variants, enabling rapid iteration in model design workflows.
• The approach was effective even when applied to complex Inception modules, demonstrating scalability in real-world deep learning systems.