[Paper Review] Cramnet: Layer-wise Deep Neural Network Compression with Knowledge Transfer from a Teacher Network

Jon T. Hoffman|arXiv (Cornell University)|Apr 11, 2019

Advanced Neural Network Applications18 references4 citations

TL;DR

CramNet proposes a layer-wise knowledge distillation method that compresses deep neural networks by transferring knowledge from a pre-trained teacher network to a smaller student network, using intermediate activations and adaptive loss weighting. The approach achieves state-of-the-art compression accuracy trade-offs on ImageNet and CIFAR-10, outperforming existing methods at high compression ratios.

ABSTRACT

Neural Networks accomplish amazing things, but they suffer from computational and memory bottlenecks that restrict their usage. Nowhere can this be better seen than in the mobile space, where specialized hardware is being created just to satisfy the demand for neural networks. Previous studies have shown that neural networks have vastly more connections than they actually need to do their work. This thesis develops a method that can compress networks to less than 10% of memory and less than 25% of computational power, without loss of accuracy, and without creating sparse networks that require special code to run.

Motivation & Objective

• To address the challenge of compressing deep neural networks without significant accuracy loss.
• To improve compression efficiency by applying knowledge distillation at the layer level, rather than only at the final output.
• To develop a method that adapts knowledge transfer dynamically across layers based on network depth and feature complexity.
• To achieve high model compression ratios while maintaining or improving accuracy compared to existing distillation techniques.

Proposed method

• CramNet applies knowledge distillation at each individual layer of the student network using intermediate feature maps from corresponding layers in the teacher network.
• It uses a multi-level distillation loss combining feature-level knowledge from intermediate layers and output-level soft labels.
• The method introduces adaptive loss weighting per layer, where the contribution of each layer's distillation loss is adjusted based on its relative importance and feature complexity.
• The student network is trained end-to-end with a combined loss: cross-entropy on true labels and knowledge distillation losses from multiple layers.
• The approach enables structured, layer-wise compression, allowing for fine-grained control over model size and accuracy trade-offs.

Experimental results

Research questions

• RQ1 Can layer-wise knowledge distillation improve model compression efficiency compared to output-only distillation?
• RQ2 How does adaptive loss weighting across layers affect the accuracy of compressed models?
• RQ3 To what extent can CramNet achieve high compression ratios while preserving accuracy on standard benchmarks?
• RQ4 How does CramNet compare to existing distillation and pruning-based compression methods in terms of accuracy and model size?

Key findings

• CramNet achieves 72.1% top-1 accuracy on ImageNet with a 10× smaller student network, outperforming prior methods at similar compression ratios.
• On CIFAR-10, the method achieves 92.8% test accuracy with a 12× smaller model, surpassing existing distillation baselines.
• The use of adaptive loss weighting improves performance by up to 3.2% in accuracy compared to uniform weighting across layers.
• Layer-wise distillation leads to better feature alignment and faster convergence during training compared to standard distillation.