[Paper Review] Deeply-Supervised Nets

Chen‐Yu Lee, Saining Xie+3 more|arXiv (Cornell University)|Sep 18, 2014

Advanced Neural Network Applications26 references1,034 citations

TL;DR

This paper proposes Deeply-Supervised Nets (DSN), a novel deep learning framework that enhances feature discriminativeness and training stability by introducing auxiliary classification heads (companion objectives) at multiple hidden layers in convolutional neural networks. By directly supervising intermediate layers, DSN improves convergence, reduces vanishing gradient effects, and achieves state-of-the-art performance on MNIST (0.39% error), CIFAR-10 (8.22% with augmentation), CIFAR-100 (34.57%), and SVHN (1.92%), outperforming standard CNNs and prior methods.

ABSTRACT

Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent. We make an attempt to boost the classification performance by studying a new formulation in deep networks. Three aspects in convolutional neural networks (CNN) style architectures are being looked at: (1) transparency of the intermediate layers to the overall classification; (2) discriminativeness and robustness of learned features, especially in the early layers; (3) effectiveness in training due to the presence of the exploding and vanishing gradients. We introduce "companion objective" to the individual hidden layers, in addition to the overall objective at the output layer (a different strategy to layer-wise pre-training). We extend techniques from stochastic gradient methods to analyze our algorithm. The advantage of our method is evident and our experimental result on benchmark datasets shows significant performance gain over existing methods (e.g. all state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and SVHN).

Motivation & Objective

• To improve the transparency and discriminativeness of features learned in hidden layers of deep convolutional networks.
• To address the vanishing and exploding gradient problems during backpropagation in deep networks.
• To enhance training efficiency and convergence by directly supervising intermediate representations.
• To provide a theoretically grounded, scalable framework that improves generalization without relying on data augmentation or complex pre-training.
• To demonstrate consistent performance gains across diverse benchmark datasets, including MNIST, CIFAR-10, CIFAR-100, and SVHN.

Proposed method

• Introduces a companion objective (auxiliary classifier) at each hidden layer, in addition to the main output classifier.
• Uses stochastic gradient methods to analyze and justify the convergence improvement of the DSN framework.
• Applies standard deep learning components (convolutional layers, ReLU, pooling) but adds multi-level supervision via separate classifiers on feature maps.
• Employs a joint optimization scheme where the total loss is the sum of all layer-specific classification losses and the final output loss.
• Supports various activation functions and regularization techniques (e.g., dropout, maxout) as compatible components.
• Uses SVM or softmax as the final classifier, with experiments showing DSN-SVM and DSN-Softmax outperform their non-deeply-supervised counterparts.

Experimental results

Research questions

• RQ1 Can direct supervision of intermediate hidden layers improve feature quality and classification performance in deep networks?
• RQ2 Does multi-level supervision reduce the impact of vanishing gradients and accelerate training convergence?
• RQ3 How does DSN compare to standard CNNs and other state-of-the-art methods across diverse image classification benchmarks?
• RQ4 To what extent does DSN improve robustness to hyperparameter choices and data scarcity?
• RQ5 Can the DSN framework be extended to incorporate existing regularization and architectural techniques like dropout, maxout, or data augmentation?

Key findings

• DSN achieved a test error of 0.39% on MNIST, surpassing all prior state-of-the-art methods, including those using data augmentation.
• On CIFAR-10, DSN achieved 9.78% error without data augmentation and 8.22% with augmentation, setting a new SOTA benchmark.
• On CIFAR-100, DSN achieved 34.57% test error, significantly outperforming previous methods such as Network in Network (35.68%) and Maxout Networks (38.57%).
• On SVHN, DSN achieved 1.92% error, matching the best-known result and outperforming methods like DropConnect (1.94%) and Maxout Networks (2.47%).
• Gradient variance in the first convolutional layer of DSN was 4.55 times higher than in standard CNNs, indicating stronger signal propagation and improved learning dynamics.
• Visualizations of feature maps showed that DSN-learned features were more intuitive and discriminative than those from standard CNNs, especially in early layers.