[Paper Review] Exploring Feature Reuse in DenseNet Architectures

Andy Hess|arXiv (Cornell University)|Jun 5, 2018

Advanced Neural Network Applications5 references3 citations

TL;DR

This paper proposes WinDenseNet, a variant of DenseNet that limits each layer's connectivity to only the previous N layers (local dense connectivity), reducing parameters and training time while improving accuracy through more efficient feature reuse. The key finding is that lower connectivity with higher growth rate outperforms full DenseNet at fixed capacity, especially in deeper blocks.

ABSTRACT

Densely Connected Convolutional Networks (DenseNets) have been shown to achieve state-of-the-art results on image classification tasks while using fewer parameters and computation than competing methods. Since each layer in this architecture has full access to the feature maps of all previous layers, the network is freed from the burden of having to relearn previously useful features, thus alleviating issues with vanishing gradients. In this work we explore the question: To what extent is it necessary to connect to all previous layers in order to reap the benefits of feature reuse? To this end, we introduce the notion of local dense connectivity and present evidence that less connectivity, allowing for increased growth rate at a fixed network capacity, can achieve a more efficient reuse of features and lead to higher accuracy in dense architectures.

Motivation & Objective

• To investigate whether full dense connectivity in DenseNets is necessary for performance gains.
• To explore the trade-off between connectivity extent and representational capacity (growth rate) in deep networks.
• To determine if local dense connectivity can achieve better parameter efficiency and higher accuracy than full DenseNet.
• To analyze how feature reuse patterns differ across connectivity window sizes.
• To provide evidence that not all previous features are equally useful for reuse in deep networks.

Proposed method

• Introduce a new architecture called WinDenseNet-N, where each layer in a dense block connects only to the previous N layers, forming a 'dense window' of connectivity.
• Parameterize the network using the window size N, with N = L+1 being equivalent to full DenseNet for a block of L layers.
• Maintain standard DenseNet components: batch normalization, ReLU, 3x3 convolutions, and transition layers with pooling for downsampling.
• Train models with identical hyperparameters to the original DenseNet-40 (SGD, batch size 64, momentum 0.9, weight decay 1e-4, dropout 80%).
• Visualize feature reuse by computing normalized dependence scores on feature maps from previous layers across different window sizes.
• Analyze filter strength and reuse patterns across dense blocks to assess how connectivity affects feature utilization.

Experimental results

Research questions

• RQ1 Is full dense connectivity in DenseNets necessary to achieve high performance?
• RQ2 Can reducing connectivity to a local window of previous layers improve parameter efficiency and accuracy?
• RQ3 How does the choice of dense window size affect feature reuse patterns in different dense blocks?
• RQ4 Does higher growth rate with limited connectivity outperform lower growth rate with full connectivity at fixed model capacity?
• RQ5 Are earlier or later feature maps more effectively reused in locally connected dense networks?

Key findings

• WinDenseNet with a window size of 5 achieved 94.9% top-1 accuracy on CIFAR-10, outperforming DenseNet-40 (94.8%) at the same parameter count.
• Reducing dense connectivity from full (N=13) to smaller values (e.g., N=5) significantly reduced the number of parameters and training time with minimal accuracy drop.
• Networks with small dense window sizes (e.g., N=2) showed strong preference for reusing the earliest available feature maps, especially in early dense blocks.
• As window size increased, feature reuse shifted toward nearby layers, with deeper blocks (e.g., block 3) showing declining reuse of distant features.
• The mean filter strength across blocks decreased with larger window sizes, indicating reduced reliance on long-range feature reuse in deeper layers.
• The results suggest that allocating capacity to increased growth rate rather than full connectivity leads to more effective use of parameters.