[Paper Review] Learning Compact Convolutional Neural Networks with Nested Dropout

Chelsea Finn, Lisa Anne Hendricks+1 more|arXiv (Cornell University)|Dec 22, 2014

Generative Adversarial Networks and Image Synthesis4 references3 citations

TL;DR

This paper proposes nested dropout as a method to automatically learn compact, task-adaptive convolutional neural network architectures by ordering filters by information content during training. By drawing from a geometric distribution to progressively drop out higher-indexed filters, the network identifies optimal filter counts (e.g., 23 for conv1) with minimal training iterations, achieving comparable accuracy to full networks while reducing parameters by 25%.

ABSTRACT

Recently, nested dropout was proposed as a method for ordering representation units in autoencoders by their information content, without diminishing reconstruction cost. However, it has only been applied to training fully-connected autoencoders in an unsupervised setting. We explore the impact of nested dropout on the convolutional layers in a CNN trained by backpropagation, investigating whether nested dropout can provide a simple and systematic way to determine the optimal representation size with respect to the desired accuracy and desired task and data complexity.

Motivation & Objective

• To investigate whether nested dropout—previously used only in autoencoders—can be effectively applied to convolutional layers in backpropagation-trained CNNs.
• To develop a systematic method for determining optimal network capacity based on data and task complexity.
• To enable automatic, incremental learning of filter count in convolutional layers without manual hyperparameter tuning.
• To demonstrate that nested dropout can yield compact, high-accuracy models with fewer training iterations than brute-force search.

Proposed method

• Nested dropout is applied to convolutional layers by drawing a number k from a geometric distribution and dropping out all filters from index k+1 to n, preserving the first k+1 filters.
• Filters are trained iteratively: after a fixed number of iterations, the sweeping index is incremented, and only remaining filters are considered in subsequent draws.
• The method is implemented in Caffe by adding a custom nested dropout layer that follows any layer (e.g., convolutional or fully-connected) and integrates with the solver for unit sweeping.
• The approach enables incremental capacity scaling: filters with lower indices converge first and are fixed, while higher-index filters are progressively dropped out.
• The process is applied layer-by-layer: first determining optimal filter count for conv1, then using nested dropout to determine filters for conv2.
• The training process uses stochastic gradient descent with mini-batches of 100 samples, and filter removal is based on convergence rather than fixed iteration counts.

Experimental results

Research questions

• RQ1 Can nested dropout be successfully applied to convolutional layers in backpropagation-trained CNNs to learn compact representations?
• RQ2 Does nested dropout enable automatic determination of optimal filter count for a given task and data complexity?
• RQ3 Can nested dropout reduce the number of required training iterations compared to brute-force hyperparameter search?
• RQ4 How does the performance of a nested dropout-trained network compare to a full network with all filters and to a pruned network with randomly selected filters?
• RQ5 Can nested dropout identify a minimal set of filters that maintain high accuracy while reducing model parameters?

Key findings

• The nested dropout network achieved maximum test accuracy with only 23 filters in conv1, outperforming a brute-force approach that required training 32 separate networks.
• The nested dropout model reached 0.787 test accuracy with 23 conv1 filters, matching the 0.786 accuracy of a full 32-filter baseline network.
• Training with nested dropout required only 90,000 iterations, compared to 2,880,000 iterations for training 32 separate networks in a brute-force search.
• Applying nested dropout to both conv1 and conv2 reduced the total number of parameters in the first two layers by 25%, from 64 to 48 filters.
• Visualizations showed that higher-index filters in the nested dropout network carried minimal information, confirming that only the first 23 filters were essential for performance.
• After 5,000 iterations, 25 conv2 filters were learned with 78% training accuracy, indicating that the network could converge to a compact, effective architecture.