[Paper Review] Dynamic Neural Network Channel Execution for Efficient Training

Simeon E. Spasov, Píetro Lió|arXiv (Cornell University)|May 15, 2019

Advanced Neural Network Applications34 references4 citations

TL;DR

This paper proposes a dynamic channel execution framework that reduces training and inference costs in CNNs by selectively activating only the most salient convolutional channels at each training step using a combinatorial upper confidence bound (CUCB) algorithm. The method achieves up to 4× reduction in FLOPs and 9× reduction in parameters with minimal accuracy drop, enabling efficient discovery of compact models without additional model complexity.

ABSTRACT

Existing methods for reducing the computational burden of neural networks at run-time, such as parameter pruning or dynamic computational path selection, focus solely on improving computational efficiency during inference. On the other hand, in this work, we propose a novel method which reduces the memory footprint and number of computing operations required for training and inference. Our framework efficiently integrates pruning as part of the training procedure by exploring and tracking the relative importance of convolutional channels. At each training step, we select only a subset of highly salient channels to execute according to the combinatorial upper confidence bound algorithm, and run a forward and backward pass only on these activated channels, hence learning their parameters. Consequently, we enable the efficient discovery of compact models. We validate our approach empirically on state-of-the-art CNNs - VGGNet, ResNet and DenseNet, and on several image classification datasets. Results demonstrate our framework for dynamic channel execution reduces computational cost up to 4x and parameter count up to 9x, thus reducing the memory and computational demands for discovering and training compact neural network models.

Motivation & Objective

• To reduce the memory and computational demands of training deep CNNs on resource-constrained devices.
• To integrate channel pruning directly into the training process rather than applying it post-training.
• To enable efficient discovery of compact, parameter-efficient neural network models through dynamic channel selection.
• To minimize model complexity while maintaining or improving accuracy during training and inference.

Proposed method

• The framework formulates channel selection as a combinatorial multi-armed bandit problem to balance exploration and exploitation of channel saliency.
• It uses the combinatorial upper confidence bound (CUCB) algorithm to select a subset of channels with the highest estimated saliency at each training step.
• Only the selected channels are activated and executed during forward and backward passes, reducing memory and FLOP usage.
• Channel saliency is tracked independently of position or connectivity, based on their relative contribution to network performance.
• After identifying salient channels, the network topology is frozen and the selected channels are fine-tuned to improve accuracy.
• The method avoids introducing additional parameters for gating, unlike prior dynamic inference methods.

Experimental results

Research questions

• RQ1 Can dynamic channel selection during training significantly reduce computational and memory costs without sacrificing model accuracy?
• RQ2 How does integrating pruning into the training process affect the efficiency and performance of compact CNNs?
• RQ3 Does using CUCB for channel selection outperform random or heuristic channel selection strategies?
• RQ4 To what extent can parameter count and FLOPs be reduced while maintaining baseline accuracy on image classification tasks?

Key findings

• On SVHN and CIFAR-10, ResNet-50 trained with 30% active channels achieved over 4× reduction in FLOPs and 3× reduction in parameters, with minimal accuracy drop.
• On CIFAR-10, the best-performing VGG model achieved a 2× reduction in parameters using 70–90% active channels.
• For very compact models (30–40% active channels), parameter reduction reached 3×–7× and FLOP reduction 2×–5×, with accuracy comparable to baseline on CIFAR-10 and SVHN.
• ResNet-50 achieved up to 9× parameter reduction on CIFAR-10 and SVHN while maintaining baseline-level performance.
• On CIFAR-100, models with 30–40% active channels showed a 2–3% accuracy drop, indicating higher capacity needs for 100-class classification.
• The CUCB-based channel selection outperformed random selection, especially at low channel ratios, with performance gaps widening as active channels decreased.