[Paper Review] Intra-Ensemble in Neural Networks

Yuan Gao, Zixiang Cai+1 more|arXiv (Cornell University)|Apr 9, 2019

Advanced Neural Network Applications52 references4 citations

TL;DR

This paper proposes Intra-Ensemble (IENet), an end-to-end neural network architecture that trains multiple diverse sub-networks within a single model using stochastic channel recombination and switchable batch normalization. By sharing most parameters and introducing controlled channel diversity, IENet achieves significant performance gains over standalone networks with minimal parameter increase, outperforming state-of-the-art models on CIFAR-10 and CIFAR-100 with only 2.78M parameters.

ABSTRACT

Improving model performance is always the key problem in machine learning including deep learning. However, stand-alone neural networks always suffer from marginal effect when stacking more layers. At the same time, ensemble is an useful technique to further enhance model performance. Nevertheless, training several independent deep neural networks for ensemble costs multiple resources. If so, is it possible to utilize ensemble in only one neural network? In this work, we propose Intra-Ensemble, an end-to-end ensemble strategy with stochastic channel recombination operations to train several sub-networks simultaneously within one neural network. Additional parameter size is marginal since the majority of parameters are mutually shared. Meanwhile, stochastic channel recombination significantly increases the diversity of sub-networks, which finally enhances ensemble performance. Extensive experiments and ablation studies prove the applicability of intra-ensemble on various kinds of datasets and network architectures.

Motivation & Objective

• To address the high computational and parameter cost of traditional deep learning ensembles by enabling ensemble learning within a single neural network.
• To improve model performance without increasing model size by leveraging parameter redundancy in over-parameterized networks.
• To increase diversity among sub-networks in a shared architecture to enhance ensemble generalization and robustness.
• To develop a training strategy that maintains high accuracy per sub-network while ensuring low correlation among their predictions.
• To demonstrate the generalizability of the method across various architectures and datasets, including image classification tasks with increasing complexity.

Proposed method

• IENet employs switchable batch normalization (S-BN) to enable training of multiple sub-networks at different widths within a single network, with each sub-network using a distinct ratio of channels.
• Stochastic channel recombination (RC) is introduced as a key operation to randomly reassign channels across sub-networks during training, increasing inter-sub-network diversity.
• Sub-networks are trained end-to-end with shared backbone weights, while S-BN layers maintain independent statistics per width to preserve training stability and performance.
• The method uses stacking or averaging of sub-network predictions during inference to form the final ensemble output, with stacking showing superior results in experiments.
• The approach is applied across multiple architectures (e.g., DenseNet-like, ResNet-like) and datasets, including CIFAR-10, CIFAR-100, and others, with minimal hyperparameter tuning.
• The method is evaluated using a similarity metric (S) to quantify correlation between sub-networks, showing that lower similarity correlates with better ensemble performance.

Experimental results

Research questions

• RQ1 Can a single neural network effectively host multiple diverse sub-networks to achieve ensemble performance without significant parameter overhead?
• RQ2 How does stochastic channel recombination compare to other channel selection strategies (e.g., random cut, random order) in enhancing sub-network diversity and ensemble accuracy?
• RQ3 Does intra-ensemble performance improve more on complex tasks with higher class counts, such as CIFAR-100, compared to simpler tasks like CIFAR-10?
• RQ4 To what extent does the trade-off between sub-network accuracy and similarity affect the final ensemble performance?
• RQ5 Can the intra-ensemble strategy be generalized across different network architectures and datasets without architectural modification?

Key findings

• IENet with 4 sub-networks and stochastic channel recombination achieved a 2.61% test error on CIFAR-100, outperforming a standalone model with 2.82M parameters and 16.04% error.
• On CIFAR-10, the 4-sub-network IENet with RC achieved a 2.61% error, a 1.64% improvement over a standalone model with similar parameter count (2.66M).
• The intra-ensemble method reduced sub-network similarity (S) from 0.857 to 0.653 while improving accuracy, demonstrating that lower correlation enhances ensemble performance.
• On CIFAR-100, the 5-sub-network IENet achieved a 16.04% error rate, a 2.62% improvement over the standalone baseline, indicating greater potential for complex, high-class tasks.
• The method generalized effectively across datasets with varying image sizes and class numbers, showing consistent performance gains on both simple and complex tasks.
• Stochastic channel recombination (RC) provided the best balance between accuracy and diversity, outperforming random order (RO) and sequential cut (SC) despite lower similarity, due to better sub-network accuracy retention.