[Paper Review] Make (Nearly) Every Neural Network Better: Generating Neural Network Ensembles by Weight Parameter Resampling

Jiayi Liu, Samarth Tripathi+2 more|arXiv (Cornell University)|Jul 2, 2018

Advanced Neural Network Applications11 references3 citations

TL;DR

This paper proposes a universal, efficient method to improve nearly any pre-trained deep neural network by resampling its weight parameters based on their uncertainty distribution, estimated during a brief fine-tuning phase. The approach achieves consistent performance gains across diverse models—such as Inception-V3 and MobileNet—by averaging or resampling weights from the final training phase, with minimal computational overhead and no need for retraining from scratch.

ABSTRACT

Deep Neural Networks (DNNs) have become increasingly popular in computer vision, natural language processing, and other areas. However, training and fine-tuning a deep learning model is computationally intensive and time-consuming. We propose a new method to improve the performance of nearly every model including pre-trained models. The proposed method uses an ensemble approach where the networks in the ensemble are constructed by reassigning model parameter values based on the probabilistic distribution of these parameters, calculated towards the end of the training process. For pre-trained models, this approach results in an additional training step (usually less than one epoch). We perform a variety of analysis using the MNIST dataset and validate the approach with a number of DNN models using pre-trained models on the ImageNet dataset.

Motivation & Objective

• To develop a fast, universal method to enhance the performance of pre-trained deep neural networks without retraining from scratch.
• To investigate how uncertainty in stochastic gradient descent (SGD) updates can be leveraged to improve model generalization.
• To evaluate the method's robustness across different architectures and optimization strategies, especially in low-data or high-computation-cost scenarios.
• To reduce the computational burden of ensemble methods while maintaining or improving predictive performance.
• To provide a practical, plug-and-play technique applicable to any DNN model, including those already trained on large datasets like ImageNet.

Proposed method

• Estimate the mean and variance of model parameters using an online Welford algorithm during a short fine-tuning phase after initial training.
• Resample model weights from a Gaussian distribution centered at the estimated mean with the estimated standard deviation.
• Construct multiple models via resampling and average their predictions to form an ensemble, improving robustness and accuracy.
• Alternatively, replace all weights with the estimated mean value, effectively creating a single, improved model with no inference cost increase.
• Use a minimal fine-tuning stage (often less than one epoch) to capture parameter uncertainty, making the method computationally efficient.
• Apply the method to both pre-trained models and models trained from scratch, validating its universality across training regimes and optimizers.

Experimental results

Research questions

• RQ1 Can parameter uncertainty captured during a brief fine-tuning phase be used to generate better-performing neural network ensembles?
• RQ2 How does the proposed resampling method compare to standard fine-tuning and existing ensemble techniques like Stochastic Weight Averaging (SWA) in terms of accuracy and efficiency?
• RQ3 To what extent is the method robust to variations in learning rate, optimizer choice (e.g., SGD vs. Adam), and model architecture?
• RQ4 Does the method improve performance on both standard and lightweight models, such as MobileNet, when applied to large-scale datasets like ImageNet?
• RQ5 What is the minimal fine-tuning duration required to reliably estimate parameter uncertainty and achieve consistent performance gains?

Key findings

• The method consistently improves accuracy on the MNIST dataset across multiple configurations, with the mean-resampled model outperforming both the baseline and fine-tuned models.
• On ImageNet, the proposed method improved top-1 accuracy for both Inception-V3 and MobileNet, with gains observed even when fine-tuning was performed with Adam optimizer.
• The performance of the resampled ensemble was robust across a wide range of learning rates, indicating low sensitivity to hyperparameter choice.
• The mean-resampled model alone achieved performance comparable to a 3-model ensemble, suggesting that the mean is a strong representative of the parameter distribution.
• Even with only 10,000 updates (25% of ImageNet), the method reliably estimated parameter uncertainty and produced consistent improvements.
• The method required less than one epoch of fine-tuning to achieve gains, making it highly efficient and practical for real-world deployment.