[Paper Review] AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification

Romain Égelé, Romit Maulik+4 more|arXiv (Cornell University)|Oct 26, 2021

Adversarial Robustness in Machine Learning28 references4 citations

TL;DR

AutoDEUQ proposes an automated method for generating diverse, high-performing deep neural network ensembles using joint neural architecture and hyperparameter search, leveraging the law of total variance to decompose predictive uncertainty into aleatoric and epistemic components. It outperforms probabilistic backpropagation, Monte Carlo dropout, and other ensemble methods on multiple regression benchmarks by improving uncertainty quantification and predictive accuracy.

ABSTRACT

Deep neural networks are powerful predictors for a variety of tasks. However, they do not capture uncertainty directly. Using neural network ensembles to quantify uncertainty is competitive with approaches based on Bayesian neural networks while benefiting from better computational scalability. However, building ensembles of neural networks is a challenging task because, in addition to choosing the right neural architecture or hyperparameters for each member of the ensemble, there is an added cost of training each model. We propose AutoDEUQ, an automated approach for generating an ensemble of deep neural networks. Our approach leverages joint neural architecture and hyperparameter search to generate ensembles. We use the law of total variance to decompose the predictive variance of deep ensembles into aleatoric (data) and epistemic (model) uncertainties. We show that AutoDEUQ outperforms probabilistic backpropagation, Monte Carlo dropout, deep ensemble, distribution-free ensembles, and hyper ensemble methods on a number of regression benchmarks.

Motivation & Objective

• To address the challenge of uncertainty quantification in deep neural networks, particularly epistemic uncertainty arising from model parameter estimation and out-of-distribution predictions.
• To overcome the limitations of fixed-architecture ensembles by enabling model diversity through automated joint search over neural architectures and hyperparameters.
• To improve uncertainty estimation by decomposing predictive variance into aleatoric (data) and epistemic (model) components using the law of total variance.
• To develop a scalable, automated framework that generates high-performing ensembles without manual tuning, enhancing robustness and reliability of deep learning predictions.
• To evaluate the method on diverse regression benchmarks and compare its performance against established uncertainty quantification techniques.

Proposed method

• AutoDEUQ employs a joint search over neural architecture and hyperparameters using an adaptive evolutionary Bayesian optimization (AgEBO) framework to generate diverse ensemble members.
• It applies the law of total variance to decompose the predictive variance of the ensemble into aleatoric and epistemic uncertainty components.
• The method uses a greedy model selection strategy to build the final ensemble from a catalog of trained models, selecting models that minimize validation loss while maintaining diversity.
• Training is performed in parallel across multiple GPUs, with each model configuration (architecture and hyperparameters) evaluated independently.
• The framework integrates Bayesian optimization to efficiently explore the search space and prioritize configurations with high predictive performance and uncertainty calibration.
• Ensemble predictions are computed via averaging, and uncertainty estimates are derived from the variance of predictions across ensemble members.

Experimental results

Research questions

• RQ1 Can an automated framework jointly optimize neural architecture and hyperparameters to generate diverse and high-performing deep neural network ensembles?
• RQ2 How does the decomposition of predictive variance into aleatoric and epistemic uncertainty improve uncertainty quantification in deep learning models?
• RQ3 Does AutoDEUQ outperform existing uncertainty quantification methods such as Monte Carlo dropout, probabilistic backpropagation, and fixed-architecture ensembles on regression benchmarks?
• RQ4 To what extent does model diversity, induced by joint architecture and hyperparameter search, enhance epistemic uncertainty estimation?
• RQ5 How does AutoDEUQ scale across datasets of varying size and complexity?

Key findings

• AutoDEUQ outperforms probabilistic backpropagation, Monte Carlo dropout, standard deep ensembles, distribution-free ensembles, and hyperensembles on multiple regression benchmarks in terms of predictive accuracy and uncertainty calibration.
• The method achieves superior epistemic uncertainty estimation by leveraging model diversity through joint architecture and hyperparameter search, reducing the risk of convergence to a single hypothesis.
• The decomposition of predictive variance using the law of total variance enables clear separation and quantification of aleatoric and epistemic uncertainty components.
• AutoDEUQ demonstrates strong performance on large-scale datasets such as YearPrediction, achieving low predictive loss and well-calibrated uncertainty estimates.
• The framework maintains computational feasibility through parallel training and efficient Bayesian optimization, even with large search spaces.
• The final ensemble constructed via greedy selection achieves lower validation loss than individual models and baseline ensemble methods, confirming the effectiveness of the selection strategy.