[Paper Review] High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach

Tim Pearce, Mohamed Zaki+2 more|arXiv (Cornell University)|Feb 20, 2018

Adversarial Robustness in Machine LearningComputer Science27 references150 citations

TL;DR

Introduces a distribution-free loss (Quality-Driven, HQ) to produce narrow prediction intervals (PIs) for regression, and shows QD-Ens improves PI quality over state-of-the-art MVE-Ens on ten benchmarks.

ABSTRACT

This paper considers the generation of prediction intervals (PIs) by neural networks for quantifying uncertainty in regression tasks. It is axiomatic that high-quality PIs should be as narrow as possible, whilst capturing a specified portion of data. We derive a loss function directly from this axiom that requires no distributional assumption. We show how its form derives from a likelihood principle, that it can be used with gradient descent, and that model uncertainty is accounted for in ensembled form. Benchmark experiments show the method outperforms current state-of-the-art uncertainty quantification methods, reducing average PI width by over 10%.

Motivation & Objective

• Quantify uncertainty in regression with high-quality prediction intervals.
• Develop a loss function for PI output that is distribution-free and aligned with the HQ principle.
• Incorporate model uncertainty via ensembles and compare to state-of-the-art uncertainty methods.

Proposed method

• Derive LossQD from the HQ principle to minimize PI width under a fixed coverage (1-α).
• Use captured MPIW (MPIWcapt) to avoid shrinking widths for missed data points.
• Adopt a Bernoulli likelihood for PI coverage and apply a one-sided loss term to enforce coverage when PICP < (1−α).
• Present a differentiable (soft) version LossQD-soft to enable gradient descent training.
• Train ensembles of neural networks with LossQD-soft to capture model uncertainty; combine ensemble bounds to form final PIs.
• Benchmark ten open datasets against MVE-Ens, reporting PICP and MPIW as PI quality metrics.

Experimental results

Research questions

• RQ1 How to construct a distribution-free loss that yields narrow PIs with specified coverage in neural networks?
• RQ2 Can an ensemble of neural networks capture model uncertainty in PI estimation effectively?
• RQ3 How does the proposed HQ-based loss compare to LUBE and MVE in terms of PI width and coverage on benchmark datasets?

Key findings

• QD-Ens outperformed MVE-Ens on PI quality metrics across ten datasets, with PICP closer to 0.95 and MPIW about 11.6% narrower on average.
• The HQ loss yields a robust, distribution-free approach that does not assume Gaussian noise and remains competitive with gradient descent when using a soft approximation.
• PI width reduction is achieved while maintaining or improving coverage, demonstrating effective uncertainty quantification without distributional assumptions.
• Ensembling provides a practical mechanism to estimate model uncertainty, with ensemble upper/lower PI bounds formed from member predictions.
• Training considerations showed LossQD-soft enables gradient-based optimization, and that lower learning rates and more epochs can improve stability compared to MVE-Ens.
• The paper reports a favorable comparison to MVE-Ens when converting MVE outputs to PIs for fair PI-criteria assessment.