[Paper Review] Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation

Dennis Ulmer, Christian Hardmeier+1 more|arXiv (Cornell University)|Oct 6, 2021

Adversarial Robustness in Machine Learning167 references18 citations

TL;DR

This survey introduces Evidential Deep Learning as a method for uncertainty estimation in deep neural networks, enabling single-forward-pass uncertainty quantification by modeling predictive distributions using Dirichlet distributions. It provides a unified overview of prior and posterior networks for classification and regression, emphasizing uncertainty calibration, OOD detection, and practical implementation techniques with empirical benchmarks and theoretical derivations.

ABSTRACT

Popular approaches for quantifying predictive uncertainty in deep neural networks often involve distributions over weights or multiple models, for instance via Markov Chain sampling, ensembling, or Monte Carlo dropout. These techniques usually incur overhead by having to train multiple model instances or do not produce very diverse predictions. This comprehensive and extensive survey aims to familiarize the reader with an alternative class of models based on the concept of Evidential Deep Learning: For unfamiliar data, they aim to admit "what they don't know", and fall back onto a prior belief. Furthermore, they allow uncertainty estimation in a single model and forward pass by parameterizing distributions over distributions. This survey recapitulates existing works, focusing on the implementation in a classification setting, before surveying the application of the same paradigm to regression. We also reflect on the strengths and weaknesses compared to other existing methods and provide the most fundamental derivations using a unified notation to aid future research.

Motivation & Objective

• To provide a comprehensive, accessible overview of Evidential Deep Learning methods for uncertainty estimation in deep neural networks.
• To unify and compare existing approaches that parameterize priors or posteriors using Dirichlet distributions in classification and regression settings.
• To highlight challenges in uncertainty calibration and propose solutions such as OOD training, knowledge distillation, and density estimation.
• To derive key equations using consistent notation to support reproducibility and future research.
• To promote wider adoption of Evidential Deep Learning beyond uncertainty quantification research communities.

Proposed method

• Models use Dirichlet distributions to parameterize the posterior predictive distribution, enabling uncertainty estimation via concentration parameters α.
• The predictive distribution is computed as a single forward pass, avoiding Monte Carlo sampling or ensemble averaging.
• For classification, the method uses a Dirichlet prior and updates it with evidence from predictions, with uncertainty derived from α₀ = ∑αₖ.
• Techniques such as OOD data augmentation, knowledge distillation, and entropy regularization are applied to improve uncertainty calibration.
• In regression, the framework extends to modeling predictive distributions using conjugate priors and evidential distributions over outputs.
• The survey derives key equations for the Dirichlet distribution, including expectations, variances, and digamma functions, to support theoretical consistency.

Experimental results

Research questions

• RQ1 How can deep neural networks be made to explicitly express uncertainty, especially for out-of-distribution inputs?
• RQ2 What are the key differences between prior networks and posterior networks in Evidential Deep Learning, and how do they affect uncertainty estimation?
• RQ3 How do techniques like OOD data augmentation and knowledge distillation improve uncertainty calibration in evidential models?
• RQ4 What are the theoretical foundations and practical implementations of Dirichlet-based uncertainty estimation in classification and regression?
• RQ5 How do evidential models compare in performance and calibration to traditional Bayesian methods like MC dropout and ensembling?

Key findings

• Evidential Deep Learning enables uncertainty estimation in a single forward pass, significantly reducing inference cost compared to ensemble or MCMC methods.
• Models using Dirichlet distributions for predictive uncertainty show improved calibration on OOD inputs by falling back to prior beliefs.
• Incorporating OOD examples during training reduces vacuity (lack of evidence) and improves detection of out-of-distribution samples.
• Knowledge distillation and entropy regularization help stabilize training and improve uncertainty estimates, especially in low-data regimes.
• The use of graph-based or kernel-based density estimation improves uncertainty modeling by incorporating local structure and neighborhood information.
• Theoretical derivations of the Dirichlet distribution, including expectations and digamma functions, are essential for consistent uncertainty quantification and are unified in this survey.