[Paper Review] A Comprehensive Survey on Evidential Deep Learning and Its Applications

Junyu Gao, Mengyuan Chen+2 more|arXiv (Cornell University)|Sep 7, 2024

Machine Learning and Data Classification4 citations

TL;DR

This survey presents Evidential Deep Learning (EDL) as a single-forward-pass framework for reliable uncertainty estimation in deep learning, leveraging subjective logic and Dirichlet distributions to model belief masses and uncertainty. It achieves efficient uncertainty quantification with minimal computational overhead, outperforming traditional methods like Bayesian neural networks and deep ensembling in inference speed while maintaining strong uncertainty calibration.

ABSTRACT

Reliable uncertainty estimation has become a crucial requirement for the industrial deployment of deep learning algorithms, particularly in high-risk applications such as autonomous driving and medical diagnosis. However, mainstream uncertainty estimation methods, based on deep ensembling or Bayesian neural networks, generally impose substantial computational overhead. To address this challenge, a novel paradigm called Evidential Deep Learning (EDL) has emerged, providing reliable uncertainty estimation with minimal additional computation in a single forward pass. This survey provides a comprehensive overview of the current research on EDL, designed to offer readers a broad introduction to the field without assuming prior knowledge. Specifically, we first delve into the theoretical foundation of EDL, the subjective logic theory, and discuss its distinctions from other uncertainty estimation frameworks. We further present existing theoretical advancements in EDL from four perspectives: reformulating the evidence collection process, improving uncertainty estimation via OOD samples, delving into various training strategies, and evidential regression networks. Thereafter, we elaborate on its extensive applications across various machine learning paradigms and downstream tasks. In the end, an outlook on future directions for better performances and broader adoption of EDL is provided, highlighting potential research avenues.

Motivation & Objective

• To provide a comprehensive, accessible introduction to Evidential Deep Learning (EDL) without assuming prior knowledge of the field.
• To clarify the theoretical foundation of EDL based on subjective logic and its distinction from other uncertainty estimation frameworks.
• To systematically review recent theoretical advancements in EDL, including evidence collection, OOD generalization, training strategies, and regression extensions.
• To survey the broad applications of EDL across machine learning paradigms and downstream tasks such as computer vision, NLP, and scientific research.
• To identify open challenges and future research directions, particularly in integrating EDL with large foundation models and high-risk applications.

Proposed method

• EDL models uncertainty using subjective logic, representing beliefs as belief masses and uncertainty as a separate mass, enabling explicit 'I don’t know' states.
• It employs a Dirichlet distribution as a bijective representation of a subjective opinion, where parameters correspond to evidence and uncertainty.
• A deep neural network acts as an evidence collector, producing sufficient statistics (evidence) for each class to define the Dirichlet posterior.
• Model optimization is performed by minimizing standard loss functions integrated over the Dirichlet distribution, enabling end-to-end training.
• The framework supports both classification and regression tasks through evidential regression networks, extending uncertainty modeling to continuous outputs.
• EDL improves uncertainty estimation via OOD sample regularization and adaptive evidence collection, enhancing robustness to distributional shift.

Experimental results

Research questions

• RQ1 How does Evidential Deep Learning differ theoretically from Bayesian neural networks and deep ensembling in modeling uncertainty?
• RQ2 What are the key theoretical advancements in evidence collection, OOD generalization, and training strategies within EDL?
• RQ3 How can EDL be effectively applied across diverse machine learning paradigms such as computer vision, NLP, and cross-modal learning?
• RQ4 What are the limitations of current EDL approaches in estimating pure aleatoric uncertainty, and how can they be addressed?
• RQ5 Can EDL be seamlessly integrated into large pre-trained foundation models without increasing computational cost or degrading performance?

Key findings

• EDL enables reliable uncertainty estimation with only a single forward pass, significantly reducing computational cost compared to deep ensembling or Bayesian neural networks.
• The use of Dirichlet distributions allows EDL to naturally model uncertainty as a separate mass, providing a principled way to express 'I don’t know'.
• EDL outperforms standard deep learning models in uncertainty calibration, particularly in detecting out-of-distribution samples, when combined with OOD-aware training.
• Despite strong theoretical grounding, EDL currently underperforms in some benchmarks compared to mainstream uncertainty methods, indicating room for improvement.
• EDL shows strong potential for integration with large foundation models, especially through distillation and compression techniques, though direct application remains underexplored.
• Future applications in content generation, such as improving diffusion models via uncertainty-aware sampling, are promising and largely unexplored.