[Paper Review] Prediction-Oriented Bayesian Active Learning
EPIG, a prediction-focused acquisition function, outperforms BALD by measuring information gain in predictions rather than model parameters, improving predictive performance across datasets and models.
Information-theoretic approaches to active learning have traditionally focused on maximising the information gathered about the model parameters, most commonly by optimising the BALD score. We highlight that this can be suboptimal from the perspective of predictive performance. For example, BALD lacks a notion of an input distribution and so is prone to prioritise data of limited relevance. To address this we propose the expected predictive information gain (EPIG), an acquisition function that measures information gain in the space of predictions rather than parameters. We find that using EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models, and thus provides an appealing drop-in replacement.
Motivation & Objective
- Motivate active learning for predictive performance rather than parameter uncertainty.
- Introduce EPIG, an acquisition function that maximizes expected predictive information gain.
- Show EPIG as a practical drop-in replacement for BALD in pool-based active learning.
Proposed method
- Derive EPIG from Bayesian experimental design as the expected information gain about predictions, not parameters.
- Define a target input distribution p*(x*) and derive EPIG as the expected reduction in predictive uncertainty on samples from p*(x*).
- Express EPIG equivalently as mutual information I((x*, y*); y | x) and as a KL divergence between predictive joint and product distributions.
- Provide Monte Carlo estimators for EPIG using samples from p*(x*) and model posterior surrogates.
- Describe practical sampling strategies for x* when p*(x*) is unknown or when only pool data is available.
- Outline estimation schemes for EPIG in classification settings with non-closed-form predictive distributions.
Experimental results
Research questions
- RQ1Does EPIG lead to better predictive performance than BALD when the goal is accurate predictions on unseen inputs?
- RQ2How does EPIG perform across synthetic data, UCI datasets, and MNIST-like vision settings compared to BALD?
- RQ3How should target input distributions p*(x*) be chosen or approximated in practice?
- RQ4Is EPIG robust to limited knowledge of the downstream target distribution when labeling is constrained?
Key findings
- EPIG often yields improvements in predictive performance over BALD across multiple datasets and models.
- EPIG is especially advantageous when the pool contains many irrelevant inputs relative to the target distribution.
- EPIG remains competitive or superior even with limited or no access to the target input distribution, showing robustness.
- BALD can disproportionately select obscure inputs at pool extremes, whereas EPIG concentrates on regions with higher predictive relevance.
- Experiments on synthetic data, UCI datasets, and MNIST variants demonstrate EPIG's advantages as a drop-in BALD replacement.
Read this paper
Open the full text in the Nubint reader and ask about any passage as you read.
Read this paper on NubintThis review was generated by AI from the original paper. Please check the paper before citing it.