[Paper Review] Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation

Jannik Kossen, Sebastian Farquhar+2 more|arXiv (Cornell University)|Feb 14, 2022

Machine Learning and Algorithms4 citations

TL;DR

This paper introduces Active Surrogate Estimators (ASEs), a novel active learning approach for label-efficient model evaluation that uses an actively learned surrogate model to predict losses across unlabeled data, outperforming state-of-the-art methods like LURE and adaptive IS in estimating deep neural network risks with significantly fewer labels. ASEs achieve this by interpolating losses via a surrogate model and using a novel acquisition strategy, XWED, tailored to estimation accuracy, enabling superior performance even in zero-shot settings without test labels.

ABSTRACT

We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels are expensive. ASEs address this active testing problem using a surrogate-based estimation approach that interpolates the errors of points with unknown labels, rather than forming a Monte Carlo estimator. ASEs actively learn the underlying surrogate, and we propose a novel acquisition strategy, XWED, that tailors this learning to the final estimation task. We find that ASEs offer greater label-efficiency than the current state-of-the-art when applied to challenging model evaluation problems for deep neural networks.

Motivation & Objective

• To address the challenge of high-cost label acquisition in model evaluation, especially in domains like medical imaging or experimental physics where labels require expert effort.
• To overcome limitations of Monte Carlo and importance sampling methods in label-efficient evaluation, such as high variance from early poor proposal weights and sensitivity to label noise.
• To develop a flexible, active learning framework that learns a surrogate model of losses across the data pool, enabling efficient and accurate risk estimation.
• To design a novel acquisition strategy, XWED, that directly optimizes for estimation performance rather than proposal quality, improving label efficiency.

Proposed method

• ASEs use a surrogate model $ g $ to predict losses at all points in the unlabeled pool, replacing traditional Monte Carlo or importance sampling estimators.
• The surrogate model $ g $ is iteratively refined using actively selected labels, allowing retrospective improvements to predictions at all points, not just newly labeled ones.
• A novel acquisition function, XWED (eXtended Weighted Expected Disagreement), selects points to label based on their expected contribution to reducing final estimator error, not just proposal quality.
• XWED computes acquisition scores based on the expected disagreement between the current surrogate and a hypothetical future surrogate, weighted by uncertainty, to prioritize informative points.
• The method supports deterministic acquisition strategies, unlike importance sampling methods that require stochastic proposals to avoid bias.
• ASEs are compatible with any differentiable loss function and can estimate accuracy, risk, or other performance metrics directly from the surrogate-predicted losses.

Experimental results

Research questions

• RQ1 Can a surrogate-based estimation framework outperform Monte Carlo and importance sampling methods in label-efficient model evaluation for deep neural networks?
• RQ2 How does actively learning a surrogate model improve label efficiency compared to traditional active learning or adaptive IS approaches?
• RQ3 To what extent can ASEs achieve accurate risk estimation without any test labels, relying solely on training data and a well-learned surrogate?
• RQ4 How does the XWED acquisition strategy compare to existing acquisition functions in terms of reducing estimator variance and improving convergence?
• RQ5 What is the impact of label noise on ASEs compared to importance sampling estimators, and can ASEs maintain accuracy under noisy observations?

Key findings

• ASEs consistently outperform the current state-of-the-art methods, including LURE and adaptive IS, in estimating the risk of deep neural networks across multiple image classification benchmarks.
• In a zero-shot setting with no test labels, ASEs achieve significantly lower error than LURE, even when LURE has access to up to 50 test labels, demonstrating the power of surrogate generalization.
• The performance of ASEs is limited by the pool size $ N $ rather than surrogate quality, indicating that the method scales well with data availability.
• ASEs maintain a substantial performance advantage over baselines even when the training set is reduced to 10,000 samples, showing robustness to limited data.
• XWED acquisition leads to faster convergence and lower variance than alternative strategies, particularly in early labeling stages, due to its direct focus on estimation error reduction.
• ASEs are robust to label noise because they model the conditional distribution of losses directly, whereas importance sampling methods suffer increased variance from noisy observations.