[Paper Review] Predicting on the Edge: Identifying Where a Larger Model Does Better

Taman Narayan, Heinrich Jiang+2 more|arXiv (Cornell University)|Feb 15, 2022

Advanced Data Storage Technologies4 citations

TL;DR

This paper proposes using a small model's predictive uncertainty to identify examples where a larger model is likely to improve, enabling a switcher mechanism that defers to the larger model only when needed. The key finding is that larger models improve most on examples where the small model is most uncertain, and a switcher model can achieve better performance and efficiency than using the larger model alone.

ABSTRACT

Much effort has been devoted to making large and more accurate models, but relatively little has been put into understanding which examples are benefiting from the added complexity. In this paper, we demonstrate and analyze the surprisingly tight link between a model's predictive uncertainty on individual examples and the likelihood that larger models will improve prediction on them. Through extensive numerical studies on the T5 encoder-decoder architecture, we show that large models have the largest improvement on examples where the small model is most uncertain. On more certain examples, even those where the small model is not particularly accurate, large models are often unable to improve at all, and can even perform worse than the smaller model. Based on these findings, we show that a switcher model which defers examples to a larger model when a small model is uncertain can achieve striking improvements in performance and resource usage. We also explore committee-based uncertainty metrics that can be more effective but less practical.

Motivation & Objective

• To understand when and why larger models improve over smaller models on individual examples.
• To develop a practical method for identifying examples where larger models are likely to outperform smaller ones without requiring labeled data.
• To explore how uncertainty and model disagreement can guide efficient, adaptive inference using smaller and larger model combinations.
• To evaluate whether committee-based uncertainty or model retraining churn can outperform individual model uncertainty in predicting larger model gains.

Proposed method

• Train a small model and use its predictive uncertainty (e.g., entropy of softmax output) as a proxy for identifying examples likely to benefit from a larger model.
• Implement a switcher model that defers prediction to a larger model only when the small model's uncertainty exceeds a threshold.
• Use per-example disagreement (churn) across multiple retrained small models as an alternative uncertainty proxy to assess model agreement and potential for improvement.
• Compare the effectiveness of individual model uncertainty, committee uncertainty (average of multiple small model uncertainties), and churn (prediction variance across retrainings) in predicting larger model performance gains.
• Evaluate the switcher model’s performance and computational efficiency against using the larger model on all examples, across multiple datasets and model pairs.
• Analyze the relationship between uncertainty, model disagreement, and performance improvement using the T5 encoder-decoder architecture on diverse NLP tasks.

Experimental results

Research questions

• RQ1 On which examples does a larger model show the greatest improvement over a smaller model?
• RQ2 Can a small model’s uncertainty reliably predict where a larger model will outperform it?
• RQ3 Under what conditions does a switcher model that defers to a larger model based on small model uncertainty outperform the larger model alone?
• RQ4 How does model disagreement (churn) across retrained small models compare to individual uncertainty in predicting larger model improvement?
• RQ5 Why do larger models sometimes underperform on examples where the smaller model is highly certain?

Key findings

• Larger models show the largest performance improvements on examples where the smaller model is most uncertain, while offering little to no improvement on examples where the smaller model is highly confident.
• On a significant fraction of examples—especially those where the small model is certain—larger models fail to improve and can even underperform the smaller model.
• A switcher model that defers to a larger model only when the small model is uncertain achieves substantial computational savings and can outperform the larger model alone, particularly when the two models are relatively close in performance.
• Per-example retraining churn (i.e., prediction disagreement across multiple small model retrainings) is a stronger predictor of larger model improvement than either individual model uncertainty or committee uncertainty.
• The committee-based uncertainty estimator, while theoretically stronger, is less effective in practice because it tends to identify examples that are already easy for the larger model, limiting the switcher’s ability to outperform the larger model.
• The counterintuitive result that some highly uncertain examples are 'easier' for larger models to improve on—despite being 'hard' for the small model—arises because uncertainty can stem from model undercapacity rather than true example difficulty.