[Paper Review] How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation

Swaroop Mishra, Anjana Arunkumar|arXiv (Cornell University)|Jun 10, 2021

Software Engineering Research4 citations

TL;DR

This paper proposes a leaderboard customization framework that re-ranks models by weighting samples based on difficulty—quantified via spurious bias, OOD characteristics, and model confidence—to enable equitable, application-specific evaluation. It introduces weighted metrics (WSBias, WMProb), demonstrates reduced performance inflation, and shows a 41% average reduction in pre-deployment effort via a visual analytics tool in a user study with industry experts.

ABSTRACT

Models that top leaderboards often perform unsatisfactorily when deployed in real world applications; this has necessitated rigorous and expensive pre-deployment model testing. A hitherto unexplored facet of model performance is: Are our leaderboards doing equitable evaluation? In this paper, we introduce a task-agnostic method to probe leaderboards by weighting samples based on their `difficulty' level. We find that leaderboards can be adversarially attacked and top performing models may not always be the best models. We subsequently propose alternate evaluation metrics. Our experiments on 10 models show changes in model ranking and an overall reduction in previously reported performance -- thus rectifying the overestimation of AI systems' capabilities. Inspired by behavioral testing principles, we further develop a prototype of a visual analytics tool that enables leaderboard revamping through customization, based on an end user's focus area. This helps users analyze models' strengths and weaknesses, and guides them in the selection of a model best suited for their application scenario. In a user study, members of various commercial product development teams, covering 5 focus areas, find that our prototype reduces pre-deployment development and testing effort by 41% on average.

Motivation & Objective

• To investigate whether current leaderboards provide equitable evaluations by favoring models that excel on easy or biased samples over more robust ones.
• To address the misalignment between benchmark performance and real-world deployment by identifying and weighting hard samples based on OOD characteristics and spurious bias.
• To reduce overestimation of AI capabilities by introducing task-agnostic, difficulty-weighted evaluation metrics that reflect application-specific needs.
• To develop a visual analytics tool enabling end users to customize leaderboards based on their focus areas, improving model selection efficiency.

Proposed method

• Quantify sample difficulty using semantic textual similarity (STS) to detect out-of-distribution (OOD) characteristics.
• Propose WSBias (Weighted Spurious Bias) to assign higher weights to samples with stronger spurious correlations.
• Introduce WMProb (Weighted Maximum Probability) to weight samples based on model confidence, penalizing high-confidence incorrect predictions.
• Use maximum softmax probability as a proxy for model self-assessment of difficulty, enabling confidence-aware evaluation.
• Design a visual analytics tool that dynamically updates model rankings based on user-defined weights across difficulty dimensions.
• Integrate the framework into a leaderboard customization pipeline that supports flexible, interactive evaluation for diverse application scenarios.

Experimental results

Research questions

• RQ1 Can leaderboards be adversarially manipulated by reweighting samples based on difficulty, revealing that top-ranked models may not be the most robust?
• RQ2 To what extent do existing evaluation metrics inflate model performance, and how can weighted metrics reduce this overestimation?
• RQ3 How does customizing leaderboards based on application-specific difficulty criteria improve model selection efficiency in real-world settings?
• RQ4 Can a visual analytics tool that supports dynamic leaderboard customization reduce pre-deployment development and testing effort for industry teams?

Key findings

• Leaderboard rankings are not robust: models that top standard leaderboards may underperform on hard or OOD samples, indicating inequitable evaluation.
• The proposed weighted metrics—WSBias and WMProb—reduce previously reported model performance by up to 15%, correcting for overestimation of AI capabilities.
• A user study with 9 industry teams across 5 focus areas shows a 41% average reduction in pre-deployment development and testing effort when using the customizable leaderboard tool.
• The framework enables detection of model weaknesses such as over-reliance on spurious cues and poor OOD generalization, which are masked in standard accuracy-based rankings.
• Ten models analyzed using the framework reveal distinct strengths and weaknesses, with some excelling in OOD generalization but failing on biased samples, highlighting the need for context-aware evaluation.
• The visual analytics tool successfully calibrates model evaluation by allowing users to prioritize difficulty dimensions relevant to their application, improving decision-making efficiency.