[Paper Review] Our Evaluation Metric Needs an Update to Encourage Generalization

Swaroop Mishra, Anjana Arunkumar+2 more|arXiv (Cornell University)|Jul 14, 2020

Adversarial Robustness in Machine Learning44 references4 citations

TL;DR

This paper proposes the WOOD Score, a novel evaluation metric that weights test samples by their semantic textual similarity (STS) to training data, increasing weight for more out-of-distribution (OOD) samples. By making models generalize to harder, more OOD samples to achieve high scores, WOOD reduces benchmark accuracy and curbs performance inflation, encouraging robustness over spurious pattern exploitation.

ABSTRACT

Models that surpass human performance on several popular benchmarks display significant degradation in performance on exposure to Out of Distribution (OOD) data. Recent research has shown that models overfit to spurious biases and `hack' datasets, in lieu of learning generalizable features like humans. In order to stop the inflation in model performance -- and thus overestimation in AI systems' capabilities -- we propose a simple and novel evaluation metric, WOOD Score, that encourages generalization during evaluation.

Motivation & Objective

• Address the overestimation of AI model capabilities due to performance inflation on standard benchmarks.
• Identify and quantify the degree of out-of-distribution (OOD) characteristics in test samples using Semantic Textual Similarity (STS) to training data.
• Develop an evaluation metric that actively encourages models to generalize by weighting harder, more OOD samples more heavily.
• Reduce benchmark accuracy inflation by shifting evaluation focus toward robustness on diverse, semantically dissimilar data.
• Provide a scalable, data-driven method to control OOD severity during evaluation without requiring external OOD datasets.

Proposed method

• Compute pairwise STS between each test sample and all training samples using pre-trained sentence embeddings.
• Sort test samples by their average STS with the top-b% most similar training samples to estimate OOD severity.
• Define a sample's OOD degree (p) as inversely proportional to the average STS with the most similar training samples, using hyperparameters a and b.
• Propose WOOD Score as a weighted evaluation metric: $ W_{opt} = abla_{X_{ ext{Test}}} E_i p_i $, where $ p_i $ increases with OOD severity.
• Define WOOD accuracy ($ W_{acc} $) as a weighted sum: $ A_1 + 2A_2 + 3A_3 $, where $ A_1, A_2, A_3 $ are accuracies on low-, medium-, and high-OOD-ness samples.
• Use STS-based splitting of test sets into seven hierarchical bins to evaluate model performance across varying degrees of distributional shift.

Experimental results

Research questions

• RQ1 Can Semantic Textual Similarity (STS) effectively distinguish between in-distribution (IID) and out-of-distribution (OOD) samples in NLP benchmarks?
• RQ2 To what extent does model performance degrade on OOD samples with increasing STS-based OOD severity, and can this be quantified?
• RQ3 Can a weighted evaluation metric that increases the importance of OOD samples reduce benchmark accuracy inflation and encourage generalization?
• RQ4 How does the WOOD Score perform across diverse models (e.g., BERT, RoBERTa, CNN, LSTM) and datasets (e.g., SST-2, IMDb) under varying OOD severity levels?
• RQ5 Can the proposed metric guide data augmentation and contrast set creation to improve model robustness?

Key findings

• STS between test samples and training data effectively differentiates OOD from IID samples, with lower STS corresponding to higher OOD severity.
• The WOOD Score significantly reduces benchmark accuracy across ten models on both SST-2 (IID) and IMDb (OOD), demonstrating a drop in performance due to increased weighting of harder samples.
• Models exhibit higher softmax probabilities on incorrect predictions when STS with training data is low, indicating that OOD samples are more likely to be misclassified.
• The number of incorrect classifications increases with decreasing STS, confirming that lower similarity correlates with higher prediction difficulty.
• The metric is robust across varying values of b (number of top training samples considered), with consistent performance trends observed in ablation studies.
• The WOOD Score enables the same dataset to serve as both IID and OOD evaluation set by hierarchically splitting based on STS, eliminating the need for external OOD data.