[Paper Review] On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations

Roy Schwartz, Gabriel Stanovsky|arXiv (Cornell University)|Apr 27, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper argues that dataset balancing—while helpful for reducing some spurious correlations—is fundamentally insufficient to eliminate all spurious correlations in NLP, especially as models grow more powerful. It proposes shifting focus from large-scale fine-tuning to zero- or few-shot learning, richer context modeling, and model abstention in uncertainty, to better handle world knowledge and common sense without discarding useful signal.

ABSTRACT

Recent work has shown that deep learning models in NLP are highly sensitive to low-level correlations between simple features and specific output labels, leading to overfitting and lack of generalization. To mitigate this problem, a common practice is to balance datasets by adding new instances or by filtering out "easy" instances (Sakaguchi et al., 2020), culminating in a recent proposal to eliminate single-word correlations altogether (Gardner et al., 2021). In this opinion paper, we identify that despite these efforts, increasingly-powerful models keep exploiting ever-smaller spurious correlations, and as a result even balancing all single-word features is insufficient for mitigating all of these correlations. In parallel, a truly balanced dataset may be bound to "throw the baby out with the bathwater" and miss important signal encoding common sense and world knowledge. We highlight several alternatives to dataset balancing, focusing on enhancing datasets with richer contexts, allowing models to abstain and interact with users, and turning from large-scale fine-tuning to zero- or few-shot setups.

Motivation & Objective

• To challenge the assumption that dataset balancing effectively eliminates spurious correlations in NLP.
• To demonstrate that even single-word feature balancing fails to eliminate all spurious correlations due to higher-order feature interactions.
• To argue that over-balancing datasets risks discarding essential world knowledge and common sense, harming model robustness.
• To advocate for alternatives such as richer context injection, model abstention, and zero/few-shot learning to improve generalization.
• To question the sustainability and desirability of large-scale fine-tuning as the dominant paradigm in NLP.

Proposed method

• Analyzes the limitations of dataset balancing by extending prior work that treats all single-word correlations as spurious.
• Demonstrates that balancing individual features is insufficient because spurious correlations emerge from combinations of features, not just unigrams.
• Proposes enhancing datasets with richer contextual information to encode world knowledge and common sense, reducing reliance on superficial cues.
• Advocates for models that can abstain from answering when context is insufficient, rather than defaulting to spurious correlations.
• Recommends shifting from large-scale fine-tuning to zero- or few-shot learning setups to reduce overfitting to dataset artifacts.
• Recommends building large test sets to improve statistical power and better evaluate model generalization, rather than relying on small test sets.

Experimental results

Research questions

• RQ1 Can dataset balancing effectively eliminate all spurious correlations in NLP, even as models grow more powerful?
• RQ2 What are the unintended consequences of over-balancing datasets, particularly in terms of discarding useful world knowledge?
• RQ3 Why is balancing single-word features insufficient to address spurious correlations that arise from feature combinations?
• RQ4 How can models be designed to handle uncertainty and avoid defaulting to spurious correlations when context is ambiguous?
• RQ5 Is large-scale fine-tuning the optimal paradigm for training NLP models, or should the field shift toward zero- or few-shot learning?

Key findings

• Balancing single-word features is insufficient to eliminate all spurious correlations, as higher-order feature interactions still enable model exploitation.
• Over-balancing datasets risks removing meaningful signal, including common sense and world knowledge, which are essential for robust inference.
• Models trained on heavily balanced datasets may fail to learn fallback answers for world knowledge facts, such as 'Joe Biden is the president of the U.S.'
• Dataset balancing is not a panacea for mitigating social biases, and may even be insufficient for this purpose due to the complexity of spurious correlations.
• The current arms race between model scale and dataset curation is unsustainable and counterproductive, as models continue to exploit increasingly subtle correlations.
• Shifting toward zero- or few-shot learning and richer context modeling offers a more sustainable and effective path to improving model generalization and robustness.