[Paper Review] Leakage and the Reproducibility Crisis in ML-based Science

Sayash Kapoor, Arvind Narayanan|arXiv (Cornell University)|Jul 14, 2022

Explainable Artificial Intelligence (XAI)139 citations

TL;DR

The paper surveys reproducibility failures in ML-based science due to data leakage across 17 fields, introduces a fine-grained leakage taxonomy, and proposes model info sheets to detect leakage, backed by a civil war prediction case study showing ML models do not outperform Logistic Regression when leakage is corrected.

ABSTRACT

The use of machine learning (ML) methods for prediction and forecasting has become widespread across the quantitative sciences. However, there are many known methodological pitfalls, including data leakage, in ML-based science. In this paper, we systematically investigate reproducibility issues in ML-based science. We show that data leakage is indeed a widespread problem and has led to severe reproducibility failures. Specifically, through a survey of literature in research communities that adopted ML methods, we find 17 fields where errors have been found, collectively affecting 329 papers and in some cases leading to wildly overoptimistic conclusions. Based on our survey, we present a fine-grained taxonomy of 8 types of leakage that range from textbook errors to open research problems. We argue for fundamental methodological changes to ML-based science so that cases of leakage can be caught before publication. To that end, we propose model info sheets for reporting scientific claims based on ML models that would address all types of leakage identified in our survey. To investigate the impact of reproducibility errors and the efficacy of model info sheets, we undertake a reproducibility study in a field where complex ML models are believed to vastly outperform older statistical models such as Logistic Regression (LR): civil war prediction. We find that all papers claiming the superior performance of complex ML models compared to LR models fail to reproduce due to data leakage, and complex ML models don't perform substantively better than decades-old LR models. While none of these errors could have been caught by reading the papers, model info sheets would enable the detection of leakage in each case.

Motivation & Objective

• Show that data leakage is a widespread driver of irreproducible results in ML-based science.
• Provide a fine-grained taxonomy of data leakage types relevant to scientific claims.
• Propose model info sheets to detect and prevent leakage in ML-based scientific reporting.
• Empirically assess leakage impact through a civil war prediction case study.

Proposed method

• Systematic literature survey across 20 papers in 17 fields to identify leakage-related pitfalls and quantify affected work.
• Develop a fine-grained taxonomy of 8 leakage types spanning data collection, preprocessing, modeling, and evaluation.
• Propose model info sheets as a reporting instrument to force explicit leakage-centered arguments (train-test separation, feature legitimacy, distributional matching).
• Conduct a reproducibility study in civil war prediction by reanalyzing 12 papers that used train-test splits and had code/data available, correcting leakage errors.

Experimental results

Research questions

• RQ1 How prevalent is data leakage as a cause of irreproducible results in ML-based science across multiple disciplines?
• RQ2 What are the distinct leakage modalities that affect ML-based scientific claims, and how can they be detected and mitigated?
• RQ3 Can model info sheets reliably reveal or prevent leakage across fields, as demonstrated in a civil war prediction case study?
• RQ4 Do complex ML models provide substantive advantages over Logistic Regression once leakage is addressed?

Key findings

• Data leakage is a pervasive pitfall across 17 fields, affecting 329 papers.
• The authors identify 8 leakage types, ranging from textbook errors to distributional misalignment and nonindependence.
• Mitigations from engineering/modeling competitions do not directly translate to ML-based science.
• Model info sheets can detect leakage and are necessary because reading papers alone cannot reveal leakage.
• In civil war prediction, papers claiming superiority of complex ML models over Logistic Regression fail to reproduce due to leakage; after correction, complex models do not perform substantially better.
• Uncertainty quantification and significance testing are often missing in papers comparing ML models.