[Paper Review] Underspecification Presents Challenges for Credibility in Modern Machine Learning

Alexander D’Amour, Katherine Heller+38 more|arXiv (Cornell University)|Nov 6, 2020

Machine Learning in Healthcare117 references430 citations

TL;DR

The paper argues that underspecification in ML pipelines causes predictors with similar iid performance to behave very differently in deployment, and it provides stress-testing evidence across multiple domains to motivate disciplined evaluation and design.

ABSTRACT

ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We show that this problem appears in a wide variety of practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

Motivation & Objective

• Define underspecification in ML pipelines and explain its impact on deployment credibility.
• Show that near-iid-optimal predictors can encode different inductive biases with divergent deployment behavior.
• Demonstrate underspecification empirically across computer vision, medical imaging, NLP, EHR-based prediction, and genomics.
• Propose stress-testing and constraints as remedies to ensure credible inductive biases for real-world deployment.

Proposed method

• Formalize the concept of underspecification in ML pipelines as multiple predictors achieving near-optimal iid performance.
• Use simple analytical models (epidemiology, random feature model, polygenic risk scores) to illustrate how different predictors with similar training performance yield different deployment outcomes.
• Apply a stress-testing protocol consisting of stratified, shifted, and contrastive evaluations to production-grade deep learning pipelines across domains.
• Document empirical evidence of underspecification across computer vision, medical imaging, NLP, and electronic health records.
• Argue for training and evaluation techniques that constrain pipelines toward credible inductive biases without sacrificing iid performance.

Experimental results

Research questions

• RQ1 What is underspecification in ML pipelines and how does it affect deployment credibility?
• RQ2 Can predictors with similar iid performance diverge in deployment due to different inductive biases?
• RQ3 How can stress tests reveal underspecification in diverse ML applications?

Key findings

• Underspecification is widespread in modern ML and leads to deployment-sensitive behavior not captured by iid evaluation.
• Stress tests (stratified, shifted, and contrastive evaluations) reveal variability in predictor behavior that standard iid tests miss.
• Different predictors with near-identical iid risk can exhibit substantially different risks under distribution shifts or adversarial shifts.
• Even when iid performance is preserved, some predictors become vulnerable to specific shifts, undermining credibility.
• The problem persists across domains: computer vision, medical imaging, NLP, EHR-based risk prediction, and medical genomics.
• Addressing underspecification through targeted training/evaluation strategies can improve credibility without necessarily harming iid performance.