[Paper Review] To what extent should we trust AI models when they extrapolate?

Roozbeh Yousefzadeh, Xuenan Cao|arXiv (Cornell University)|Jan 27, 2022

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper proposes a geometric auditing method to detect and report AI model extrapolation—where models make predictions outside the range of training data—demonstrating that such extrapolation is frequent, socially consequential, and often unnoticed. The approach uses convex hull analysis to identify when predictions rely on extrapolation, advocating for automated reporting of extrapolation extent and dimensions to improve transparency and accountability in high-stakes AI applications.

ABSTRACT

Many applications affecting human lives rely on models that have come to be known under the umbrella of machine learning and artificial intelligence. These AI models are usually complicated mathematical functions that map from an input space to an output space. Stakeholders are interested to know the rationales behind models' decisions and functional behavior. We study this functional behavior in relation to the data used to create the models. On this topic, scholars have often assumed that models do not extrapolate, i.e., they learn from their training samples and process new input by interpolation. This assumption is questionable: we show that models extrapolate frequently; the extent of extrapolation varies and can be socially consequential. We demonstrate that extrapolation happens for a substantial portion of datasets more than one would consider reasonable. How can we trust models if we do not know whether they are extrapolating? Given a model trained to recommend clinical procedures for patients, can we trust the recommendation when the model considers a patient older or younger than all the samples in the training set? If the training set is mostly Whites, to what extent can we trust its recommendations about Black and Hispanic patients? Which dimension (race, gender, or age) does extrapolation happen? Even if a model is trained on people of all races, it still may extrapolate in significant ways related to race. The leading question is, to what extent can we trust AI models when they process inputs that fall outside their training set? This paper investigates several social applications of AI, showing how models extrapolate without notice. We also look at different sub-spaces of extrapolation for specific individuals subject to AI models and report how these extrapolations can be interpreted, not mathematically, but from a humanistic point of view.

Motivation & Objective

• To challenge the common assumption that AI models primarily interpolate rather than extrapolate when making predictions.
• To demonstrate that extrapolation in AI models is frequent, non-negligible, and can have significant social consequences in healthcare, finance, and employment.
• To develop a systematic geometric auditing method that identifies and quantifies extrapolation in both continuous and categorical features.
• To advocate for integrating extrapolation reporting into AI pipelines to enhance transparency, especially under regulations like GDPR.
• To guide model selection by linking the extent of extrapolation to model complexity and performance trade-offs.

Proposed method

• Define the convex hull of the training data ($\mathcal{H}^{tr}$) as the geometric boundary of all training samples, where points inside are interpolated and points outside are extrapolated.
• For any test sample $x$, determine whether $x$ lies inside or outside $\mathcal{H}^{tr}$ to classify the prediction as interpolation or extrapolation.
• Use geometric decomposition to identify which features (e.g., age, BMI, race) contribute most to the extrapolation, enabling dimension-specific reporting.
• Apply approximation techniques (e.g., from Blum et al.) to estimate the convex hull when full training data access is restricted due to privacy concerns.
• Integrate the auditing procedure into AI pipelines to automatically log and report the extent and direction of extrapolation for each prediction.
• Propose that models with statistically excessive extrapolation be flagged for human review, especially in sensitive domains like healthcare and criminal justice.

Experimental results

Research questions

• RQ1 To what extent do AI models extrapolate in real-world applications such as healthcare, salary prediction, and algorithmic trading?
• RQ2 How can we systematically detect and quantify extrapolation in models with mixed feature types (continuous, categorical, integer)?
• RQ3 What are the social consequences of undetected extrapolation in AI models used for high-stakes decisions?
• RQ4 How does the extent of extrapolation relate to model complexity and performance, and can it inform model selection?
• RQ5 What auditing mechanisms can be built into AI pipelines to report extrapolation in a privacy-preserving and interpretable way?

Key findings

• AI models frequently extrapolate in real-world applications, with extrapolation occurring in a substantial portion of test samples—often more than expected.
• Extrapolation is not limited to continuous variables; it also occurs in categorical features such as race and gender, even when training data includes diverse groups.
• The extent of extrapolation varies significantly across individuals and can be socially consequential, such as when a model recommends a clinical procedure for a patient older than any in the training set.
• Models trained on diverse populations may still extrapolate significantly in race-related dimensions, indicating that representation alone does not prevent extrapolation.
• Deep learning models tend to exhibit higher extrapolation in tasks where they outperform simpler models, suggesting a link between model complexity and extrapolation risk.
• Automated extrapolation auditing—reporting the extent and feature dimensions of extrapolation—can enhance transparency and support regulatory compliance, such as under GDPR.