[Paper Review] Learning in High Dimension Always Amounts to Extrapolation

Randall Balestriero, Jerome Pesenti+1 more|arXiv (Cornell University)|Oct 18, 2021

Sparse and Compressive Sensing TechniquesEngineering23 references26 citations

TL;DR

The paper argues that in high-dimensional spaces (d > 100), new samples almost surely lie outside the convex hull of training data, meaning interpolation is vanishingly rare and extrapolation dominates, challenging traditional links between interpolation and generalization.

ABSTRACT

The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation. Interpolation occurs for a sample $x$ whenever this sample falls inside or on the boundary of the given dataset's convex hull. Extrapolation occurs when $x$ falls outside of that convex hull. One fundamental (mis)conception is that state-of-the-art algorithms work so well because of their ability to correctly interpolate training data. A second (mis)conception is that interpolation happens throughout tasks and datasets, in fact, many intuitions and theories rely on that assumption. We empirically and theoretically argue against those two points and demonstrate that on any high-dimensional ($>$100) dataset, interpolation almost surely never happens. Those results challenge the validity of our current interpolation/extrapolation definition as an indicator of generalization performances.

Motivation & Objective

• Demonstrate theoretically and empirically that interpolation almost surely does not occur in high-dimensional spaces (>100).
• Show that current models operate in an extrapolation regime regardless of data manifold intrinsic dimension.
• Investigate how dataset size, ambient/convex hull dimensions, and embeddings affect interpolation probability.
• Examine implications for how interpolation/extrapolation relates to generalization in modern ML.
• Provide guidance on geometric definitions of interpolation fitting high-dimensional data.

Proposed method

• Theoretical result (Theorem 1) shows that for i.i.d. uniform samples from a d-dimensional ball, the probability a new sample lies in the hull goes to 0 unless N grows exponentially with d.
• Empirical experiments assess interpolation probability on synthetic data with varying ambient and convex-hull dimensions, and on real datasets (MNIST, CIFAR, ImageNet) under different dimensionality reductions and embeddings.
• Experiments vary intrinsic manifold dimension, convex hull dimension d*, and observed dimensions to study how interpolation probability decays with dimensionality (and sample size).
• Analysis includes embedding-space and dimensionality-reduction perspectives to test whether interpolation persists in latent representations or after common reductions.
• Theoretical synthesis with existing results (e.g., Valtr’s formulas, Buchta’s limit, Kabluchko & Zaporozhets non-asymptotics) to characterize interpolation/extrapolation probabilities in high dimensions.

Experimental results

Research questions

• RQ1 Does interpolation probability for new samples vanish in high dimensions for realistic dataset sizes?
• RQ2 How do intrinsic dimension, ambient dimension, and convex hull (smallest affine subspace containing data) affect interpolation likelihood?
• RQ3 Do common embeddings and dimensionality-reduction techniques preserve interpolation/extrapolation information?
• RQ4 What implications do high-dimensional interpolation/extrapolation properties have for model generalization?
• RQ5 Can known theoretical results extend to real-world data distributions and embeddings?

Key findings

• Interpolation probability decays exponentially with dimensionality and requires exponentially many samples to maintain a constant interpolation probability.
• The smallest affine subspace containing the data (convex hull dimension d*) governs interpolation probability more than intrinsic manifold dimension; increasing ambient dimension with fixed d* does not help maintain interpolation.
• On real datasets (MNIST, CIFAR, ImageNet) and various embeddings, test samples fall in extrapolation regime with respect to training data as dimensions grow, indicating extrapolation dominates in practice.
• Dimensionality-reduction methods and many embeddings tend to erase interpolation/extrapolation distinctions, potentially misleading interpretations of data geometry.
• Johnson–Lindenstrauss-type reductions cannot preserve interpolation probabilities when N grows only linearly with d, reinforcing that high-dimensional interpolation is unlikely under practical data scales.