[Paper Review] Modeling Generalization in Machine Learning: A Methodological and Computational Study

Pietro Barbiero, Giovanni Squillero+1 more|arXiv (Cornell University)|Jun 28, 2020

Machine Learning and Data Classification44 references28 citations

TL;DR

This study analyzes 109 public classification datasets to model machine learning generalization, focusing on how data set characteristics influence model performance. It demonstrates that the convex hull of training data is a critical factor in distinguishing interpolation from extrapolation, revealing that dimensionality has surprisingly weak correlation with generalization—challenging the conventional 'curse of dimensionality' assumption and suggesting that high-capacity models generalize well even in high-dimensional spaces.

ABSTRACT

As machine learning becomes more and more available to the general public, theoretical questions are turning into pressing practical issues. Possibly, one of the most relevant concerns is the assessment of our confidence in trusting machine learning predictions. In many real-world cases, it is of utmost importance to estimate the capabilities of a machine learning algorithm to generalize, i.e., to provide accurate predictions on unseen data, depending on the characteristics of the target problem. In this work, we perform a meta-analysis of 109 publicly-available classification data sets, modeling machine learning generalization as a function of a variety of data set characteristics, ranging from number of samples to intrinsic dimensionality, from class-wise feature skewness to $F1$ evaluated on test samples falling outside the convex hull of the training set. Experimental results demonstrate the relevance of using the concept of the convex hull of the training data in assessing machine learning generalization, by emphasizing the difference between interpolated and extrapolated predictions. Besides several predictable correlations, we observe unexpectedly weak associations between the generalization ability of machine learning models and all metrics related to dimensionality, thus challenging the common assumption that the extit{curse of dimensionality} might impair generalization in machine learning.

Motivation & Objective

• To investigate which data set characteristics correlate with machine learning generalization performance.
• To assess whether the convex hull of training data can serve as a reliable proxy for distinguishing interpolation from extrapolation in ML predictions.
• To challenge the widely held belief that high dimensionality inherently impairs generalization in machine learning.
• To develop a meta-model that predicts generalization ability based on data set characteristics, particularly focusing on in- and out-of-hull predictions.

Proposed method

• The authors performed a meta-analysis on 109 publicly available classification datasets from curated sources like OpenML.
• They computed a range of data set characteristics, including sample size, number of features, class-wise feature skewness, and intrinsic dimensionality.
• The convex hull of the training set was computed to classify test points as either inside (interpolation) or outside (extrapolation) the hull.
• State-of-the-art classifiers (e.g., Logistic Regression, SVM, Random Forest) were trained and evaluated on both in-hull and out-of-hull test points.
• Symbolic regression was used to model associations between data set characteristics and model performance metrics such as F1-score.
• Pareto front comparisons were conducted to assess the relative impact of data set properties on model performance inside versus outside the convex hull.

Experimental results

Research questions

• RQ1 How do data set characteristics correlate with the generalization performance of machine learning models?
• RQ2 To what extent does the convex hull of the training data predict the generalization ability of a model?
• RQ3 Is there a significant relationship between dimensionality and generalization performance, as implied by the 'curse of dimensionality'?
• RQ4 How do different machine learning models (e.g., LR, SVC, RF) differ in their ability to generalize based on data set characteristics?
• RQ5 Can data set characteristics reliably predict whether a model will generalize well on in-hull versus out-of-hull test points?

Key findings

• The convex hull of the training data is a strong predictor of generalization, with models performing significantly better on in-hull (interpolated) predictions than on out-of-hull (extrapolated) ones.
• The study found unexpectedly weak correlations between generalization performance and all dimensionality-related metrics, challenging the assumption that high dimensionality inherently harms generalization.
• High-capacity models like Random Forest showed more robust generalization across both in-hull and out-of-hull regions, suggesting they are less sensitive to data set-specific characteristics.
• Predicting interpolation performance (F1_in) from data set characteristics was feasible and well-modeled, while predicting extrapolation performance (F1_out) was significantly harder.
• The intrinsic dimensionality ratio and class-wise feature correlation showed a weak positive correlation (ρ = 0.45), indicating limited influence of feature redundancy on generalization.
• The results suggest that real-world data sets may be a non-representative subset of all possible data sets, potentially explaining why ML models generalize better than theoretical models predict.