[Paper Review] A Constructive Prediction of the Generalization Error Across Scales

Jonathan S. Rosenfeld, Amir Rosenfeld+2 more|arXiv (Cornell University)|Sep 27, 2019

Advanced Neural Network Applications47 references49 citations

TL;DR

The paper proposes a simple, predictive functional form for generalization error that depends on both model size and data size, enabling extrapolation from small to large scales under a fixed scaling policy.

ABSTRACT

The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.

Motivation & Objective

• Understand how generalization error depends jointly on model size and dataset size under scaling policies.
• Propose a simple, interpretable functional form that fits error landscapes across vision and language tasks.
• Show that the proposed form can interpolate and extrapolate generalization error across scales with high accuracy.

Proposed method

• Empirically analyze error landscapes across multiple datasets and models in vision and language to identify scaling patterns.
• Propose a parametric error function that combines power-law terms in data and model size with a transition envelope.
• Fit the function to small-scale measurements using 10-fold cross-validation and evaluate fit quality across datasets.
• Demonstrate extrapolation from seen to unseen large-scale configurations and assess predictive accuracy.

Experimental results

Research questions

• RQ1 What is the functional relation between generalization error, model size, and dataset size under a given scaling policy?
• RQ2 Can a simple parametric form accurately fit and predict the error landscape across both vision and language tasks?
• RQ3 To what extent can small-scale measurements predict large-scale performance under the proposed scaling policy?
• RQ4 How well does the model generalize to unseen (larger) scales when fit on smaller-scale data?

Key findings

• The error landscape exhibits regions where power-law behavior approximates the error with respect to both data size and model size (holding the other fixed).
• A proposed function epsilon(m, n) ≈ a(m)n^(-α(m)) + b(n)m^(-β(n)) + c_infty captures the joint dependence of error on model and data sizes.
• An envelope allows smooth transition from an initial random-guess error to the power-law regime, enabling a practical fit across scales.
• Across datasets, the mean divergence of the estimated vs. actual error is under 1% with standard deviation under 5% in most cases.
• Extrapolation from small to large scales achieves mean divergences around a few percent (e.g., 4.5% on ImageNet, 0.5% on WikiText-103) under various configurations.
• The approach remains accurate across multiple architectures and optimizers, including ResNet, WRN, VGG, DenseNet, AWD-LSTM, and Transformer variants.