[Paper Review] Configuration-to-Performance Scaling Law with Neural Ansatz

Huaqing Zhang, Kaiyue Wen+1 more|arXiv (Cornell University)|Feb 10, 2026

Machine Learning in Materials Science0 citations

TL;DR

NCPL uses a fine-tuned language model to map full pretraining configurations to training outcomes, enabling accurate final-loss and loss-curve predictions and aiding joint hyperparameter tuning under constraints.

ABSTRACT

Researchers build scaling laws to forecast the training performance of expensive large-scale runs with larger model size N and data size D. These laws assume that other training hyperparameters are optimally chosen, which can require significant effort and, in some cases, be impossible due to external hardware constraints. To improve predictability across a broader set of hyperparameters and enable simpler tuning at scale, we propose learning a extit{Configuration-to-Performance Scaling Law} (CPL): a mapping from the extit{full training configuration} to training performance. Because no simple functional form can express this mapping, we parameterize it with a large language model (LLM), and fit it with diverse open-source pretraining logs across multiple sources, yielding a extit{Neural} Configuration-to-Performance Scaling Law (NCPL). NCPL accurately predicts how training configurations influence the final pretraining loss, achieving 20-40% lower prediction error than the configuration-agnostic Chinchilla law and generalizing to runs using up to 10 x more compute than any run in the training set. It further supports joint tuning of multiple hyperparameters with performance comparable to hyperparameter scaling law baselines. Finally, NCPL naturally and effectively extends to richer prediction targets such as loss-curve prediction.

Motivation & Objective

• Motivate the need for predicting pretrained model performance under diverse hyperparameters without exhaustive tuning.
• Propose a neural (LLM-based) Configuration-to-Performance Scaling Law (CPL) that maps configurations to performance.
• Demonstrate NCPL’s ability to predict final loss and loss curves, and enable joint hyperparameter optimization.
• Show generalization to out-of-distribution model sizes and extrapolation to larger compute.
• Highlight benefits and limitations of using open-source logs and foundation models for CPL.

Proposed method

• Parameterize CPL with a pretrained language model fine-tuned as a regressor f_theta to map full training configurations C to performance P.
• Use input features including source, architecture (N, layers, heads, hidden dim), data size D, optimizer and hyperparameters.
• Predict residuals relative to a Chinchilla-law baseline ell_chinchilla(N,D) and train with MSE on residual targets.
• Adopt a two-stage fine-tuning scheme (Stage 1: update numerical field encoder and head; Stage 2: fine-tune all parameters).
• Predict targets as (i) final pretraining loss and (ii) intermediate losses to reconstruct loss curves.
• Evaluate on in-distribution (ID) and out-of-distribution (OOD) splits using Marin and StepLaw datasets.
• Compare NCPL with XGBoost and Chinchilla-law baselines and perform ablations (backbone size, fine-tuning vs scratch).
• Demonstrate hyperparameter selection by sweeping configurations and comparing to a power-law baseline.

Figure 1 : An Overview of NCPL’s Performance Across Tasks. We split the collected pretraining logs by the model size. In-distribution (ID) means the model size is within the range of the model size in the training set used for NCPL and out-of-distribution (OOD) means the model size is larger. Left:

Experimental results

Research questions

• RQ1 Can a neural network, specifically a fine-tuned language model, learn the mapping from full training configurations to pretraining performance (C → P)?
• RQ2 Does NCPL improve prediction accuracy over configuration-agnostic scaling laws (e.g., Chinchilla) when forecasting final loss and loss curves?
• RQ3 Can NCPL enable joint hyperparameter tuning and outperform hand-designed hyperparameter scaling baselines in ID and OOD settings?
• RQ4 To what extent does NCPL generalize to larger compute beyond the training set, and to richer targets like loss curves?
• RQ5 What interactions between hyperparameters (e.g., optimizer and weight decay) can NCPL discover from open-source logs?

Key findings

• NCPL achieves lower prediction error and higher rank correlation than the Chinchilla baseline for final-loss prediction on both ID and OOD data.
• NCPL enables joint tuning of hyperparameters and attains competitive performance to dedicated hyperparameter scaling laws.
• NCPL can predict the entire loss curve, not just final loss, across optimizers and hyperparameter settings.
• NCPL qualitatively learns nontrivial interactions between hyperparameters, such as optimizer-specific weight-decay effects.
• Fine-tuning a foundation-model-based regressor yields stronger results on diverse, heterogeneous configurations than training-from-scratch or non-neural baselines.
• NCPL generalizes to OOD runs with up to 10x more compute than runs in the training set.

Figure 3 : Predicted loss vs. ground-truth loss. Each point visualizes the predicted vs. ground-truth final pretraining loss of an individual run from the Marin dataset (for StepLaw dataset, see Figure ˜ 1 left). NCPL uses the full training configuration as input, whereas the Chinchilla law only dep