[Paper Review] Efficient Hyperparameter Tuning for Large Scale Kernel Ridge Regression

Giacomo Meanti, Luigi Carratino+2 more|arXiv (Cornell University)|Jan 17, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes an efficient, data-dependent complexity regularization criterion for hyperparameter tuning in large-scale kernel ridge regression using the Nyström approximation. By leveraging a data-driven penalty that accounts for variance from stochastic data, the method enables stable, automated tuning of regularization, kernel, and Nyström center parameters—demonstrated to outperform existing strategies in extensive experiments and integrated into the Falkon library for scalable kernel methods.

ABSTRACT

Kernel methods provide a principled approach to nonparametric learning. While their basic implementations scale poorly to large problems, recent advances showed that approximate solvers can efficiently handle massive datasets. A shortcoming of these solutions is that hyperparameter tuning is not taken care of, and left for the user to perform. Hyperparameters are crucial in practice and the lack of automated tuning greatly hinders efficiency and usability. In this paper, we work to fill in this gap focusing on kernel ridge regression based on the Nyström approximation. After reviewing and contrasting a number of hyperparameter tuning strategies, we propose a complexity regularization criterion based on a data dependent penalty, and discuss its efficient optimization. Then, we proceed to a careful and extensive empirical evaluation highlighting strengths and weaknesses of the different tuning strategies. Our analysis shows the benefit of the proposed approach, that we hence incorporate in a library for large scale kernel methods to derive adaptively tuned solutions.

Motivation & Objective

• Address the lack of automated hyperparameter tuning in large-scale kernel methods, particularly for kernel ridge regression with Nyström approximation.
• Overcome the limitations of existing tuning strategies that are either computationally prohibitive or restricted to few hyperparameters.
• Develop a theoretically grounded, efficient criterion for tuning regularization, kernel, and Nyström center hyperparameters in large-scale settings.
• Provide a practical implementation that integrates adaptive tuning into the FALKON library for scalable kernel methods.
• Conduct a comprehensive empirical evaluation comparing multiple tuning strategies to validate the proposed method’s stability and performance.

Proposed method

• Propose a novel complexity regularization criterion based on a data-dependent bound that separates sources of variance due to data stochasticity.
• Formulate the hyperparameter tuning problem as an optimization of this criterion, which is shown to improve stability compared to standard methods.
• Design an efficient algorithm to compute the data-dependent penalty using trace estimation via randomized linear algebra techniques.
• Integrate the proposed criterion into the FALKON framework, enabling scalable, differentiable hyperparameter optimization for large-scale kernel methods.
• Use efficient solvers and GPU acceleration to handle billions of data points while maintaining low computational cost for tuning.
• Implement the method in a production-ready library (Falkon) with support for joint tuning of regularization, kernel, and Nyström center parameters.

Experimental results

Research questions

• RQ1 How can hyperparameter tuning be made both efficient and stable in large-scale kernel ridge regression with Nyström approximation?
• RQ2 What is the impact of using a data-dependent penalty in complexity regularization on the stability and generalization of hyperparameter selection?
• RQ3 How does the proposed criterion compare empirically to established methods like cross-validation, generalized cross-validation, and trace-based estimators in terms of performance and computational cost?
• RQ4 Can the proposed method scale to massive datasets (e.g., billions of points) while maintaining accuracy and efficiency?
• RQ5 What are the relative strengths and weaknesses of different hyperparameter tuning strategies when applied to large-scale kernel methods?

Key findings

• The proposed data-dependent complexity regularization criterion leads to significantly more stable hyperparameter tuning compared to standard methods like generalized cross-validation or trace-based estimators.
• The method achieves competitive generalization performance on large-scale datasets, including those with up to one billion points, while maintaining low computational overhead.
• Empirical evaluation shows that the proposed approach outperforms baseline tuning strategies in terms of test error and robustness across diverse data distributions.
• The integration of the method into the FALKON library enables end-to-end, adaptive tuning of multiple hyperparameters (regularization, kernel, Nyström centers) in a single, scalable pipeline.
• Trace estimation techniques enable efficient computation of the data-dependent penalty, making the method practical for real-world, large-scale applications.
• The approach demonstrates superior stability under varying data sampling and noise conditions, confirming the theoretical benefits of separating variance sources in the penalty.