[Paper Review] Kernel machines that adapt to GPUs for effective large batch training

Siyuan Ma, Mikhail A. Belkin|arXiv (Cornell University)|Jun 15, 2018

Advanced Neural Network Applications30 references4 citations

TL;DR

This paper introduces EigenPro 2.0, a principled framework that modifies classical kernel machines to enable extended linear scaling on GPUs by analytically adapting the kernel to computational resources. The method achieves near-instant training—under an hour on a single Titan Xp GPU for ImageNet with 1.3 million samples—without altering the model’s predictive function, enabling fast, interactive machine learning with minimal hyperparameter tuning.

ABSTRACT

Modern machine learning models are typically trained using Stochastic Gradient Descent (SGD) on massively parallel computing resources such as GPUs. Increasing mini-batch size is a simple and direct way to utilize the parallel computing capacity. For small batch an increase in batch size results in the proportional reduction in the training time, a phenomenon known as linear scaling. However, increasing batch size beyond a certain value leads to no further improvement in training time. In this paper we develop the first analytical framework that extends linear scaling to match the parallel computing capacity of a resource. The framework is designed for a class of classical kernel machines. It automatically modifies a standard kernel machine to output a mathematically equivalent prediction function, yet allowing for extended linear scaling, i.e., higher effective parallelization and faster training time on given hardware. The resulting algorithms are accurate, principled and very fast. For example, using a single Titan Xp GPU, training on ImageNet with $1.3 imes 10^6$ data points and $1000$ labels takes under an hour, while smaller datasets, such as MNIST, take seconds. As the parameters are chosen analytically, based on the theoretical bounds, little tuning beyond selecting the kernel and the kernel parameter is needed, further facilitating the practical use of these methods.

Motivation & Objective

• To address the fundamental limit of linear scaling in large-batch training of kernel machines, where training time plateaus beyond a critical batch size $m^*$.
• To develop a method that extends $m^*$ to match the parallel computing capacity of modern GPUs without changing the learned predictor function.
• To enable fast, interactive, and principled kernel learning by making optimization parameters analytically derived from theoretical bounds.
• To reduce reliance on heuristic tuning by providing a framework that automatically adapts to hardware, improving training efficiency and scalability.

Proposed method

• The framework modifies a standard kernel machine by introducing a data- and resource-dependent kernel that preserves the mathematical equivalence of the original prediction function.
• It extracts limited second-order information to reweight the optimization process, enabling higher effective batch sizes and extended linear scaling.
• The method uses an iterative optimization scheme (EigenPro iteration) that is analytically tuned based on theoretical bounds, minimizing the need for hyperparameter search.
• The framework supports efficient implementation on GPUs by reducing memory and computational costs through kernel reweighting and batch processing.
• It leverages the Laplacian kernel for improved robustness and larger $m^*$, enabling better parallelization compared to Gaussian kernels.
• Dimensionality reduction via PCA is integrated as a practical acceleration technique, reducing feature space without significant accuracy loss.

Experimental results

Research questions

• RQ1 Can the limit of linear scaling in kernel machine training be analytically extended to match the parallel capacity of modern GPUs?
• RQ2 How can a kernel machine be modified to maintain the same predictive function while enabling significantly higher batch sizes and faster training?
• RQ3 What role does the choice of kernel (e.g., Laplacian vs. Gaussian) play in determining the effective batch size $m^*$ and training speed?
• RQ4 To what extent can dimensionality reduction via PCA accelerate kernel training without degrading model performance?
• RQ5 Can the optimization process be made analytically principled and nearly tuning-free while achieving state-of-the-art training speeds?

Key findings

• The proposed framework extends linear scaling beyond the standard $m^*$ limit, enabling training time to decrease proportionally with batch size even at very large scales.
• On a single Titan Xp GPU, training on ImageNet with 1.3 million samples and 1000 labels completes in under one hour, demonstrating high scalability.
• For smaller datasets like MNIST, training completes in under 5 seconds, enabling interactive and exploratory machine learning workflows.
• The Laplacian kernel yields a larger $m^*$ than the Gaussian kernel, allowing for more effective parallelization and reduced need for hyperparameter tuning.
• Using PCA to reduce feature dimensionality from 1536 to 500 on ImageNet reduced training cost significantly with less than 0.2% accuracy drop.
• The method outperforms LibSVM and ThunderSVM in training speed—achieving comparable or better test accuracy in a fraction of the time—while requiring minimal tuning.