[Paper Review] Gradient-Based Empirical Risk Minimization using Local Polynomial Regression

Ali Jadbabaie, Anuran Makur+1 more|arXiv (Cornell University)|Nov 4, 2020

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper proposes a novel gradient-based empirical risk minimization algorithm that leverages local polynomial regression to exploit smoothness of loss functions in data, achieving improved oracle complexity over both batch and stochastic gradient descent in low-data-dimensional regimes. The method learns gradients via local interpolation, yielding an oracle complexity of $\tilde{O}((p\epsilon^{-1})^{d/(2\eta)})$, which outperforms GD and SGD when the data dimension $d$ is small and the loss is $\eta$-Hölder smooth in data.

ABSTRACT

In this paper, we consider the problem of empirical risk minimization (ERM) of smooth, strongly convex loss functions using iterative gradient-based methods. A major goal of this literature has been to compare different algorithms, such as gradient descent (GD) or stochastic gradient descent (SGD), by analyzing their rates of convergence to $ε$-approximate solutions. For example, the oracle complexity of GD is $O(n\log(ε^{-1}))$, where $n$ is the number of training samples. When $n$ is large, this can be expensive in practice, and SGD is preferred due to its oracle complexity of $O(ε^{-1})$. Such standard analyses only utilize the smoothness of the loss function in the parameter being optimized. In contrast, we demonstrate that when the loss function is smooth in the data, we can learn the oracle at every iteration and beat the oracle complexities of both GD and SGD in important regimes. Specifically, at every iteration, our proposed algorithm performs local polynomial regression to learn the gradient of the loss function, and then estimates the true gradient of the ERM objective function. We establish that the oracle complexity of our algorithm scales like $ ilde{O}((p ε^{-1})^{d/(2η)})$ (neglecting sub-dominant factors), where $d$ and $p$ are the data and parameter space dimensions, respectively, and the gradient of the loss function belongs to a $η$-Hölder class with respect to the data. Our proof extends the analysis of local polynomial regression in non-parametric statistics to provide interpolation guarantees in multivariate settings, and also exploits tools from the inexact GD literature. Unlike GD and SGD, the complexity of our method depends on $d$ and $p$. However, when $d$ is small and the loss function exhibits modest smoothness in the data, our algorithm beats GD and SGD in oracle complexity for a very broad range of $p$ and $ε$.

Motivation & Objective

• Address the limitation of standard gradient-based ERM methods that ignore smoothness of loss functions in data, despite its prevalence in practical ML problems.
• Overcome the high oracle complexity of batch gradient descent (GD) for large $n$, and the suboptimal convergence of stochastic gradient descent (SGD) in terms of $\epsilon$-dependence.
• Develop a new optimization framework that exploits smoothness of the loss function with respect to training data, not just model parameters.
• Establish theoretical guarantees on convergence rates and oracle complexity under realistic smoothness assumptions in multivariate, non-parametric settings.
• Provide a rigorous analysis of local polynomial regression in multivariate settings with supremum norm control, enabling interpolation-based gradient estimation.

Proposed method

• At each iteration, perform local polynomial regression on a virtual batch of data points to estimate the gradient of the loss function with respect to the data.
• Use the estimated local gradient to construct a proxy for the true gradient of the empirical risk minimization (ERM) objective function.
• Apply inexact gradient descent with the interpolated gradient estimate, treating the local regression as a surrogate oracle.
• Leverage multivariate local polynomial interpolation with supremum norm guarantees to ensure accurate gradient approximation under Hölder smoothness assumptions.
• Integrate tools from inexact gradient descent theory to bound the convergence error and derive oracle complexity bounds.
• Assume the loss function is $\eta$-Hölder smooth in the data (not the parameters), enabling the method to exploit data smoothness for faster convergence.

Experimental results

Research questions

• RQ1 Can we improve the oracle complexity of ERM beyond GD and SGD by exploiting smoothness of the loss function in the data?
• RQ2 What is the theoretical convergence rate of an algorithm that learns gradients via local polynomial regression in data space?
• RQ3 How does the oracle complexity of the proposed method scale with data dimension $d$, parameter dimension $p$, and desired accuracy $\epsilon$?
• RQ4 Under what conditions does the proposed method outperform both GD and SGD in terms of gradient query efficiency?
• RQ5 Can we establish rigorous supremum norm error bounds for multivariate local polynomial regression under Hölder smoothness assumptions?

Key findings

• The proposed LIP-GD algorithm achieves an oracle complexity of $\tilde{O}((p\epsilon^{-1})^{d/(2\eta)})$, which depends on both data dimension $d$ and parameter dimension $p$, unlike GD and SGD.
• For small data dimension $d$ and moderate smoothness $\eta = \Theta(d)$, the method outperforms both GD and SGD across a broad range of $p$ and $\epsilon$.
• The oracle complexity of LIP-GD is strictly better than GD and SGD in the limit as $n \to \infty$, under the condition $\tau > \max\{1, \alpha^{-1}\}$, where $\tau$ controls the rate of convergence.
• The analysis provides new supremum norm guarantees for multivariate local polynomial regression, extending non-parametric statistics to high-dimensional interpolation with error control.
• The method's convergence is established using tools from inexact gradient descent, with error bounds derived via Hölder continuity of the loss gradient in data.
• The theoretical framework demonstrates that data-smoothness, often ignored in standard ERM analysis, can be leveraged to design provably faster optimization algorithms.