[Paper Review] The Landscape of Empirical Risk for Non-convex Losses

Mei Song, Yu Bai+1 more|arXiv (Cornell University)|Jul 22, 2016

Machine Learning and Algorithms44 references59 citations

TL;DR

This paper establishes uniform convergence of the gradient and Hessian of empirical risk to their population counterparts for non-convex losses, enabling a one-to-one correspondence between stationary points of the empirical and population risks. It demonstrates that under mild sample size conditions (n ≳ p log n), descent algorithms converge to global minima in problems like non-convex binary classification, robust regression, and Gaussian mixture models.

ABSTRACT

Most high-dimensional estimation and prediction methods propose to minimize a cost function (empirical risk) that is written as a sum of losses associated to each data point. In this paper we focus on the case of non-convex losses, which is practically important but still poorly understood. Classical empirical process theory implies uniform convergence of the empirical risk to the population risk. While uniform convergence implies consistency of the resulting M-estimator, it does not ensure that the latter can be computed efficiently. In order to capture the complexity of computing M-estimators, we propose to study the landscape of the empirical risk, namely its stationary points and their properties. We establish uniform convergence of the gradient and Hessian of the empirical risk to their population counterparts, as soon as the number of samples becomes larger than the number of unknown parameters (modulo logarithmic factors). Consequently, good properties of the population risk can be carried to the empirical risk, and we can establish one-to-one correspondence of their stationary points. We demonstrate that in several problems such as non-convex binary classification, robust regression, and Gaussian mixture model, this result implies a complete characterization of the landscape of the empirical risk, and of the convergence properties of descent algorithms. We extend our analysis to the very high-dimensional setting in which the number of parameters exceeds the number of samples, and provide a characterization of the empirical risk landscape under a nearly information-theoretically minimal condition. Namely, if the number of samples exceeds the sparsity of the unknown parameters vector (modulo logarithmic factors), then a suitable uniform convergence result takes place. We apply this result to non-convex binary classification and robust regression in very high-dimension.

Motivation & Objective

• To understand the computational complexity of M-estimators in high-dimensional non-convex settings where classical convexity assumptions fail.
• To characterize the landscape of empirical risk—specifically, stationary points and their stability—for non-convex loss functions.
• To establish conditions under which descent algorithms converge to global minima despite non-convexity.
• To extend these results to the high-dimensional regime where p ≫ n, under sparsity assumptions.
• To provide a theoretical foundation for the empirical success of non-convex optimization in problems like robust regression and mixture models.

Proposed method

• Propose a framework to study the landscape of empirical risk via uniform convergence of gradient and Hessian to their population counterparts.
• Use empirical process theory to show that if n ≳ p log n, the empirical risk inherits the geometric properties of the population risk.
• Establish a one-to-one correspondence between stationary points of the empirical and population risks under mild regularity conditions.
• Apply the framework to three canonical problems: non-convex binary classification, robust regression with non-convex ρ-functions, and Gaussian mixture models.
• Extend analysis to high-dimensional settings by assuming sparsity and showing uniform convergence when n ≳ s log n, where s is the sparsity of the true parameter.
• Leverage trust region methods to prove global convergence to a global minimum under the derived landscape properties.

Experimental results

Research questions

• RQ1 Under what conditions does the empirical risk landscape mirror the population risk landscape in non-convex M-estimation?
• RQ2 Can descent algorithms like gradient descent or trust region methods globally converge to a global minimum in non-convex problems?
• RQ3 How does the sample size n relate to the number of parameters p (or sparsity s) to ensure that empirical risk inherits favorable geometric properties of the population risk?
• RQ4 What is the role of uniform convergence of gradient and Hessian in establishing convergence guarantees for non-convex optimization?
• RQ5 In high-dimensional settings with p ≫ n, can we still achieve global convergence for non-convex M-estimators under sparsity assumptions?

Key findings

• When n ≳ p log n, the gradient and Hessian of the empirical risk uniformly converge to those of the population risk, ensuring a one-to-one correspondence of stationary points.
• For non-convex binary classification with squared loss, the empirical risk landscape has exactly two local minima near the true parameters, and descent methods converge to one of them.
• In robust regression with non-convex ρ-functions, the empirical risk landscape inherits the absence of spurious local minima under the same sample size condition.
• For Gaussian mixture models, the empirical risk has three stationary points: two local minima near the true component means and one saddle point at the origin, with trust region methods converging to a global minimum.
• In the high-dimensional regime with p ≫ n, if the true parameter is s-sparse and n ≳ s log n, uniform convergence of gradient and Hessian still holds, enabling global convergence of descent algorithms.
• Trust region methods converge to a global minimum for the Gaussian mixture model when initialized within a neighborhood of the origin, provided n ≳ d log d.