[Paper Review] A theory on the absence of spurious optimality

Cédric Josz, Yi Ouyang+3 more|arXiv (Cornell University)|May 21, 2018

Tensor decomposition and applications12 references2 citations

TL;DR

This paper introduces 'global functions'—continuous functions with no spurious local minima—and establishes their theoretical properties, including stability under composition and change of variables. The authors prove that certain nonconvex, nonsmooth tensor decomposition problems are global functions, providing the first theoretical guarantee that the ℓ₁ norm effectively avoids outliers in such settings.

ABSTRACT

We study the set of continuous functions that admit no spurious local optima (i.e. local minima that are not global minima) which we term extit{global functions}. They satisfy various powerful properties for analyzing nonconvex and nonsmooth optimization problems. For instance, they satisfy a theorem akin to the fundamental uniform limit theorem in the analysis regarding continuous functions. Global functions are also endowed with useful properties regarding the composition of functions and change of variables. Using these new results, we show that a class of nonconvex and nonsmooth optimization problems arising in tensor decomposition applications are global functions. This is the first result concerning nonconvex methods for nonsmooth objective functions. Our result provides a theoretical guarantee for the widely-used $\ell_1$ norm to avoid outliers in nonconvex optimization.

Motivation & Objective

• To identify and characterize continuous functions that lack spurious local minima, termed 'global functions'.
• To establish foundational theorems for global functions analogous to classical results in analysis, such as uniform limit theorems.
• To demonstrate that nonconvex, nonsmooth optimization problems in tensor decomposition belong to the class of global functions.
• To provide a theoretical justification for the effectiveness of ℓ₁ regularization in avoiding outliers in nonconvex optimization.

Proposed method

• Define global functions as continuous functions where every local minimum is a global minimum.
• Prove that global functions are closed under uniform limits, composition with continuous functions, and smooth change of variables.
• Apply the theory to analyze a class of nonconvex, nonsmooth optimization problems arising in tensor decomposition.
• Use the structural properties of global functions to show that ℓ₁-regularized tensor decomposition objectives are free of spurious local minima.
• Leverage the invariance of global functions under diffeomorphic transformations to extend results to reparameterized optimization problems.
• Establish a novel theoretical framework that generalizes classical convexity-like guarantees to nonconvex, nonsmooth settings.

Experimental results

Research questions

• RQ1 Which classes of nonconvex, nonsmooth functions are guaranteed to have no spurious local minima?
• RQ2 How do global functions behave under composition and change of variables in optimization?
• RQ3 Can tensor decomposition problems with ℓ₁ regularization be proven to be free of spurious local minima?
• RQ4 What theoretical properties ensure that ℓ₁ regularization avoids outliers in nonconvex optimization?
• RQ5 To what extent can classical analysis theorems, such as uniform limit theorems, be extended to nonconvex and nonsmooth settings?

Key findings

• The class of global functions is closed under uniform limits, extending classical analysis results to nonconvex and nonsmooth settings.
• Global functions are preserved under composition with continuous functions and smooth changes of variables, enabling broader applicability.
• A class of nonconvex, nonsmooth tensor decomposition problems with ℓ₁ regularization are proven to be global functions, implying no spurious local minima exist.
• This is the first theoretical result establishing the absence of spurious minima in nonconvex, nonsmooth optimization with ℓ₁ regularization.
• The ℓ₁ norm is theoretically justified to enhance robustness against outliers in nonconvex optimization due to the absence of spurious local minima in the objective.