[Paper Review] Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms

Arda Sahiner, Tolga Ergen+2 more|arXiv (Cornell University)|Dec 24, 2020

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper establishes that training two-layer vector-output ReLU neural networks with weight decay is equivalent to solving a finite-dimensional convex copositive program, enabling the first polynomial-time algorithms for finding the global minimum when the data rank is fixed. It reveals a deep connection between neural network optimization and copositive programming, with exact solutions achievable via soft-thresholded SVD under certain conditions and a tight copositive relaxation that matches SGD behavior in practice.

ABSTRACT

We describe the convex semi-infinite dual of the two-layer vector-output ReLU neural network training problem. This semi-infinite dual admits a finite dimensional representation, but its support is over a convex set which is difficult to characterize. In particular, we demonstrate that the non-convex neural network training problem is equivalent to a finite-dimensional convex copositive program. Our work is the first to identify this strong connection between the global optima of neural networks and those of copositive programs. We thus demonstrate how neural networks implicitly attempt to solve copositive programs via semi-nonnegative matrix factorization, and draw key insights from this formulation. We describe the first algorithms for provably finding the global minimum of the vector output neural network training problem, which are polynomial in the number of samples for a fixed data rank, yet exponential in the dimension. However, in the case of convolutional architectures, the computational complexity is exponential in only the filter size and polynomial in all other parameters. We describe the circumstances in which we can find the global optimum of this neural network training problem exactly with soft-thresholded SVD, and provide a copositive relaxation which is guaranteed to be exact for certain classes of problems, and which corresponds with the solution of Stochastic Gradient Descent in practice.

Motivation & Objective

• To understand the global optimization landscape of two-layer vector-output ReLU neural networks with weight decay.
• To identify the convex semi-infinite dual of the non-convex neural network training problem.
• To develop provably convergent algorithms for finding the global minimum of the training problem.
• To establish a theoretical link between neural network optimization and copositive programming.
• To characterize conditions under which soft-thresholded SVD yields the global optimum.

Proposed method

• Derive the convex semi-infinite strong dual of the vector-output ReLU network training problem, showing equivalence to a finite-dimensional copositive program.
• Represent the dual using the convex hull of completely positive matrices, linking neural networks to copositive optimization.
• Propose a cutting plane method to solve the dual, with complexity O(n^d poly(nc)) in general and O(n^r poly(nc)) for rank-r data.
• Introduce a copositive relaxation that is exact for certain problem classes and matches SGD solutions in practice.
• Extend results to general convex loss functions via Fenchel duality and to linear-activation networks via nuclear norm regularization.
• Apply Frank-Wolfe methods to the dual formulation, with modifications for general convex losses.

Experimental results

Research questions

• RQ1 Can the global minimum of the vector-output ReLU neural network training problem be characterized via convex optimization?
• RQ2 What is the precise connection between neural network training and copositive programs?
• RQ3 Under what conditions can the global optimum be found exactly using soft-thresholded SVD?
• RQ4 How does the computational complexity scale with data dimension and rank?
• RQ5 Is there a provably tight relaxation of the neural network problem that matches SGD behavior in practice?

Key findings

• The vector-output ReLU neural network training problem is equivalent to a finite-dimensional convex copositive program, establishing a novel and fundamental connection between deep learning and copositive optimization.
• For data matrices of fixed rank r, the global minimum can be found in polynomial time, with complexity O(n^r poly(nc)) using a cutting plane method.
• Exact global solutions are provably attainable via soft-thresholded SVD when the data matrix has low rank and satisfies specific structural conditions.
• A copositive relaxation of the problem is exact for certain classes of problems and empirically matches the solutions found by Stochastic Gradient Descent.
• The method generalizes to arbitrary convex loss functions through Fenchel duality, yielding a finite-dimensional convex dual with the same structure.
• For linear-activation networks, the global minimum corresponds to a nuclear norm-regularized problem, with a finite-dimensional strong bi-dual.