[Paper Review] Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training

Fangda Gu, Armin Askari+1 more|arXiv (Cornell University)|Nov 1, 2018

Advanced Neural Network ApplicationsComputer Science19 citations

TL;DR

Fenchel lifted networks introduce a novel neural network training framework that formulates activation functions as biconvex constraints and uses Lagrange relaxation to derive a rigorous lower bound on the standard training objective. The method enables efficient block-coordinate descent optimization with parallelization across data points and layers, achieving performance on par with or better than standard fully connected and convolutional networks.

ABSTRACT

Despite the recent successes of deep neural networks, the corresponding training problem remains highly non-convex and difficult to optimize. Classes of models have been proposed that introduce greater structure to the objective function at the cost of lifting the dimension of the problem. However, these lifted methods sometimes perform poorly compared to traditional neural networks. In this paper, we introduce a new class of lifted models, Fenchel lifted networks, that enjoy the same benefits as previous lifted models, without suffering a degradation in performance over classical networks. Our model represents activation functions as equivalent biconvex constraints and uses Lagrange Multipliers to arrive at a rigorous lower bound of the traditional neural network training problem. This model is efficiently trained using block-coordinate descent and is parallelizable across data points and/or layers. We compare our model against standard fully connected and convolutional networks and show that we are able to match or beat their performance.

Motivation & Objective

• To address the non-convexity and optimization difficulty in training deep neural networks.
• To develop a lifted neural network architecture that maintains performance while introducing structural benefits from convex relaxation.
• To provide a rigorous lower bound on the standard neural network training objective using Lagrange multipliers.
• To enable efficient, parallelizable training via block-coordinate descent on the lifted problem.
• To demonstrate that lifted models can match or surpass classical networks in performance without degradation.

Proposed method

• Representing each activation function as an equivalent biconvex constraint to structure the optimization landscape.
• Applying Lagrange relaxation to derive a lower bound of the original non-convex training problem.
• Using block-coordinate descent to optimize the relaxed dual problem, enabling parallelization across data points and layers.
• Formulating the training objective as a Fenchel conjugate-based dual problem to ensure convexity in subproblems.
• Maintaining end-to-end differentiability and compatibility with standard network architectures through the lifting framework.
• Solving the dual problem iteratively to converge to a solution that bounds the original network's loss.

Experimental results

Research questions

• RQ1 Can a lifted neural network framework achieve performance comparable to or better than standard networks while improving optimization structure?
• RQ2 Does the use of Lagrange relaxation and biconvex constraints provide a tighter and more tractable lower bound on the original training objective?
• RQ3 Can the proposed method be efficiently optimized using block-coordinate descent with parallelization across data and layers?
• RQ4 How does the performance of Fenchel lifted networks compare to standard fully connected and convolutional networks on benchmark tasks?
• RQ5 Does the lifting approach eliminate the performance degradation often seen in prior lifted models?

Key findings

• Fenchel lifted networks achieve performance on par with or better than standard fully connected and convolutional networks on benchmark tasks.
• The method provides a rigorous lower bound on the original non-convex training objective through Lagrange relaxation of activation constraints.
• Block-coordinate descent enables efficient and scalable optimization, with parallelization possible across data points and network layers.
• The framework avoids the performance degradation typically observed in prior lifted models, maintaining competitive accuracy.
• The approach successfully balances structural optimization benefits with high model capacity and generalization.