[Paper Review] Deep Neural Network Training with Frank-Wolfe

Sebastian Pokutta, Christoph Spiegel+1 more|arXiv (Cornell University)|Oct 14, 2020

Stochastic Gradient Optimization Techniques44 references4 citations

TL;DR

This paper proposes using stochastic Frank-Wolfe algorithms for training deep neural networks with constrained weight parameters, avoiding costly projection steps by leveraging linear minimization oracles. It demonstrates that, with appropriate convex feasible regions like L2- or Linfty-norm balls, Frank-Wolfe training achieves test accuracy exceeding standard SGD and matching state-of-the-art results with L2 regularization, while also enabling control over model sparsity and learned representations.

ABSTRACT

This paper studies the empirical efficacy and benefits of using projection-free first-order methods in the form of Conditional Gradients, a.k.a. Frank-Wolfe methods, for training Neural Networks with constrained parameters. We draw comparisons both to current state-of-the-art stochastic Gradient Descent methods as well as across different variants of stochastic Conditional Gradients. In particular, we show the general feasibility of training Neural Networks whose parameters are constrained by a convex feasible region using Frank-Wolfe algorithms and compare different stochastic variants. We then show that, by choosing an appropriate region, one can achieve performance exceeding that of unconstrained stochastic Gradient Descent and matching state-of-the-art results relying on $L^2$-regularization. Lastly, we also demonstrate that, besides impacting performance, the particular choice of constraints can have a drastic impact on the learned representations.

Motivation & Objective

• To investigate the feasibility and efficacy of using projection-free stochastic Frank-Wolfe methods for training deep neural networks with constrained parameters.
• To compare stochastic Frank-Wolfe variants against standard stochastic gradient descent (SGD) and adaptive methods in terms of generalization and performance.
• To examine how the choice of convex feasible region impacts model generalization, sparsity, and learned representations.
• To demonstrate that constrained Frank-Wolfe training can match or exceed the performance of SGD with L2 regularization.
• To provide open-source implementations in TensorFlow and PyTorch to support reproducibility and further research.

Proposed method

• The method employs stochastic Frank-Wolfe (SFW) with momentum, replacing projection steps in constrained optimization with linear minimization oracle (LMO) calls.
• At each iteration, the LMO solves $ v_t = \arg\min_{v \in \mathcal{C}} \langle \tilde{\nabla}L(\theta_t), v \rangle $, where $ \mathcal{C} $ is a convex and compact feasible region.
• Parameters are updated via $ \theta_{t+1} = \theta_t + \alpha (v_t - \theta_t) $, ensuring feasibility through convex combination.
• The approach avoids expensive projections by relying on efficient LMOs, particularly effective when $ \mathcal{C} $ is structured (e.g., L2-ball, Linfty-ball, K-sparse polytope).
• The method is applied to fully connected and convolutional networks on MNIST, CIFAR-10, CIFAR-100, and ImageNet, with hyperparameters tuned for fair comparison.
• The authors use momentum and variance-reduction techniques inspired by ORGFW and SPIDER-FW to improve practical convergence.

Experimental results

Research questions

• RQ1 Can stochastic Frank-Wolfe algorithms effectively train deep neural networks when parameters are constrained to a convex feasible region?
• RQ2 How does the performance of Frank-Wolfe training compare to standard SGD and SGD with weight decay across standard benchmarks?
• RQ3 To what extent does the choice of feasible region influence model sparsity and the structure of learned representations?
• RQ4 Can Frank-Wolfe training achieve state-of-the-art test accuracy without explicit L2 regularization?
• RQ5 What is the impact of different LMOs (e.g., for L2, Linfty, or sparse polytopes) on training dynamics and generalization?

Key findings

• Frank-Wolfe training with L2-norm constrained parameters achieves test accuracy exceeding standard SGD and matching the performance of SGD with L2 regularization on CIFAR-10 and ImageNet.
• When constraining parameters to the Linfty-norm ball, SFW still outperforms standard SGD without weight decay, indicating a regularization effect from the constraint itself.
• Networks constrained to K-sparse polytopes exhibit significantly reduced active parameters during training, demonstrating the ability to induce sparsity through the feasible region choice.
• Visualization of learned weights shows that different feasible regions—such as L2-balls or sparse polytopes—lead to drastically different learned representations, with distinct patterns in feature activation.
• Runtime performance of SFW was comparable to SGD due to efficient LMOs, particularly for L2- and Linfty-norm constraints, making the method computationally viable.
• The authors released open-source implementations in TensorFlow and PyTorch, enabling reproducibility and further research in constrained deep learning.