[Paper Review] Deep Online Convex Optimization by Putting Forecaster to Sleep

David Balduzzi|arXiv (Cornell University)|Sep 6, 2015

Advanced Bandit Algorithms Research36 references3 citations

TL;DR

This paper establishes a rigorous connection between error backpropagation in convolutional neural networks and online convex optimization by introducing 'circadian games'—a generalization of convex games with similar convergence properties. It proves that backpropagation is equivalent to playing a circadian game, where the waking-regret of network units controls overall convergence, enabling game-theoretic interpretations of learned representations and adaptive training strategies.

ABSTRACT

Methods from convex optimization such as accelerated gradient descent are widely used as building blocks for deep learning algorithms. However, the reasons for their empirical success are unclear, since neural networks are not convex and standard guarantees do not apply. This paper develops the first rigorous link between online convex optimization and error backpropagation on convolutional networks. The first step is to introduce circadian games, a mild generalization of convex games with similar convergence properties. The main result is that error backpropagation on a convolutional network is equivalent to playing out a circadian game. It follows immediately that the waking-regret of players in the game (the units in the neural network) controls the overall rate of convergence of the network. Finally, we explore some implications of the results: (i) we describe the representations learned by a neural network game-theoretically, (ii) propose a learning setting at the level of individual units that can be plugged into deep architectures, and (iii) propose a new approach to adaptive model selection by applying bandit algorithms to choose which players to wake on each round.

Motivation & Objective

• To establish a formal theoretical link between error backpropagation in deep networks and online convex optimization.
• To explain the empirical success of optimization methods like accelerated gradient descent in non-convex settings such as neural networks.
• To introduce circadian games as a generalization of convex games with comparable convergence behavior.
• To show that backpropagation in convolutional networks is equivalent to playing a circadian game.
• To enable new training paradigms through game-theoretic interpretations of network units and adaptive player waking strategies.

Proposed method

• Introduce circadian games as a generalization of convex games that preserve key convergence properties.
• Model each neuron or layer in a convolutional network as a player in a circadian game.
• Formalize backpropagation as the process of minimizing waking-regret in the game, where 'waking' corresponds to updating a unit.
• Use regret minimization in the game to control the overall convergence rate of the network.
• Apply bandit algorithms to dynamically select which network units ('players') to update ('wake') in each training round.
• Derive game-theoretic interpretations of learned representations and training dynamics.

Experimental results

Research questions

• RQ1 How can error backpropagation in convolutional networks be formally linked to online convex optimization?
• RQ2 What generalization of convex games preserves convergence properties while allowing for non-convex settings?
• RQ3 How does the waking-regret of individual network units relate to the overall convergence of the model?
• RQ4 Can learning dynamics at the unit level be reinterpreted through a game-theoretic lens?
• RQ5 Can adaptive model selection be achieved by treating network units as players in a bandit-style game?

Key findings

• Error backpropagation in a convolutional network is mathematically equivalent to playing a circadian game, establishing a formal bridge between deep learning and online optimization.
• The waking-regret of individual network units directly controls the overall convergence rate of the model, providing a new interpretability mechanism.
• The learned representations in the network can be interpreted as equilibria in the circadian game, offering a game-theoretic understanding of feature learning.
• A novel training strategy is proposed where individual network units are selectively updated based on bandit algorithms, enabling adaptive model selection.
• The framework allows for plug-in learning rules at the unit level that are consistent with global optimization objectives.