[Paper Review] Optimization, Learning, and Games with Predictable Sequences

Sasha Rakhlin, Karthik Sridharan|arXiv (Cornell University)|Dec 5, 2013

Advanced Bandit Algorithms ResearchDecision Sciences11 references118 citations

TL;DR

This paper introduces Optimistic Mirror Descent, an online learning algorithm leveraging predictable sequences to achieve convergence to minimax equilibrium in zero-sum games at rate O((log T)/T). It extends Mirror Prox to Holder-smooth functions, enables partial information learning, and provides a simple algorithm for approximate Max Flow via convex programming.

ABSTRACT

We provide several applications of Optimistic Mirror Descent, an online learning algorithm based on the idea of predictable sequences. First, we recover the Mirror Prox algorithm for offline optimization, prove an extension to Holder-smooth functions, and apply the results to saddle-point type problems. Next, we prove that a version of Optimistic Mirror Descent (which has a close relation to the Exponential Weights algorithm) can be used by two strongly-uncoupled players in a finite zero-sum matrix game to converge to the minimax equilibrium at the rate of O((log T)/T). This addresses a question of Daskalakis et al [6]. Further, we consider a partial information version of the problem. We then apply the results to convex programming and exhibit a simple algorithm for the approximate Max Flow problem.

Motivation & Objective

• To develop an online learning algorithm based on predictable sequences for optimization and game-theoretic applications.
• To recover and extend the Mirror Prox algorithm for offline optimization, particularly to Holder-smooth functions.
• To demonstrate convergence of Optimistic Mirror Descent to minimax equilibrium in finite zero-sum matrix games under strongly uncoupled player dynamics.
• To address the open question of convergence rate in such games, as posed by Daskalakis et al.
• To apply the framework to convex programming and the approximate Max Flow problem under partial information.

Proposed method

• Uses Optimistic Mirror Descent, an online learning algorithm that incorporates predictable sequence predictions to improve convergence.
• Applies the algorithm to recover Mirror Prox for offline optimization by leveraging predictable sequence structure.
• Adapts the method to handle Holder-smooth functions by extending the smoothness assumptions in the analysis.
• Establishes convergence to minimax equilibrium in zero-sum games via a version of Optimistic Mirror Descent with O((log T)/T) rate.
• Introduces a partial information variant of the algorithm to handle settings with limited feedback.
• Applies the framework to convex programming by reducing the Max Flow problem to a saddle-point problem.

Experimental results

Research questions

• RQ1 Can Optimistic Mirror Descent achieve O((log T)/T) convergence rate in zero-sum games with strongly uncoupled players?
• RQ2 How can Mirror Prox be extended to Holder-smooth functions beyond the standard Lipschitz-continuous case?
• RQ3 What is the role of predictable sequences in improving convergence in online and offline optimization?
• RQ4 Can the algorithm be adapted to partial information settings while preserving convergence guarantees?
• RQ5 Can the framework be used to design a simple, efficient algorithm for approximate Max Flow in convex programming?

Key findings

• Optimistic Mirror Descent achieves a convergence rate of O((log T)/T) to the minimax equilibrium in finite zero-sum matrix games with strongly uncoupled players.
• The paper extends the Mirror Prox algorithm to Holder-smooth functions, broadening its applicability to a wider class of saddle-point problems.
• The method provides a solution to the open problem posed by Daskalakis et al. regarding convergence rate in uncoupled player settings.
• A partial information variant of the algorithm is developed, enabling learning under limited feedback.
• A simple algorithm for the approximate Max Flow problem is derived by reducing it to a convex programming problem using the proposed framework.
• The theoretical results are grounded in the use of predictable sequences, which enhance the convergence properties of the online learning dynamics.