[Paper Review] Follow the Perturbed Leader: Optimism and Fast Parallel Algorithms for Smooth Minimax Games

Arun Sai Suggala, Praneeth Netrapalli|arXiv (Cornell University)|Jun 13, 2020

Advanced Bandit Algorithms Research11 references4 citations

TL;DR

This paper introduces Optimistic Follow the Perturbed Leader (OFTPL), a novel online learning algorithm that enhances regret bounds by incorporating optimism when loss functions are predictable, while preserving optimal worst-case regret. It achieves $O(T^{-1/2})$ convergence for smooth minimax games using only $O(T^{1/2})$ parallel iterations and $T$ oracle calls, significantly improving parallelizability over prior methods.

ABSTRACT

We consider the problem of online learning and its application to solving minimax games. For the online learning problem, Follow the Perturbed Leader (FTPL) is a widely studied algorithm which enjoys the optimal $O(T^{1/2})$ worst-case regret guarantee for both convex and nonconvex losses. In this work, we show that when the sequence of loss functions is predictable, a simple modification of FTPL which incorporates optimism can achieve better regret guarantees, while retaining the optimal worst-case regret guarantee for unpredictable sequences. A key challenge in obtaining these tighter regret bounds is the stochasticity and optimism in the algorithm, which requires different analysis techniques than those commonly used in the analysis of FTPL. The key ingredient we utilize in our analysis is the dual view of perturbation as regularization. While our algorithm has several applications, we consider the specific application of minimax games. For solving smooth convex-concave games, our algorithm only requires access to a linear optimization oracle. For Lipschitz and smooth nonconvex-nonconcave games, our algorithm requires access to an optimization oracle which computes the perturbed best response. In both these settings, our algorithm solves the game up to an accuracy of $O(T^{-1/2})$ using $T$ calls to the optimization oracle. An important feature of our algorithm is that it is highly parallelizable and requires only $O(T^{1/2})$ iterations, with each iteration making $O(T^{1/2})$ parallel calls to the optimization oracle.

Motivation & Objective

• To address the limitation of Follow the Perturbed Leader (FTPL) in leveraging predictable loss sequences, which currently only achieves optimal worst-case regret but not tighter bounds under predictability.
• To develop a variant of FTPL—Optimistic FTPL (OFTPL)—that exploits predictability to achieve improved regret guarantees without sacrificing worst-case robustness.
• To design a highly parallelizable algorithm for solving smooth convex-concave and nonconvex-nonconcave minimax games with fast convergence.
• To enable efficient large-scale training in applications like GANs and adversarial training by reducing iteration count and enabling parallel oracle access.

Proposed method

• Proposes OFTPL, a modification of FTPL that incorporates optimistic prediction of future loss functions using past information.
• Uses a dual view of perturbation as regularization to analyze the stochastic and optimistic components of the algorithm.
• For smooth convex-concave games, relies on a linear optimization oracle; for nonconvex-nonconcave games, uses a perturbed best response oracle.
• Achieves $O(T^{-1/2})$ convergence rate with only $O(T^{1/2})$ iterations, each making $O(T^{1/2})$ parallel calls to the optimization oracle.
• Employs a carefully tuned regularization parameter $\eta$ and oracle sampling size $m$ to balance regret and variance in the analysis.
• Derives regret bounds via Fenchel conjugate duality and strong convexity/smoothness properties, leveraging Theorems H.5 and H.6 on conjugate functions.

Experimental results

Research questions

• RQ1 Can optimism in FTPL improve regret bounds when loss functions are predictable, without degrading worst-case performance?
• RQ2 How can the stochastic and optimistic components of OFTPL be analyzed using a dual view of perturbation as regularization?
• RQ3 What is the optimal trade-off between convergence rate, iteration count, and parallelization in online minimax game solving?
• RQ4 Can OFTPL achieve $O(T^{-1/2})$ convergence with significantly fewer iterations than prior methods, while maintaining oracle efficiency?
• RQ5 How does OFTPL compare to FTRL-style optimistic algorithms in terms of regret guarantees and computational efficiency for minimax games?

Key findings

• OFTPL achieves a regret bound of $O(d^2 D^2 (L+1) \log d + G D \sqrt{\log(8/\delta)}) + O(\min(D^2 L T, \frac{d^2 G^2 \log T}{L} + \frac{d G^2 \log(8/\delta)}{L}))$ for both players in minimax games.
• The algorithm converges to an $O(T^{-1/2})$-accurate solution using only $T$ calls to the optimization oracle.
• It requires only $O(T^{1/2})$ iterations, each with $O(T^{1/2})$ parallel oracle calls, enabling high parallelizability.
• The analysis establishes that perturbation acts as regularization, enabling tighter regret bounds under optimism.
• For smooth convex-concave games, only a linear optimization oracle is needed; for nonconvex-nonconcave games, a perturbed best response oracle suffices.
• The method retains the $O(T^{1/2})$ worst-case regret guarantee of standard FTPL even when losses are unpredictable.