[Paper Review] Minmax Optimization: Stable Limit Points of Gradient Descent Ascent are Locally Optimal.

Chi Jin, Praneeth Netrapalli+1 more|arXiv (Cornell University)|Feb 2, 2019

Stochastic Gradient Optimization TechniquesComputer Science48 citations

TL;DR

This paper introduces the local minmax point as a superior optimality criterion for nonconvex-nonconcave minmax optimization, showing that gradient descent ascent (GDA) converges to these points when the ascent step size dominates descent. It establishes that all stable limit points of GDA are locally optimal in a game-theoretically meaningful sense under this condition.

ABSTRACT

Minmax optimization, especially in its general nonconvex-nonconcave formulation, has found extensive applications in modern machine learning frameworks such as generative adversarial networks (GAN), adversarial training and multi-agent reinforcement learning. Gradient-based algorithms, in particular gradient descent ascent (GDA), are widely used in practice to solve these problems. Despite the practical popularity of GDA, however, its theoretical behavior has been considered highly undesirable. Indeed, apart from possiblity of non-convergence, recent results (Daskalakis and Panageas, 2018; Mazumdar and Ratliff, 2018; Adolphs et al., 2018) show that even when GDA converges, its stable limit points can be points that are not local Nash equilibria, thus not game-theoretically meaningful. In this paper, we initiate a discussion on the proper optimality measures for minmax optimization, and introduce a new notion of local optimality---local minmax---as a more suitable alternative to the notion of local Nash equilibrium. We establish favorable properties of local minmax points, and show, most importantly, that as the ratio of the ascent step size to the descent step size goes to infinity, stable limit points of GDA are exactly local minmax points up to degenerate points, demonstrating that all stable limit points of GDA have a game-theoretic meaning for minmax problems.

Motivation & Objective

• Address the theoretical shortcomings of gradient descent ascent (GDA) in minmax optimization, where stable limit points may not correspond to meaningful equilibria.
• Identify the limitations of the local Nash equilibrium concept in nonconvex-nonconcave settings, where such points may not represent stable or optimal outcomes.
• Propose local minmax points as a more appropriate optimality criterion for minmax problems in modern machine learning applications.
• Demonstrate that GDA's stable limit points are locally minmax under a specific step size ratio condition, ensuring game-theoretic relevance.
• Provide theoretical justification for the practical success of GDA in applications like GANs and adversarial training by linking convergence to meaningful equilibria.

Proposed method

• Introduce the concept of local minmax points as a refinement of local Nash equilibria, defined by a local minimax property in the joint strategy space.
• Analyze the dynamics of gradient descent ascent (GDA) under varying step size ratios, particularly focusing on the regime where ascent step size dominates descent.
• Use stability analysis to characterize the limit points of GDA, showing they coincide with local minmax points up to degenerate cases.
• Employ differential equation approximations and Lyapunov-based arguments to study the convergence behavior of GDA trajectories.
• Establish conditions under which the ratio of ascent to descent step size approaching infinity ensures that all stable limit points are local minmax points.
• Formalize the notion of local minmax via a local optimization condition involving the Hessian of the Lagrangian or payoff function.

Experimental results

Research questions

• RQ1 Can stable limit points of GDA in nonconvex-nonconcave minmax problems be guaranteed to have game-theoretic significance?
• RQ2 Is the local Nash equilibrium concept insufficient as an optimality criterion in general minmax problems?
• RQ3 Does a more suitable optimality concept exist that better aligns with the behavior of GDA in practice?
• RQ4 Under what conditions does GDA converge to points that are locally optimal in a meaningful game-theoretic sense?
• RQ5 How does the ratio of ascent to descent step sizes affect the nature of stable limit points in GDA?

Key findings

• Local minmax points are introduced as a more appropriate optimality criterion than local Nash equilibria for nonconvex-nonconcave minmax problems.
• All stable limit points of GDA are shown to be local minmax points when the ratio of the ascent step size to the descent step size tends to infinity, up to degenerate points.
• This result establishes that GDA converges to solutions with game-theoretic meaning under the proposed optimality criterion.
• The paper demonstrates that local Nash equilibria can be unstable or meaningless in practice, while local minmax points avoid such pitfalls.
• The theoretical framework justifies the empirical success of GDA in applications such as GANs and adversarial training by linking convergence to meaningful equilibria.
• The analysis reveals that the step size ratio is a critical control parameter for ensuring convergence to optimally meaningful solutions.