[Paper Review] Linear Convergence of the Primal-Dual Gradient Method for Convex-Concave Saddle Point Problems without Strong Convexity

Simon S. Du, Wei Hu|arXiv (Cornell University)|Feb 5, 2018

Advanced Optimization Algorithms Research50 citations

TL;DR

The authors prove that the vanilla primal-dual gradient method converges linearly for convex-concave saddle point problems even when the primal function is not strongly convex, provided the coupling matrix A has full column rank.

ABSTRACT

We consider the convex-concave saddle point problem $\\min_{x}\\max_{y} f(x)+y^\ op A x-g(y)$ where $f$ is smooth and convex and $g$ is smooth and strongly convex. We prove that if the coupling matrix $A$ has full column rank, the vanilla primal-dual gradient method can achieve linear convergence even if $f$ is not strongly convex. Our result generalizes previous work which either requires $f$ and $g$ to be quadratic functions or requires proximal mappings for both $f$ and $g$. We adopt a novel analysis technique that in each iteration uses a "ghost" update as a reference, and show that the iterates in the primal-dual gradient method converge to this "ghost" sequence. Using the same technique we further give an analysis for the primal-dual stochastic variance reduced gradient (SVRG) method for convex-concave saddle point problems with a finite-sum structure.

Motivation & Objective

• Motivate solving convex-concave saddle point problems with first-order methods in large-scale settings.
• Show that linear convergence is achievable without primal strong convexity under a full-column-rank coupling A.
• Provide a novel analysis technique using a ghost reference sequence to prove convergence.
• Extend the analysis to primal-dual stochastic variance reduced gradient (SVRG) methods for finite-sum structures.

Proposed method

• Study the saddle point problem min_x max_y f(x) + y^T A x - g(y) with f smooth and convex, g smooth and strongly convex.
• Use Algorithm 1 (primal-dual gradient updates) and establish linear convergence under specific parameter choices.
• Introduce a ghost-sequence reference corresponding to a gradient descent on the primal problem to analyze convergence.
• Define a potential function P_t = λ a_t + b_t with a_t = ||x_t - x*|| and b_t = ||y_t - ∇g^*(Ax_t)||, and show geometric decay.
• Extend the framework to a primal-dual SVRG method for finite-sum problems and prove O((n + κ)d log(1/ε))-type complexity.
• Provide corollaries detailing iteration complexity: O*(log(P_0/ε)) to achieve ε-accuracy.

Experimental results

Research questions

• RQ1 Can the vanilla primal-dual gradient method achieve linear convergence when the primal function f is not strongly convex, given g is strongly convex and A has full column rank?
• RQ2 How does the convergence analysis extend to stochastic variance reduced gradient (SVRG) variants for finite-sum saddle point problems?
• RQ3 What roles do smoothness constants and the condition number of A play in guaranteeing linear convergence and determining step sizes?

Key findings

• Under mild smoothness and convexity assumptions (f convex and ρ-smooth; g α-strongly convex and β-smooth) with rank(A) = d1, the primal-dual gradient method converges linearly to (x*, y*).
• A novel ghost-reference analysis shows the primal iterates converge toward a ghost sequence generated by gradient descent on the primal problem, enabling linear rates without primal strong convexity.
• For finite-sum problems, the primal-dual SVRG method achieves O((n + κ)d log(1/ε)) iteration complexity, matching standard SVRG rates for smooth strongly convex objectives.
• Empirical results corroborate linear convergence of both the primal-dual gradient method and its SVRG variant, with SVRG offering faster rates especially at high condition numbers.