[Paper Review] Limited Memory Kelley's Method Converges for Composite Convex and Submodular Objectives

Song Ping Zhou, Swati Gupta+1 more|arXiv (Cornell University)|Jul 19, 2018

Advanced Optimization Algorithms Research3 references3 citations

TL;DR

This paper introduces Limited Memory Kelley’s Method (L-KM), a novel variant of Kelley’s cutting plane method for minimizing composite convex and submodular objectives. By leveraging duality with a limited-memory Frank-Wolfe variant (L-FCFW) on the submodular base polytope, L-KM achieves linear convergence when the convex component is both smooth and strongly convex, while maintaining at most $n+1$ constraints regardless of iteration count—solving an open problem on convergence rates for such methods.

ABSTRACT

The original simplicial method (OSM), a variant of the classic Kelley's cutting plane method, has been shown to converge to the minimizer of a composite convex and submodular objective, though no rate of convergence for this method was known. Moreover, OSM is required to solve subproblems in each iteration whose size grows linearly in the number of iterations. We propose a limited memory version of Kelley's method (L-KM) and os OSM that requires limited memory (at most n + 1 constraints for an n-dimensional problem) independent of the iteration. We prove convergence for L-KM when the convex part of the objective (g) is strongly convex and show it converges linearly when g is also smooth. Our analysis relies on duality between minimization of the composite objective and minimization of a convex function over the corresponding submodular base polytope. We introduce a limited memory version, L-FCFW, of the Fully-Corrective Frank-Wolfe (FCFW) method with approximate correction, to solve the dual problem. We show that L-FCFW and L-KM are dual algorithms that produce the same sequence of iterates; hence both converge linearly (when g is smooth and strongly convex) and with limited memory. We propose L-KM to minimize composite convex and submodular objectives; however, our results on L-FCFW hold for general polytopes and may be of independent interest.

Motivation & Objective

• To address the lack of convergence rate guarantees for the Original Simplicial Method (OSM), which converges but without known rate.
• To overcome the memory inefficiency of OSM, which accumulates constraints linearly over iterations.
• To develop a limited-memory algorithm for composite convex and submodular objectives that maintains convergence and efficiency.
• To establish a duality between L-KM and a limited-memory Frank-Wolfe variant (L-FCFW) to enable convergence analysis.
• To prove linear convergence of L-KM under smoothness and strong convexity of the convex component.

Proposed method

• Proposes Limited Memory Kelley’s Method (L-KM), which approximates the Lovász extension using at most $n+1$ supporting hyperplanes, independent of iteration count.
• Establishes duality between L-KM and a limited-memory variant of the Fully-Corrective Frank-Wolfe method (L-FCFW) applied to the submodular base polytope.
• Uses the dual problem $\max_{w \in B(F)} h(w)$, where $h$ is smooth and concave, to analyze convergence via duality gap.
• Introduces L-FCFW as a limited-memory, approximately corrected Frank-Wolfe method that maintains active sets of size at most $n+1$.
• Employs a primal-dual framework where the duality gap $p^{(i)} - d^{(i)}$ is used to measure convergence.
• Applies a correction mechanism in L-FCFW that allows pruning of inactive vertices while preserving convergence guarantees.

Experimental results

Research questions

• RQ1 Can a limited-memory variant of Kelley’s method be designed for composite convex and submodular objectives that maintains convergence?
• RQ2 Does such a limited-memory method achieve a provable convergence rate, particularly linear convergence?
• RQ3 Can the duality between L-KM and L-FCFW be exploited to derive convergence guarantees for both algorithms?
• RQ4 Is it possible to bound the active set size in L-FCFW to $n+1$ while preserving convergence?
• RQ5 Can the duality gap be used to establish linear convergence when the convex component is both smooth and strongly convex?

Key findings

• L-KM converges linearly when the convex component $g$ is both smooth and strongly convex, with the duality gap $p^{(i)} - d^{(i)}$ decreasing linearly.
• The duality gap satisfies $p^{(i)} - d^{(i)} \leq (p^\star - d^{(i)}) + M^2/(2\beta)$ when $(p^\star - d^{(i)}) \geq M^2/(2\beta)$, and $p^{(i)} - d^{(i)} \leq M\sqrt{2(p^\star - d^{(i)})/\beta}$ otherwise, with $p^\star - d^{(i)}$ converging linearly.
• OSM also converges linearly under the same conditions, as a corollary of the duality between OSM and L-FCFW.
• L-FCFW maintains active sets of size at most $n+1$, and can be pruned using dual optimal solutions when strict complementary slackness holds.
• Experiments show L-KM uses significantly less memory than OSM, converges at a similar rate in iterations, and improves per-iteration runtime as iterations progress.
• L-FCFW outperforms standard Frank-Wolfe with away steps in dual convergence and maintains smaller subproblems, achieving faster convergence with the same accuracy tolerance.