[Paper Review] A Fast, Principled Working Set Algorithm for Exploiting Piecewise Linear Structure in Convex Problems

Tyler B. Johnson, Carlos Guestrin|arXiv (Cornell University)|Jul 20, 2018

Machine Learning and Algorithms62 references3 citations

TL;DR

This paper introduces BlitzWS, a fast, principled working set algorithm that exploits piecewise linear structure in convex optimization problems by solving subproblems with theoretical guarantees on progress per iteration. It outperforms LibLinear in convergence speed across multiple problems, including sparse logistic regression and linear SVMs, while introducing BlitzScreen, a novel safe screening test that simplifies the objective more effectively than prior methods.

ABSTRACT

By reducing optimization to a sequence of smaller subproblems, working set algorithms achieve fast convergence times for many machine learning problems. Despite such performance, working set implementations often resort to heuristics to determine subproblem size, makeup, and stopping criteria. We propose BlitzWS, a working set algorithm with useful theoretical guarantees. Our theory relates subproblem size and stopping criteria to the amount of progress during each iteration. This result motivates strategies for optimizing algorithmic parameters and discarding irrelevant components as BlitzWS progresses toward a solution. BlitzWS applies to many convex problems, including training L1-regularized models and support vector machines. We showcase this versatility with empirical comparisons, which demonstrate BlitzWS is indeed a fast algorithm.

Motivation & Objective

• To develop a working set algorithm with theoretical guarantees on progress per iteration, moving beyond heuristic-based subproblem selection.
• To exploit piecewise linear structure in convex problems—such as in ℓ₁-regularized models and SVMs—by formulating the objective as a sum of piecewise terms.
• To design a safe screening test, BlitzScreen, that identifies and discards irrelevant components with broader applicability and greater simplification than prior methods.
• To empirically validate BlitzWS’s superiority in convergence speed across diverse machine learning problems, especially in sparse logistic regression.
• To provide a unified, principled framework for subproblem selection, parameter adaptation, and component discarding in working set optimization.

Proposed method

• BlitzWS formulates optimization as a sequence of subproblems by selectively replacing piecewise terms in the objective with linear subfunctions, enabling faster minimization.
• It uses a theoretical framework to relate subproblem size and stopping criteria to guaranteed progress, enabling principled subproblem construction.
• The algorithm dynamically adapts parameters based on progress analysis, improving efficiency as it converges.
• It introduces a capsule approximation to efficiently compute working sets by approximating the true active set using geometric constructs.
• BlitzScreen is a safe screening test that identifies components guaranteed to be zero in the solution, simplifying the problem before optimization.
• The method applies to a broad class of problems, including ℓ₁-regularized regression, SVMs, and min-norm polytope problems, via a unified piecewise problem formulation.

Experimental results

Research questions

• RQ1 Can a working set algorithm be designed with theoretical guarantees on progress per iteration, rather than relying on heuristics?
• RQ2 How can subproblem size, component selection, and stopping criteria be systematically optimized to maximize convergence speed?
• RQ3 Can a safe screening test be developed that applies to a broader class of problems and simplifies the objective more effectively than existing methods?
• RQ4 To what extent does safe screening improve overall convergence time in practice, despite theoretical advantages?
• RQ5 How does BlitzWS compare in performance to established solvers like LibLinear across diverse convex optimization problems?

Key findings

• BlitzWS significantly outperforms LibLinear in convergence speed, especially for sparse logistic regression, with consistent improvements across multiple datasets.
• The capsule approximation in BlitzWS maintains high fidelity to the true working set, with |W_cap| ≈ |W| in all tested cases, indicating minimal impact on subproblem quality.
• BlitzScreen simplifies the objective more than prior safe screening methods due to a smaller safe region, enabling greater problem reduction.
• Despite its theoretical advantages, BlitzScreen had negligible impact on overall convergence time in practice, suggesting screening is less critical than subproblem design.
• BlitzWS achieves faster convergence than LibLinear in nearly all cases, demonstrating that principled subproblem selection is more impactful than screening for performance gains.