[Paper Review] HONES: A Fast and Tuning-free Homotopy Method For Online Newton Step

Yuting Ye, Lihua Lei+1 more|arXiv (Cornell University)|Oct 14, 2016

Advanced Optimization Algorithms Research3 citations

TL;DR

HONES is a fast, tuning-free homotopy method for solving sequential generalized projections in Online Newton Step, leveraging warm-starting and sparsity-aware updates to achieve near-linear time complexity. It exploits the rank-one structure of matrix updates and adaptively tracks solution sparsity, reducing computation from O(n³) to O(nk_r) per iteration, with empirical speedups over interior-point methods and no need for parameter tuning.

ABSTRACT

In this article, we develop and analyze a homotopy continuation method, referred to as HONES , for solving the sequential generalized projections in Online Newton Step, as well as the generalized problem known as sequential standard quadratic programming. HONES is fast, tuning-free, error-free (up to machine error) and adaptive to the solution sparsity. This is confirmed by both careful theoretical analysis and extensive experiments on both synthetic and real data.

Motivation & Objective

• To address the computational bottleneck in Online Newton Step, which requires solving sequential quadratic programs with time-varying positive definite matrices.
• To develop a method that is both fast and free of hyperparameter tuning, suitable for high-frequency online optimization.
• To exploit the rank-one update structure of the Hessian matrix and the sparsity of optimal solutions to accelerate convergence.
• To eliminate reliance on interior-point or projected gradient methods that require interior initializers or lack efficient stopping rules.
• To provide a robust, error-free algorithm that adapts to solution sparsity and maintains numerical stability up to machine precision.

Proposed method

• HONES uses a homotopy continuation framework that tracks the solution path across time steps by exploiting the rank-one update structure of the Hessian matrix A^(t).
• It maintains a support set S of non-zero variables and dynamically expands or shrinks it using a two-phase update: lambda-based path-following and u-lambda-based refinement.
• The algorithm employs two key update routines: UPDATE_EXPAND_SUPPORT and UPDATE_UTILDE_EXPAND_SUPPORT, which efficiently handle support changes using sparse matrix updates.
• It uses intermediate variables (Par₁, Par₂, Par₃) to encode problem parameters and enables incremental computation without full matrix re-inversion.
• The method combines warm-starting with adaptive support identification and uses a line search strategy with a stopping rule based on feasibility and optimality conditions.
• It avoids iterative solvers like interior-point methods by directly tracking the solution path via parametric continuation, reducing complexity from O(n³) to O(nk_r).

Experimental results

Research questions

• RQ1 Can a homotopy method be designed to solve sequential generalized projections in Online Newton Step without requiring parameter tuning?
• RQ2 Can the method exploit the rank-one update structure of the Hessian to achieve sub-cubic time complexity?
• RQ3 Can the algorithm adaptively track solution sparsity and maintain numerical accuracy without requiring interior initializers?
• RQ4 Does the method outperform standard interior-point or projected gradient methods in terms of speed and scalability on large-scale online problems?
• RQ5 Can the algorithm maintain O(log T) regret guarantee while reducing per-iteration cost from O(n³) to O(nk_r)?

Key findings

• HONES achieves a per-iteration complexity of O(ns_* + nk_r) where s_* is the number of non-zero variables in the initial solution and k_r is the number of support changes, significantly reducing from O(n³) of interior-point methods.
• The method is tuning-free and error-free up to machine precision, with no need for line search or stopping rule heuristics.
• Empirical results on synthetic and real-world data show HONES achieves speedups of up to 100× over interior-point methods and 10× over SPG on high-dimensional problems.
• The algorithm naturally adapts to solution sparsity, with performance improving as the optimal solution becomes sparser.
• HONES maintains the O(log T) regret bound of Online Newton Step while drastically reducing computational cost per iteration.
• The method successfully handles boundary solutions (common in simplex-constrained problems) where traditional methods like SPG or mirror descent fail due to zero-entry constraints.