[Paper Review] Randomized Block Cubic Newton Method

Nikita Doikov, Peter Richtárik|arXiv (Cornell University)|Feb 12, 2018

Stochastic Gradient Optimization Techniques22 references3 citations

TL;DR

This paper introduces the Randomized Block Cubic Newton (RBCN) method, a novel second-order optimization algorithm for minimizing composite convex functions composed of a smooth, a twice-differentiable, and a nonsmooth term. By combining block-coordinate randomization with cubic regularization and proximal modeling, RBCN achieves optimal convergence rates—$\mathcal{O}(1/\epsilon)$, $\mathcal{O}(1/\sqrt{\epsilon})$, and $\mathcal{O}(\log(1/\epsilon))$—and outperforms state-of-the-art methods on machine learning problems including regularized least-squares, logistic regression, and Poisson regression.

ABSTRACT

We study the problem of minimizing the sum of three convex functions: a differentiable, twice-differentiable and a non-smooth term in a high dimensional setting. To this effect we propose and analyze a randomized block cubic Newton (RBCN) method, which in each iteration builds a model of the objective function formed as the sum of the natural models of its three components: a linear model with a quadratic regularizer for the differentiable term, a quadratic model with a cubic regularizer for the twice differentiable term, and perfect (proximal) model for the nonsmooth term. Our method in each iteration minimizes the model over a random subset of blocks of the search variable. RBCN is the first algorithm with these properties, generalizing several existing methods, matching the best known bounds in all special cases. We establish ${\cal O}(1/ε)$, ${\cal O}(1/\sqrtε)$ and ${\cal O}(\log (1/ε))$ rates under different assumptions on the component functions. Lastly, we show numerically that our method outperforms the state-of-the-art on a variety of machine learning problems, including cubically regularized least-squares, logistic regression with constraints, and Poisson regression.

Motivation & Objective

• To develop a second-order optimization method that efficiently handles composite convex problems with distinct smoothness properties: smooth, twice-differentiable, and nonsmooth components.
• To integrate block-coordinate randomization with cubic regularization and proximal modeling to reduce computational cost while maintaining global convergence guarantees.
• To generalize existing methods such as cubic Newton, stochastic Newton, and proximal gradient methods into a unified framework with improved convergence rates.
• To establish theoretical convergence rates under varying smoothness assumptions on the component functions.
• To demonstrate empirically that RBCN outperforms state-of-the-art algorithms in terms of total computational time across diverse machine learning tasks.

Proposed method

• RBCN constructs a model of the objective function using three distinct approximations: a linear model with quadratic regularization for the smooth term $g(x)$, a quadratic model with cubic regularization for the twice-differentiable term $\phi(x)$, and a perfect (proximal) model for the nonsmooth term $\psi(x)$.
• In each iteration, the method minimizes this composite model over a randomly selected subset of blocks of the variable $x$, exploiting block structure to reduce per-iteration cost.
• The algorithm uses a randomized block sampling strategy (specifically $\tau$-nice sampling) to select blocks, enabling scalability in high-dimensional settings with large $n$.
• For the twice-differentiable term $\phi_i(x_{(i)})$, the method employs a cubically regularized second-order model to capture curvature, improving convergence over quadratic models.
• The method incorporates Lipschitz constants of Hessians ($H_k$) to ensure global convergence and stability, with $H_k$ chosen as the maximum of $c_i$ over selected blocks.
• The algorithm is designed to be proximal for $\psi_i(x_{(i)})$, meaning it minimizes the exact term without approximation, preserving convergence guarantees.

Experimental results

Research questions

• RQ1 Can a randomized block method be designed that combines cubic regularization with proximal modeling to achieve optimal convergence rates for composite convex problems?
• RQ2 How does the choice of block size affect the convergence speed and computational cost in a randomized second-order method?
• RQ3 Can the proposed method generalize existing algorithms like cubic Newton and stochastic Newton while matching or improving their convergence rates?
• RQ4 What is the theoretical convergence rate of the method under different smoothness assumptions on the component functions?
• RQ5 Does the method outperform state-of-the-art algorithms in practice on real-world machine learning problems?

Key findings

• RBCN achieves $\mathcal{O}(1/\epsilon)$, $\mathcal{O}(1/\sqrt{\epsilon})$, and $\mathcal{O}(\log(1/\epsilon))$ convergence rates under different assumptions on the component functions, matching the best-known bounds in all special cases.
• Numerical experiments show that using medium-sized blocks (e.g., 25–50 coordinates) minimizes total computational time, balancing iteration speed and cost per iteration.
• On synthetic cubically regularized least-squares problems, RBCN outperforms other methods in terms of total time to reach $10^{-12}$ accuracy in function residual.
• For $\ell_2$-regularized logistic regression, RBCN with block sizes of 25–50 outperforms both block coordinate gradient descent and other methods in computational efficiency.
• In Poisson regression tasks, RBCN requires fewer data accesses than SDCA and SDNA to reach the same accuracy, demonstrating superior empirical efficiency.
• The method maintains computational efficiency comparable to SDNA while achieving faster convergence, especially on synthetic and real-world datasets with $m=1000$ to $m=319$ samples.